Telemetry Uploader was built to move telemetry from front-end Event Hubs into a shared analytical storage backend, where research and analysis teams can consume it reliably. The service reads telemetry events, loads referenced blob payloads, converts them into telemetry envelopes, and writes them in a durable protobuf format.
At small or moderate scale, this is mostly an ingestion problem. At production scale, it becomes a distributed systems problem. The service has to deal with regional latency, Event Hub partition ownership, checkpointing semantics, write limits, batching efficiency, memory pressure, and large differences in telemetry volume across regions.
What Telemetry Uploader does
Telemetry producers write source payloads into object storage and send blob references through Event Hub. Telemetry Uploader consumes those messages and performs the heavy lifting:
- Read a batch of telemetry events from Event Hub.
- Read the associated blob payloads.
- Convert payloads into telemetry envelope objects.
- Route entities into in-memory channels by entity type, partition, and time.
- Packetize data into storage-compatible protobuf payloads.
- Flush the packets to the analytical storage backend.
- Report processing, retry, failure, latency, checkpoint, and throughput metrics.
Architecture overview
Telemetry Uploader runs as a stateless service and relies on an Event Hub telemetry library for reads, checkpointing, retry handling, cooldown behavior, and error accounting. The service owns producer logic, while a shared entity library owns packet construction and write abstractions.
The service is deployed on Kubernetes across multiple Azure regions to improve resiliency, regional failover options, and proximity to high-volume telemetry sources.
A key design choice is partition-aware processing. Event Hub partitions define the natural concurrency model. The service scales horizontally by assigning workers to partitions through a consumer group.
The write-size constraint
The pipeline writes through an SDK path that constrains individual writes to a bounded packet size. That limit shapes the implementation. The service cannot append arbitrarily large telemetry batches; it must batch efficiently while preserving protobuf boundaries.
Each packet includes a delimiter, signature, payload metadata, integrity hash, and protobuf payload. This structure allows downstream readers to process appended files safely and validate packet integrity.
When latency exposes checkpointing behavior
One high-latency region exposed an important interaction between slow processing and Event Hub checkpointing. Reads from blob storage and writes to the analytical backend were both slow. That reduced throughput, but it also created cases where workers did not complete checkpointing before partition ownership moved to another worker.
When checkpointing does not complete, already uploaded data can be reprocessed. From Event Hub's perspective, the partition position has not advanced. A new owner can pick up the same events again, causing duplicate work and blocking forward progress.
Observed symptoms
- No checkpointing for some partitions.
- Partition ownership transferring before processing and checkpointing completed.
- Successfully uploaded data being reprocessed.
- Processing becoming blocked under sustained latency pressure.
Original correctness-first design
The original design prioritized no data loss. Batches were processed before messages were marked complete, and checkpointing advanced only after the service had enough confidence that data had been written safely.
This design avoids checkpointing before durable writes and reduces checkpoint frequency. But under high latency, the downside becomes visible. If a batch takes too long, partition ownership can move before checkpointing finishes. The system then does not reflect work already completed.
The per-entity write problem
A second scaling issue was write granularity. The intended design was to batch data into larger packets, similar to prior forwarding systems. But the implementation became too conservative: to guarantee no data loss, it could flush data per entity, even when an entity was very small.
At large scale, that does not work. Billions of entities multiplied by one write per entity creates an enormous number of write calls. This increases latency, reduces throughput, and makes the system more sensitive to regional network conditions.
Current behavior to avoid at scale
- Read events from Event Hub.
- Push each event to an entity channel.
- Immediately write small payloads.
- Checkpoint after processing.
- Repeat with high write-call volume.
Scalable direction
- Read events from Event Hub.
- Push events into in-memory channels.
- Batch until a bounded packet threshold.
- Flush asynchronously.
- Advance checkpoints safely and progressively.
Scalable Direction Diagram
Proposed batching redesign
The redesigned approach moves Telemetry Uploader closer to the original bulk-write intent. Instead of flushing tiny payloads immediately, entity channels accumulate data until they approach the packet threshold and then flush asynchronously.
This dramatically reduces write-call volume. It also helps the Event Hub processing loop avoid stop-and-wait behavior, allowing the service to continue reading and parsing while writes complete in parallel.
The checkpointing tradeoff
The redesign introduces a real tradeoff. If the service checkpoints earlier while data is still buffered in memory, a worker crash could lose unflushed data. The exposure can be bounded, but it is not zero.
The right answer is not simply choosing safety or speed. The practical design is to batch aggressively for throughput, keep checkpointing semantics honest, add recovery mechanisms for buffered data, and make residual risk visible and bounded.
Event Hub checkpointing constraints
Event Hub checkpointing has ordering constraints. A consumer cannot safely checkpoint a later message and then go back to an earlier one. The checkpoint represents a position in the partition stream.
That means per-message checkpointing is only safe when all earlier messages in the partition have been processed or intentionally classified as failed. If processing is parallelized, the service needs to know the highest consecutive completed offset before checkpointing.
Healthy services can still fall behind
Another high-volume region showed a related but different scaling problem. The service was healthy: workers were running, processing was active, and errors were not the primary driver. The issue was sustained throughput. Incoming volume exceeded the rate at which data could be written.
This distinction matters. A healthy service can still build a backlog if write latency is high, payload sizes grow, or write parallelism is insufficient.
Cross-region processing
A major short-term improvement came from validating cross-region processing. Instead of running workers only in the source region, the team tested running workers from a lower-latency region while continuing to read source-region data.
This validated an important hypothesis: the bottleneck was not only Event Hub reads or worker count. The network relationship between the processing region and the storage backend materially affected ingestion throughput.
Increasing write parallelism
The next major improvement is increasing write parallelism by changing output partitioning. The original entity channel model grouped data by entity type, partition, and larger time windows. A more granular model creates more independent output targets and allows the service to spread writes across more files.
The tradeoff is more files and more coordination, but for high-volume regions the throughput gain is worth it.
Handling dominant entities separately
When one entity type accounts for a large share of traffic, it can dominate buffers, write pressure, and backlog behavior. Treating dominant entities separately gives the service more control over batching, prioritization, and write concurrency.
At small scale, uniform entity handling is simpler. At large scale, skew matters.
Observability and operational safety
Telemetry Uploader exposes operational metrics and logs for cooldown, retry spikes, processing failures, critical errors, checkpoint failures, queue pressure, rate limiting, and memory pressure.
The current operating target is 120 terabytes uploaded per day. The acceptable latency SLA is a maximum 5-minute delay at the 95th percentile. Real-time processing remains the north-star goal as the system evolves.
The most important lesson is that success counts alone are insufficient. Backlog and creation-to-processing latency are critical signals. For telemetry pipelines, processed eventually is not always good enough.
Lessons learned
- Healthy services can still fall behind. Throughput and latency problems may not show up as simple error-rate problems.
- Latency can expose checkpointing weaknesses. Slow reads and writes can cause partition ownership transfers before checkpointing completes.
- Per-entity writes do not scale. A correctness-first implementation that flushes tiny entities immediately can become the bottleneck.
- Batching must be deliberate. Larger packets reduce write calls, but the service must preserve payload boundaries and manage buffered-data risk.
- Checkpointing must respect partition order. Parallel processing is useful, but checkpoints must advance only through the highest consecutive completed position.
- Process stability is non-negotiable. Long-running ingestion workers must avoid memory leaks, dispose objects efficiently, and continuously monitor memory growth to prevent throughput collapse and restarts.
- Infrastructure is not the only lever. Service-level throughput improvements should come first when the pipeline is inefficient.
- Observability must measure freshness. Backlog and creation-to-processing latency are critical operational signals.
Conclusion
Telemetry Uploader began as a telemetry bridge from Event Hubs to a shared analytical backend. At production scale, it became a distributed ingestion system shaped by regional latency, partition ownership, checkpoint semantics, write limits, memory pressure, and telemetry growth.
The path forward is clear: batch writes closer to the packet limit, increase write parallelism, checkpoint safely but more progressively, recover buffered data through a storage-backed safety mechanism, and place processing closer to the best write path where needed.
Telemetry Uploader's evolution is a useful example of production engineering in practice: start with a correct design, observe where real traffic breaks the assumptions, and then reshape the architecture around the bottlenecks that actually matter.