Datadog evolved its metrics aggregation system through three generations to address scaling constraints caused by state management during recovery and processing. By shifting from fixed partition ownership to a model where durable storage reconciles partial aggregates and late data, the system successfully decoupled state from the streaming layer. This architecture allows workers to resume at the live edge without needing to restore hours of historical aggregation state.
Key points
Fixed state ownership on individual workers limits horizontal scaling and complicates recovery operations.
Decoupling durable state from the processing layer allows stream workers to resume near the live edge without historical state restoration.
Storage layers can efficiently handle late-arriving data and partial aggregates using reconciliation rather than blocking on stateful processing.
Aggregating dimensions at the edge reduces downstream costs and query complexity while maintaining mathematical accuracy.