Datadog developed Distributed DataFusion, an open-source Rust framework, to enable cross-node query execution for Apache DataFusion. By extending DataFusion's capabilities, this framework allows large-scale interactive queries to maintain low latency across distributed compute resources. The project provides a composable foundation for building distributed query engines rather than acting as a fixed, proprietary service.
Key points
Apache DataFusion requires a distributed extension framework to handle large-scale datasets that exceed single-machine memory or compute capacity.
A framework-based approach allows developers to inject custom data sources, optimization rules, and network stacks into the query execution lifecycle.
Decoupling the distributed query layer from the data source allows for shared infrastructure across diverse telemetry and observability datasets.
Leveraging Apache Arrow and Substrait provides the necessary standardization for building interoperable, composable query systems.