Datadog developed gitretriever to address CI performance bottlenecks caused by massive monorepo cloning traffic. By utilizing a content-addressed, disposable architecture instead of relying on traditional replicated backend scaling, the system reduced CPU load by 3–4x while maintaining low latency during high-volume periods.
Key points
Standard horizontal scaling (adding nodes) often fails in Git environments because high concurrency in large monorepos leads to extreme CPU contention.
Content-addressable storage enables safe, coordination-free replication by verifying data by its hash rather than its source.
Making systems disposable (rebuildable) instead of authoritative simplifies architecture and removes complex consistency requirements.
Optimizing CI performance is often more effective by skipping full repository clones in favor of targeted API calls.