Datadog addressed the high cost of using frontier LLMs for automated incident investigation by fine-tuning a smaller, specialized Qwen3.5-9B model. By using investigation traces generated by a teacher model to train the smaller model, they achieved 87% recall while reducing inference costs by 20x. This approach enables scalable agentic investigations that were previously cost-prohibitive to perform across high-volume production alerts.
Key points
Fine-tuning smaller models using traces from frontier teacher models can replicate high-reasoning capabilities at a fraction of the cost.
A training data flywheel allows for the continuous expansion of datasets as new production investigations provide fresh labels.
Cost-effective AI inference is a critical requirement for integrating agentic reasoning into high-scale monitoring and observability workflows.
Rules-based systems provide low-latency diagnostics, while agentic models provide deeper, context-aware reasoning for complex incidents.