Cloudflare optimized its internal AI code review pipeline by identifying and reducing token usage and refining model selection. Through message consistency improvements and aggressive effort management, the team reduced average review costs by 68% and cut median execution time by more than half.
Key points
Model performance improvements often lead to larger inputs and higher token costs over time.
Caching strategies for LLM prompts are highly effective when request message order is made consistent across different commits.
Cost optimization should be balanced against quality metrics to ensure the reduction in effort does not impact the detection of real issues.
Reviewing agent session configurations can identify significant waste in token consumption.