Airbnb engineers detail their approach to operationalizing and evaluating coding agents powered by Amazon Bedrock. They implement a dual-layered strategy of pre-deployment evaluations using custom task sets and post-deployment observability using CloudWatch metrics to manage model selection, cost, and latency.
Key points
Define cost-per-task metrics based on actual agent workloads rather than relying solely on model vendor benchmarks.
Monitor latency and throughput using Bedrock runtime metrics like InvocationLatency and EstimatedTPMQuotaUsage to anticipate throttling.
Establish side-by-side performance comparisons between new model releases and existing production deployments before full rollout.
Utilize specific CloudWatch namespaces to track token consumption across different API patterns like Converse and InvokeModel.