This article outlines a data-driven methodology for configuring auto-scaling policies on latency-sensitive text-to-speech (TTS) endpoints using Amazon SageMaker. By measuring the maximum sustainable concurrency (Q_max) and the end-to-end scale-out time (T_total), engineers can derive precise scaling thresholds that prevent service-level objective (SLO) breaches during traffic spikes. The approach replaces speculative configuration with empirical measurements applicable to any concurrency-bound real-time workload.
Key points
Default auto-scaling metrics often fail for non-autoregressive TTS models because they lack token-generation loops to instrument.
The primary bottleneck for real-time TTS endpoints is concurrent in-flight requests rather than raw throughput.
Measure Q_max (maximum sustainable concurrency) and T_total (scale-out duration) to calculate precise scaling policy thresholds.
Using native Amazon SageMaker metrics for auto-scaling prevents common failures associated with custom metric publishing.
The measurement-driven scaling methodology generalizes to any latency-sensitive, concurrency-bound SageMaker endpoint.