This article explores optimizing high-density reinforcement learning environments using Firecracker microVMs on Amazon EC2 metal instances. It details a methodology for balancing rollout density against tail latency to minimize costs and maximize throughput in training loops. The authors provide a measurement harness, 'labsweep', to help engineers identify the optimal tuning point for their specific hardware and workload configurations.
Key points
High-density sandboxing with Firecracker requires careful balancing of CPU topology, cgroup hierarchies, and NUMA-aware memory placement to avoid latency degradation.
Tail latency in RL training environments often creates a non-linear cliff as environment density increases, which can be measured using tools like labsweep.
Training algorithm group sizes significantly amplify the impact of stragglers; larger group sizes require more conservative density limits.
Host agents running as Kubernetes DaemonSets can effectively manage microVM lifecycles and warm pools to reduce dispatch latency on metal hardware.