Slack migrated over 700 production data pipeline jobs from an SSH-based execution model to a REST-based architecture to improve security and infrastructure modularity. By replacing direct SSH access to EMR master nodes with a standardized API-driven approach, the team eliminated technical debt, resolved resource contention, and unblocked their transition to Kubernetes-based orchestration.
Key points
Replacing SSH-based job execution with REST APIs eliminates security surface area and simplifies audit logging.
Decoupling job execution from cluster master nodes prevents resource contention and improves overall job reliability.
Centralized monitoring of migration progress through database tracking is critical for managing large-scale infrastructure changes.
