The Future of AI Infra: From Kubernetes to Agent Sandboxes

Why it matters

Akshat Bubna, CTO of Modal, joins Latent Space to explore the evolution of AI infrastructure, arguing that traditional Kubernetes no longer fits AI workloads and envisioning agent sandboxes as the next-generation runtime. Drawing on Modal's engineering practice, he dissects GPU scheduling, elastic scaling, and security isolation, offering forward-looking insights for building efficient and scalable AI applications.

Key takeaways

• Kubernetes was built for microservices, struggling to meet the demanding GPU orchestration and dynamic scheduling needs of AI workloads.
• Future AI infrastructure will shift toward agent-centric sandboxes, enabling isolated, reusable, and secure execution.
• Elastic scaling and cold-start optimization are key to cost control in AI applications, making serverless the mainstream model.
• Infrastructure must abstract underlying complexity so developers can focus on models and business logic, not operations.