We’re in the middle of the biggest shift in how we build software. Traditionally, software is very deterministic: whatever pre-defined instructions you write, your software behaves exactly like that. With deterministic software, it’s pretty intuitive to secure it, because you mostly know the points of failure and you secure them.

But this shift has been tremendous with the coming of agentic AI that uses large language models. The old rules still apply, but they’re not enough anymore, and figuring out why they’re not enough and how you can fix it is basically what this blog is about.

Running AI infrastructure breaks a lot of the traditional ways we’re used to dealing with systems. In this blog, I write about how you can serve AI models efficiently for inference on Kubernetes on hardware like GPUs (which are super expensive!!!), and how vLLM can help you manage memory efficiently while llm-d can help load balance the system smartly to your needs.