Hello Reader,

This question is getting more popular fast, driven largely by companies worried about sending proprietary data to a model provider. In this edition, we will go over how to self-host it on Kubernetes.

Good answer:

• The agent code itself is straightforward. Containerize it, push it to Amazon ECR, and run it as a pod.
• The model is the hard part. A model has two pieces: the model image, containing the tokenizer and configuration, and the model weights, which for an 80 billion parameter model means 80 billion floating point numbers.

How the weights work:

This part is just for your understanding.

When you send a prompt, the tokenizer converts your text into tokens, and those tokens get pushed through every one of those weights to predict the next token. That process repeats for every single word the model generates.

A CPU has a small number of powerful cores built for sequential tasks: application logic, branching, one thing after another. Running 80 billion weight calculations one at a time would take forever.

A GPU has thousands of simpler cores built to do the same operation across massive amounts of data simultaneously. Gen AI inference is essentially matrix multiplication at scale, and that maps directly onto GPU parallelism. What a CPU works through step by step, a GPU processes in parallel, an order of magnitude faster.

That is why model weights need a GPU (or a comparable chip like Inferentia) to run at any usable speed, and why GPU instances like Nvidia-based P4 and P5 exist as a distinct EC2 category from general purpose CPU instances like M series and C series.

Delightful answer:

Say the "Good answer" part and then, walk through the full stack.

• Model weights live in S3 and run on EC2 with Nvidia GPUs, or on Inferentia using compiled models for Neuron cores.
  • As we learnt from above, these weights need massive parallel processing. That's why we need Nvidia GPU with CUDA Cores or Inferentia EC2s with Neuron Cores
• vLLM virtualizes access to the model so it can scale under load, the same way a hypervisor virtualizes a bare metal instance into multiple EC2 instances.
• MCP servers run via FastMCP for tools.
• Memory runs on an open-source vector database like Milvus, backed by object storage on a persistent volume.
• Karpenter and horizontal pod autoscaler handle scaling the whole thing.
• This is re-using all the Kubernetes constructs, hence you can apply your organization's security rules, existing observability (such as prometheus, grafana, along with langfuse), RBAC, platform engineering etc.

The balance:

Full self-hosting buys you security and control over your data. It costs you real operational complexity across every layer, from GPU provisioning to vector database management.

You do not have to go all in either way. Plenty of teams mix and match:

• You can run the model in Amazon Bedrock in your own account which does NOT send the data to the model provider. Then self-host the other components
• Self-host the model and agent core for complete control, but use Amazon Bedrock AgentCore for hosting memory, use AgentCore Gateway for MCP etc.

As always, there is no one size fits all. You have to pick based on your organizations requirement, and knowledge base of the team.

Keep learning and keep rocking 🚀,

Raj

P.S - If you want to get an AWS Solutions Architect job without coding or learning every AWS service, the 9th cohort for AWS SA Bootcamp is launching on Aug 1st, 12 PM ET (Eastern Time) via live workshop. This program now includes our updated Gen AI curriculum. Please register below:

Here’s what you get when you show up LIVE:

1. The myths keeping most people stuck - and what actually gets you hired as an SA - I've conducted over 300 SA interviews, so I know what I'm talking about!
2. How GenAI is reshaping the SA role in 2026, and the exact AI concepts (RAG, agents, MCP, eval etc.) you need to speak fluently in interviews.
3. A first look at my new product feature, built to help you practice real-world, interview-relevant hands-on work instead of copy-paste tutorials.
4. Full bootcamp breakdown for Cohort 9, plus a special offer only for live attendees.
5. My exclusive Solutions Architect framework to prep you for today's job market! But if you’re not live, you won’t get it. No second chances.

And good news - it already worked for last cohort's students who secured cloud jobs in top companies, including at AWS, Microsoft, Google, JPMorgan, Reddit, and some of them didn't even have cloud experience 💰.

Spots are limited, so don't miss it!