Hello Reader,

Six months ago, an interviewer asking about Gen AI was a bonus round. Today it is standard. At AWS, Microsoft, Meta, JP Morgan, Verizon, and most large enterprise technology teams, Gen AI questions are showing up in every SA, FDE, and AI engineer interview regardless of the role's primary focus.

The candidates who answer these well are not the ones who have read the most blog posts. They are the ones who can connect the concepts to real architecture decisions and explain the trade-offs without getting lost in jargon.

Here are the questions coming up most often right now and the answers that actually land.

What is the difference between CPU and GPU?

The average answer

CPU is good for general purpose applications, and GPU is good for running AI ML workloads

Why average?

This answer doesn't highlight the superpowers considering this is a Gen AI interview question. Think of it like this - if interviewer asks you "What is the difference between EC2 and Lambda?", and you answer EC2 is servers/Virtual Machines on the cloud, and Lambda is Serverless, that won't be a good answer.

Good answer

A CPU or Central Processing Unit has a small number of powerful cores optimized for sequential, general-purpose tasks such as application logic, branching, database operations. A GPU or Graphical Processing Unit has thousands of simpler cores optimized for parallel workloads where you're doing the same operation on massive amounts of data simultaneously.

​
Gen AI inference is basically matrix multiplication at scale, which maps perfectly to GPU parallelism. A CPU would work through that sequentially; a GPU processes it in parallel and is orders of magnitude faster. That's why LLM training, and inference is best in GPU because that's essentially a lot of parallel processing.

Then add an AWS example to delight the interviewer - some examples of CPU EC2 instance families are M, C series, and sample GPU instance families are P4, P5.

How will you cost optimize Gen AI workflow and application?

This question is getting more popular fast, driven largely by companies worried about sending proprietary data to a model provider.

The average answer

Some of the average answers I hear are:

• I will optimize prompts
• I will use cheaper models
• I will use cache
• I will reduce usage

Why average?

• Vague answers without showing architect level thinking. For example - you simply can't use cheaper models sacrificing quality
• There are different caches, need to specify to show SA depth
• These are just basic techniques, there are intermediate and advanced techniques you must know

Good answer

Understand and memorize 3-5 from the below to delight the interviewer, as well as optimize cost in real-world projects.

• Use right model for the right task. For example, for summarization you don't need to use the most powerful model, Haiku is good enough. However, for each task, use Eval to find out if model is underperforming. We can also use intelligent prompt routing which can automatically route requests to different models based on prompts
• The biggest cost factor is LLM processing the tokens. Caching helps reduce that. There are two main types of caching in Gen AI
  • Prompt Caching - this is offered by the model providers itself. If the first part of the prompt ( includes system prompt, tools, or documents) matches completely with a previously processed prompt, then LLM provider sends the cached answer
  • Semantic Caching - Whereas prompt caching needs exact match of the prompt, semantic caching can understand the underlying intent of the prompts, and if the intent of current prompt is same as a previous one, it sends the cached answer. Example: A user asking "How do I reset my password?" and a second user asking "I forgot my login credentials, what do I do?" receive the exact same cached reply without triggering a new AI generation. Semantic caching is implemented by the application. Example - Redis Caching
• Remove unnecessary MCP servers. Every connected MCP server loads all its tool definitions into context on every message. If you type a two-word prompt and wonder why the token count is already high, MCP overhead is the reason.
• This one is growing (study this on it's own for separate questions) - Utilize memory. Without memory, the model rediscovers the same context every session. With memory, the model retains summaries, preferences, and prior decisions so you pick up where you left off.
• Vector database cost sneaks up on you. OpenSearch Serverless has low latency but charges for reserved capacity even when idle. Aurora PostgreSQL with pgvector is a strong middle ground if you are already running a SQL database. The S3 vector database option is the most cost-effective for batch processing and cost-sensitive production RAG workloads, with slightly higher latency that is often within acceptable SLA ranges. Test your specific workload before committing.
• Remove unnecessary and stale data from vector databases
• Standard cloud cost practices apply to GenAI workloads the same way they apply to everything else. Enterprise discounts, reserved capacity on Bedrock, spot instances for EC2 inference, right-sizing, scaling inference endpoints to zero when idle, batch inference, and cost allocation tags by team and project.

What is an AI agent?

The average answer

Agents can perform complex tasks with prompts. Then mention random Gen AI terms like MCP, LLM, Tool.

Why average?

This answer doesn't have the depth. And you need to correlate the terms in regards to AI Agent. Think of it like this - if interviewer asks you "What is a microservice?", and you just say the name of AWS services like EC2, Lambda, ALB, APi Gateway, that won't be a good answer

Good answer

An AI agent is a software process that autonomously and independently makes decisions and takes actions until task completion. Standalone large language model cannot take action and cannot go back and forth till the task is done.

Then you mention an example. So let me take an example. Let's say I have AI agent for troubleshooting. I can send a prompt like identify issues in my AWS application and fix them. And to fix this, the agent may need to go to different places, logs of the application, infrastructure or database, some metrics, etc.

And it happens using agentic code interacting with large language model and a bunch of tools. And you don't have to code or specify how many times a tool needs to be called. It's all happening automatically. And once the issue is found, this agentic code using a tool can also go fix it.

Now regarding the components of AI agent which ties to the next question. So you can add part from the next answer to this answer as well to impress the interviewer.

What are the components of AI Agent (Also asked as : Give me a sample AI Agent Architecture)

AI agent has the following components:

• The large language model (LLM) and that could be hosted in something like Amazon bedrock.
• Then we have the ever important agentic framework or sometimes called agent harness which is the actual code. It could be written in something like strands agent or langchain, langraph with your language of choice and this will run in Amazon Bedrock AgentCore runtime.
• This code needs access to tools and that will be done via MCP. One practical implementation of this is Amazon AgentCore Gateway.
• Say this part to delight the interviewer - Now all modern agents give you the capability of resuming your session and that is done via memory. So this strand agent code has also access to short-term memory and then a memory extraction modules periodically extracts information and puts it into long-term memory and all these could be implemented using AgentCore memory.
• In addition, we will implement security and observability to ensure AI agent is acting appropriately

So this question and the previous question is kind of overlapping so you can use them accordingly.

If you get these questions, make sure to crush it! I will cover more Gen AI questions in future editions.

Keep learning and keep rocking 🚀,

Raj

P.S - If you want to get an AWS Solutions Architect job without coding or learning every AWS service, the 9th cohort for AWS SA Bootcamp is launching on Aug 1st, 12 PM ET (Eastern Time) via live workshop. This program now includes our updated GenAI curriculum. Please register below:

Here’s what you get when you show up LIVE:

1. The myths keeping most people stuck - and what actually gets you hired as an SA - I've conducted over 300 SA interviews, so I know what I'm talking about!
2. How GenAI is reshaping the SA role in 2026, and the exact AI concepts (RAG, agents, MCP, eval etc.) you need to speak fluently in interviews.
3. A first look at my new product feature, built to help you practice real-world, interview-relevant hands-on work instead of copy-paste tutorials.
4. Full bootcamp breakdown for Cohort 9, plus a special offer only for live attendees.
5. My exclusive Solutions Architect never released before tool to prep you for today's job market! But if you’re not live, you won’t get it. No second chances.

And good news - it already worked for last cohort's students who secured cloud jobs in top companies, including at AWS, Microsoft, Google, JPMorgan, Reddit, and some of them didn't even have cloud experience 💰.

Spots are limited, so don't miss it!