Hello Reader,

I ran GPT-6 Astra through the exact Solutions Architect interview I used at AWS. Same five rounds, same rubric, same interruptions I give real candidates when they ramble. I wanted to know something specific: not whether it knows AWS, but whether it would pass the bar that actually gets someone an offer.

It passed some rounds cleanly. It failed the ones that matter most. Here is exactly where, and why the gap is not something a smarter model fixes.

Round one: tell me about yourself

Astra gave a textbook answer. Cloud-focused architect, led migrations, designed landing zones, aligned security and product teams, turned fuzzy goals into roadmaps. Correct. Also completely generic. Every word in that answer could describe a thousand different candidates.

Here is what most people miss about this question. I have already read your resume before you walk in. I am not asking you to repeat it. I am asking you to pick two or three specific things that connect directly to the job in front of you, attach numbers to them, and stop talking.

Astra did not do that. Neither do most candidates, and it is one of the most common reasons an interview starts on the wrong foot.

Round two: the technical questions

First question, factual: what is IRSA in Kubernetes. Astra gave the correct definition, IAM roles for service accounts via OIDC, least-privilege permissions without node-wide credentials. Technically accurate, and also the floor, not the ceiling.

The answer that delights an interviewer adds one more sentence: "However, IRSA is being replaced by EKS Pod Identity, which is faster and more secure." That single addition signals you are not just correct, you are current.

Second question, the one I have asked more than any other in seven years: how do you make your application scalable for a high traffic day? Astra gave a reasonable-sounding answer: stateless app, load balancer, auto scaling, offload static content to CloudFront, cache aggressively, decouple with SQS or EventBridge.

The problem is it never asked what "application" means. Is this EC2? Lambda? Kubernetes? The question is deliberately vague, the same way a real stakeholder would ask it, and the correct move is to state your assumption out loud before designing anything.

Astra also recommended decoupling a synchronous system into a queue-based architecture as a fix for a single big traffic day, which is a massive architectural change you would never greenlight for a temporary spike. Everyone knows auto scaling groups. The interview is testing whether you go past that.

Third question, GenAI: how do you reduce hallucinations in an agent? Astra correctly identified grounding, validation, and evals, and when I pushed with a follow-up on what an eval actually is, it gave a solid definition.

Still missing the delight layer: there are three distinct places to run evals, agent-level, LLM-answer-level, and RAG-retrieval-level. Naming all three is what separates knowing the term from understanding the system.

Round three: system design, the one I care about most

This is where the real test happens, because in the real job, nobody hands you a spec. A director says "build me this" and you have to figure out what "this" means before you touch a whiteboard.

I asked it to design an e-commerce website, deliberately vague. Astra did the right first move: it paused and asked clarifying questions before designing anything. Browse, search, cart, checkout, order tracking? Single retailer or multi-seller marketplace? That is exactly what separates a real architect from someone who jumps straight to drawing boxes.

Where it fell short: it nailed functional requirements but skipped non-functional requirements and design priorities entirely.

In a one-hour interview you cannot design for everything. You have to explicitly pick two or three design factors to prioritize, cost, latency, consistency, availability, and defend that choice.

Astra jumped to assumed numbers and a diagram without ever forcing that prioritization conversation. It is a subtle miss, but it is exactly the kind of miss that costs real candidates the offer.

Round four: the behavioral question, and the one it had no chance of passing

Tell me about a time you failed a customer project. Astra's response: "Here's a fictional example you can adapt," followed by a clean, well-structured story about a migration deadline missed due to insufficient scale testing.

Here is the actual problem, and it is not about writing quality. A behavioral question asks what happened to you. Nothing has ever happened to this model.

The structure also broke down against the STAR-L format required for failure stories specifically, situation, task, action, result with data, and lesson.

Astra skipped the data entirely. No real deadline pressure, no specific traffic numbers, no concrete metric on the recovery, and the "lesson" it offered, validate dependencies and scale before promising a date, is generic advice dressed up as a takeaway.

A real lesson sounds more like: "I didn't realize I could cache at multiple layers, not just CloudFront. In the next project, I added caching at the API Gateway layer too, and we handled 9,000 transactions per second." Specific, learned, applied.

The hire or no-hire decision

No Hire. And it is not close, even though Astra technically answered more questions correctly than some human candidates I have interviewed over the years.

The reason is simple. The questions it failed are the ones that test whether you have ever actually been responsible for something. That is not a knowledge gap a better model closes next quarter. It is a structural gap, because the job itself is built around judgment under ambiguity and ownership of outcomes, not retrieval of correct facts.

What this actually means for your own interview prep

Look at where it excelled: facts, definitions, clean answers to clearly scoped technical questions. That is exactly the territory where competing on memorized knowledge alone will not differentiate you anymore, because a model can already produce that answer instantly.

Look at where it struggled: asking clarifying questions before designing, prioritizing trade-offs explicitly, admitting a real gap or mistake, pushing back on a vague premise, and telling a specific, data-backed failure story. Those are the exact places I have watched human candidates lose offers for seven years straight.

The candidates who got hired were never the ones who knew the most facts. They were the ones who asked the right question before answering, caught the trap hiding in a follow-up, and had one real story about something actually going wrong, with real numbers attached.

Do not compete with the model on recall. Compete on the things it structurally cannot fake: ownership, prioritization, and a story that could only have come from something you actually lived through.

Keep learning and keep rocking 🚀,

Raj

P.S. If you have found this newsletter helpful, and want to dive deeper...

Everything I teach on AWS, System Design, Kubernetes, and DevOps - all in one Skool classroom. Build the technical depth at your own pace, and get every new course I release: Join Here​

🎥 Free tutorials on Cloud Gen AI + interview prep on my YouTube channel: Here​

🚀 The SA Bootcamp is the full transformation - hands-on practice through Cloud Colosseum, interview prep, mock interviews, and resume reviews, and so much more...
Everything you need to land the $200k+ Solutions Architect role.

​Join the waitlist to be first in when the next cohort opens.