Auditioning Engineering: Software Engineering Interviews in the Age of AI
By Srivathsava Rangarajan
For the better part of three decades now, software engineers have endured the initiation ritual that is the software engineering interview process. You implement an algorithm, invert a tree, find the shortest route through a network, all while an engineer watches. It’s very reminiscent of agentic coding. In fact, it is precisely the thing large language models are designed for. It’s a puzzle with a well-defined prompt, a “just-gotta-know-it” trick with a single right answer.
AI didn’t make interviewing worse. Software engineering interviewing was bad to begin with, and AI just blew it out of the water. So instead of policing AI use in interviews, we told candidates to use it, and we built the interview around what’s left when the tool has done its part: where does the engineering go when the AI generates the code?
What’s left?
The answer to this question seems so obvious that one wonders if there is a trick being missed. The engineering doesn’t go anywhere, it just remains where it has always been, in the art of translating intent into software.
We have been “generating” code for decades. The age of hand-crafting every machine instruction ended with compiled languages in the 50s; IDEs added predictive auto-completion (glance at a Java programmer’s keyboard and you’ll notice a well-worn Tab key); the Internet and Stack Overflow added Ctrl, C and V. LLMs are just the latest tool in that lineage.
At WMG we believe software engineering is an art. Software engineering is an art for the same reason there are thousands of covers of the same song: style. Audiences have tastes, organizations have culture. Engineers translate intent into action, we don’t transliterate it, and in the interpretation we find room for expressing organizational culture through personal technical judgement. Python or Go? Monolith or services? Simplicity or completeness?
The prevailing interview process placed an overwhelming emphasis on the technique and not enough on the art. So we asked a better question: how do we let AI handle the execution, and test the art of engineering?
The art of coding
We built our coding interview from the areas of our codebases with the most churn: routine engineering problems peppered with some interesting challenges, such as HTTP-based API interactions including network failures, “unmodifiable” legacy endpoints with non-standard designs and no idempotency, and joining and enriching data across systems.
The candidate implements a client to spec, while a live server acts as the testbench and provides feedback through a periodically logged score. The workload is randomized and the score resets on every restart, which simulates the rapidly iterative nature of everyday engineering without allowing for jerry-rigged solutions. The workload is calibrated to surface increasingly complicated problems, so performance should largely correlate with experience. The best way to describe it is learning to play a song by ear, blindfolded: the feedback doesn’t tell you which notes are wrong, just whether it sounds right. There’s also no single way to play it right, but it is easy to tell when it sounds wrong. The artform is in finding your own way to blend organizational culture with your technical style.
Working SDKs are included as starter code instead of a blank canvas, and internal dependencies are modeled as well-documented, pre-installed modules, since realistically that discovery would take days of collaboration, code comprehension and testing.
Get WMG Lab’s stories in your inbox
Join Medium for free to get updates from this writer.
LLMs are a ubiquitous part of our development lifecycle, so we treat them as a precondition of this audition and encourage their use. The trick to not trivializing a coding interview while encouraging LLM usage is to not distill the question down in the name of formalization, for then it becomes a prompt. The ability to artfully distill and state a problem formally is an important place engineering craft lives on even when AI generates the code.
To calibrate the difficulty, we ground-tested the interview internally with dozens of volunteers of varying seniority, and an internal leaderboard saw quite a few engineers join in for the fun of it.
The art of designing systems
Our system design interview was inspired by sampling the problems that show up most in our internal RFC and incident reviews: data modeling and distributed systems problems, with concerns like retries, transactionality and locking, exactly-once semantics, idempotency and race conditions, and referential integrity. The key was recasting the interview from vague problem statements like “design a payments app” into detail-oriented discussions of the mechanical implications of design choices.
It is a collaborative editing exercise on an existing system, expressed in pseudo-code, to realize new features, improvements and bug fixes. Candidates must first build a mental model of the system and then propose design alterations to it, which the interviewer notates as changes to the pseudo-code, like an AI agent would. Candidates can ask to “deploy” a change and get back observed behavior in domain language, without identifying or tracing causality: “Users are still reporting 429s, although the volume seems to be lower than before,” rather than “the lock on line 42 helps, but without an expiry there’s a chance of starvation.” More complex designs take longer to notate and to “deploy”, which introduces a natural cost to the candidate’s design choices.
We use pseudo-code instead of formal code, since coding skill is already under test in the coding interview, and we limit tool choices to our organization’s standard infrastructure. Overall, this exercise is akin to composing music as a band. You write the chart around your band’s sound, test your limits with something new, and listen closely to what vibes. Designing in that tension is art.
Without a stack trace or a score, candidates have to rapidly build and iterate mental models of the system and correlate execution to code entirely by feel and imagination. This ability to mentally simulate systems is another place where engineering craft continues to reside even as AI generates the code.
What we’ve learned
Feedback volunteered by candidates has been overwhelmingly positive. Candidates felt the experience to be more engaging than trying, and also felt like they did their best when they brought their most authentic engineering selves to the interview as opposed to “studying” to pass it.
The coding interview has been easy to train interviewers on and proctor, since it is mostly hands-off: the interviewer’s main task is logistics and taking notes on AI usage. The system design interview is intense to proctor, and a limited pool of senior engineers is trained to run it, which makes scheduling harder. Making new coding interviews has also been challenging, because each needs a new server with a scoring system, and new iterations tend to feel more like reskins than distinct challenges; new system design interviews are significantly easier, thanks to the standardized solution framework and an abundance of source material.
When WMG launched its new interview process in late 2024, our best AI coding agents could barely clear our junior engineer bar. Today they comfortably clear our staff engineer bar (although, still lose to our best engineers on the leaderboard). We expect this space to remain a work in progress given the pace of LLM development. But until model providers start underwriting their products’ work, engineers remain accountable for the code shipped, irrespective of who or what wrote it. Technical judgement remains the most valuable commodity in the industry and the north star of our interview process. What we’re doing is not necessarily perfect, but it is different, and we believe it is better.
If you’ve run an interview loop through the past few years, what held up?
