Your customer has clicked Pay twice, and you are staring at a Kubernetes dashboard that seems delighted with itself. Every pod is Running, readiness checks are green, and CPU usage looks ordinary. Meanwhile, checkout still fails, which makes all that reassuring green feel a little less reassuring.

Honestly, this is the kind of debugging exercise I want in my practice notes because it exposes a very tempting assumption. A process can answer its health check while the operation a customer needs is broken.

Let’s work through a hypothetical checkout failure and use one request to decide where to look next. By the end, you should have a small investigation you can explain in an interview or adapt to your own application.

Start with the thing your customer cannot finish

Before opening another dashboard, turn the complaint into something you can investigate. Can the customer load checkout but fail when submitting payment? Does every payment method fail? Does the error appear immediately or after a long wait?

Suppose the page loads normally, but submitting a card payment returns a 502 after roughly three seconds. Another payment method succeeds. These observations are invented for our walkthrough, but notice how much they narrow the search compared with a vague report that checkout is broken.

Take the failed request ID and compare its route, timestamp, response code, and downstream spans with a successful checkout from the same period. Suppose the failing trace reaches the checkout application, then spends nearly all its time waiting for one payment provider.

Now we have a specific branch to investigate. We still need to distinguish a provider problem from network access, credentials, or client configuration. A slow span tells us where time went; it does not explain why by itself.

Ask what that green check actually measured

Readiness helps decide whether a pod receives normal Service traffic. Liveness can trigger a container restart. Neither check automatically exercises a full checkout, so a lightweight health endpoint can succeed while the payment path fails.

In our example, that means a successful health check and a failed payment can both be accurate observations. We need the request path to connect them.

Check whether the Service selects the intended pods and whether its EndpointSlices contain the expected backends. Kubernetes’ official Service debugging guide explains how to inspect selectors, endpoints, and ports. Also compare the Service’s target port with the port the application actually serves.

These read-only commands provide a starting point. Replace the placeholders with your namespace and Service name, and keep the failing request’s timestamp nearby.

kubectl get pods -n <namespace> -o wide
kubectl describe service <service> -n <namespace>
kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service>
kubectl get events -n <namespace>

Compare the endpoint addresses with the expected pod IPs and inspect their readiness conditions. Events can help explain scheduling or container problems, but they are not an HTTP request history. To establish which pod handled checkout, use the application logs or traces that actually record that request.

Make each check shrink the search area

I prefer a sequence that leaves me with a smaller set of possibilities after each step. Otherwise, debugging can turn into an impressive collection of terminal tabs with very little progress.

1. Confirm the failure at the edge. Use an existing trace or reproduce it safely in a test environment. Record the route, response code, latency, and request ID. For payments, avoid creating real charges just to reproduce an error.
2. Follow routing to the intended workload. Check the gateway or Ingress route, then the Service selector, ports, and EndpointSlices. The existence of healthy pods does not establish that traffic reaches the right ones.
3. Inspect the pod that handled the request. Compare its logs, version, restart history, and resource pressure around the failure. If a container restarted, previous-container logs may hold a clue that current logs no longer show.
4. Follow its outbound calls. Check timing, errors, and retry counts for the relevant database, queue, identity provider, or payment API. Use the failing span to choose the next inspection.
5. Compare a success with a failure. Look for the earliest meaningful difference in route, version, configuration, dependency, or input. Similar successful requests are especially useful because they reduce the number of differences you have to explain.

For our checkout, imagine that both requests reach the intended workload, but only the failing one calls the card provider. That observation moves the investigation toward that integration. If the failures also began after a deployment, compare the provider endpoint, credentials, timeout settings, and network configuration between the working and failing versions.

A different pattern should change the plan. If failures occur only on one replica, inspect that replica’s configuration and connections before blaming a provider shared by every pod. That is the habit worth practising, because evidence should choose the next command.

During a live incident, preserve request IDs and relevant logs when it is quick to do so, but avoid delaying an approved mitigation to collect a perfect evidence bundle. Restoring checkout matters more than finishing the incident detective story.

Fix the signal without creating a restart party

Kubernetes defines startup, readiness, and liveness probes separately. Startup probes give initialization time before the other probes begin. Readiness failures make a pod unready for normal Service routing; liveness failures can restart the container after the configured threshold.

Before changing a probe, decide what action its failure should justify. Restarting every checkout pod during a shared payment-provider outage could add disruption while leaving the original problem untouched. Making readiness depend on that same provider could also remove otherwise useful capacity across the fleet.

For this exercise, add a separate checkout-success signal and, where appropriate, a safe synthetic journey. Track outcomes by payment method so an overall success rate does not hide the broken branch. A test-mode checkout can reveal integration problems, although it still cannot reproduce every production condition.

Let an AI helper earn its next suggestion

An AI assistant can help organise the evidence, but I want more from it than a confident paragraph naming a likely cause. Give it the failed request, a comparable success, and the configuration details you can safely share. Then ask which observation supports its explanation and what would make it change its mind.

Try this first in a controlled staging incident with read only access. Ask for the next diagnostic, what its possible results would mean, and which result would rule out the current theory. Keep production changes such as restarts, scaling, and configuration updates subject to your team’s review process.

That gives you a useful investigation partner while keeping responsibility for changes clear. A suggestion becomes much easier to evaluate when you can see what evidence it depends on.

Practice the explanation before memorising more commands

Close the terminal for a moment and explain our example to a teammate. Checkout loads, card payments fail after a delay, requests reach the intended workload, and the failing branch waits on one provider. Your next step is to distinguish the integration’s possible failure modes using the trace and configuration evidence.

Notice how much more useful that is than reporting that every pod is green. Someone else can pick up the investigation without retracing every command you ran.

For a focused practice run, try this Kubernetes pod troubleshooting question and explain what each check would change about your next step. If you cannot say why the result matters, pause before adding another command.

Which clue would you check first if checkout failed on only one payment method while every pod stayed Ready?

I’m curious whether you’d start with the request trace, a recent deployment, or something else entirely.

~ Emily

A practical way to budget time, bound retries, and keep optional work from blocking the core request.

Your API can be healthy on its dashboard while the page in front of a customer is still spinning. One dependency has started taking two seconds to answer, and every request that waits for it is holding a connection, a worker, or a place in a queue.

That is the little design problem I want to practise today. Not how to add another box to the architecture diagram, but how to stop one slow box from borrowing resources from everything around it.

Imagine a product page that needs the product record and also asks a recommendation service for optional suggestions. Product details are the promise the page has to keep. Recommendations are useful, but the page can still work without them. If both calls share one unbounded wait, a slowdown in recommendations can make the whole page feel broken.

This is a hypothetical example, not a report of a real outage. The useful part is the way we reason through it, one request at a time.

First, trace one request instead of drawing a bigger system

Start at the user action and follow the work until the response comes back. For this page, the path might be browser → API → product database, with a second call from the API to recommendations.

Now ask which parts are required for a useful response. The product name, price, and availability may be essential. Recommendations may not be. That distinction changes the design more than choosing between a queue and a cache too early.

Next, follow the waiting. If the API accepts 120 requests per second and each request can sit for two seconds, Little’s Law gives us about 240 requests in flight during steady state, before counting retries or other overhead. That number is only an illustrative estimate, but it makes the risk visible: slow downstream work can occupy upstream capacity even when the API’s own code is doing very little.

When the dependency gets slower, more requests remain in flight. If the server runs out of workers or connections, otherwise healthy requests start waiting too. Google’s SRE guidance describes this kind of positive feedback as a common shape of cascading failure: local overload pushes more load onto the remaining healthy capacity.

The first useful interview question is therefore not which component we should add. It is where requests wait, what each wait consumes, and which work can be skipped without breaking the user’s main task.

Give the whole request a deadline

Timeouts work best when they fit inside an end-to-end budget. Setting every downstream call to wait for five seconds does not make a five-second user experience acceptable; it just hides the time being spent at each hop.

Suppose the product page has an illustrative total budget of 900 milliseconds. We might reserve time for the API, the essential product lookup, optional recommendations, and the response back to the browser. The exact numbers depend on the product and its latency goal, so they should come from an explicit budget rather than habit.

Each child call should receive only the time that remains in the parent deadline. If the product lookup already used most of the budget, the API should not start an optional recommendation call that cannot finish in time. Likewise, it should stop work when the caller has disconnected, where the stack supports cancellation.

Timeouts are not simply a way to make errors happen faster. They bound how long scarce resources remain tied up. Set them too generously and a growing queue can consume the service; set them too aggressively and normal network variation may turn healthy requests into failures. The right value is something to measure and revisit, not copy from a blog post.

A retry is extra traffic, so make it earn its place

Retries can help when a failure is brief, but they can also send more work to a service that is already struggling. Before retrying, ask whether the operation is safe to repeat, whether the error is likely to be temporary, and whether enough of the request deadline remains to make another attempt useful.

For a read-only lookup, one carefully bounded retry may be reasonable. For a payment or other state-changing operation, the caller needs an idempotency strategy before retrying can be considered safe. A timeout does not prove the original operation never happened.

If a retry is justified, use backoff with jitter and a maximum attempt count or elapsed-time limit. Avoid retrying at every layer, because three layers each making three attempts can multiply the work reaching the dependency. AWS’s guidance on timeouts, retries, and backoff with jitter explains why retries need limits and why synchronized clients can create a fresh traffic spike.

Put a fence around the waiting room

Timeouts limit how long one request waits. They do not automatically cap how many requests wait at once. A system can still accumulate a large queue before its timers fire, so I would also ask how many concurrent calls and queued requests each dependency is allowed to consume.

That is where bounded queues, concurrency limits, and bulkheads help. A bulkhead keeps one kind of work from using every worker or connection needed by other work. A circuit breaker can temporarily stop calls after repeated failures, then allow a small number of probes to check whether the dependency has recovered.

None of those controls repairs the dependency. Their job is to keep the failure contained while the system chooses a sensible response. For example, a product page may return product details without recommendations. A checkout may need to fail clearly if a required inventory check cannot complete, rather than quietly pretending stock is available.

That distinction is important because graceful degradation should preserve a truthful result. Optional content can disappear for a while; a critical decision should not be guessed just to keep the page green.

Make the fallback part of the product decision

Engineers sometimes treat the fallback as a technical detail that can be picked after the architecture is finished. In practice, it changes what the user sees, and that choice belongs alongside the product requirement.

For each dependency, write down what the customer should receive when it is slow, unavailable, or returning questionable data. Maybe the API serves a cached recommendation set with a clear freshness limit. Maybe it omits recommendations entirely. Maybe a critical endpoint returns a retryable error and explains what the user can do next.

Then check the trade-off. A stale value can be better than no value for one product and dangerous for another. A fallback should have an owner, a freshness rule, and a way to tell that it is being used. Otherwise, a temporary path can silently become the normal one.

Test the slow case, not only the red case

The easiest failure to imagine is a dependency that returns an error. Slow responses are often more revealing because they keep connections and workers occupied while the dependency technically remains reachable.

I’d test a few concrete situations in a staging environment or a controlled fault-injection test. First, delay the dependency just beyond its normal latency and see whether the caller respects its deadline. Next, return transient errors and confirm the retry budget stays bounded. Then stop the dependency and check whether unrelated endpoints still have capacity. Finally, restore it and look for a retry burst or a queue that takes too long to drain.

Useful signals include dependency latency percentiles, timeout rate, retry rate, in-flight requests, queue depth, circuit-breaker state, and the success rate of the customer-facing operation. A dependency’s health check alone will not tell you whether users can complete the task that depends on it.

Google’s chapter on addressing cascading failures is a useful deeper read because it connects overload, resource exhaustion, and failure spreading across components.

A compact way to explain the design

When this comes up in a system-design conversation, I’d keep the explanation anchored to one request. Clarify the user-visible promise, trace each call, separate required work from optional work, and set a deadline the entire path can respect. Then explain which requests may retry, how waiting capacity is bounded, what the user sees when a dependency stays slow, and which metrics would reveal the failure before it spreads.

That sequence gives the interviewer something concrete to challenge. They can make the dependency slower, ask what happens after a retry, or change the endpoint from optional content to a critical write. Each follow-up tests whether the design still matches the product promise.

If you want another practice prompt with the same kind of failure reasoning, try designing device monitoring over an unreliable public network. It asks you to reason about late and repeated messages, unreachable devices, and what operators should see when the network goes quiet.

The next time a diagram looks perfectly healthy, follow one request through its waits. That is often where the real design starts.

~ Emily

You open an orders page with 20 rows. The endpoint returns the right customers, the response looks perfectly normal, and nothing throws an error. Then you check the query log and see 21 database calls.

One query loaded the orders while the other 20 fetched a customer for each order.

That pattern is called an N+1 query problem, and it is easy to miss because the code that triggers it often sits inside a serializer, template, or property rather than next to the query that loaded the list. Let’s trace a small example, fix the relationship load, and make sure the fix stays useful as the response grows.

First, follow the data all the way to the response

Imagine an Order has a foreign key to Customer. The view loads a page of orders, then the serializer includes the customer’s name:

orders = Order.objects.filter(status="open")[:20]
payload = [
    {"id": order.id, "customer": order.customer.name}
    for order in orders
]

The first line looks like one read. But unless the related customer was loaded already, each access to order.customer can trigger another query. That makes the query count grow with the number of orders: one list query plus one relationship lookup per order.

In an interview, I’d make the hidden work visible before suggesting a fix. I’d trace the complete response path, including serializer fields, template properties, and helper methods. Then I’d inspect the actual SQL or query log for repeated statements that differ only by customer ID. Counting only the view’s first queryset can miss the expensive part entirely.

Match the loading strategy to the relationship

For a single-valued foreign-key relationship such as Order.customer, Django’s select_related loads the related row through a SQL join. The example becomes:

orders = (
    Order.objects
    .filter(status="open")
    .select_related("customer")
    [:20]
)

Now the serializer can read order.customer.name without making a separate customer query for every order. The database still returns the order and customer data, but the application asks for them together.

The important detail is that the relationship is single-valued. If an order has many line items, joining every collection into one large result can repeat the order columns for each line item. For a collection, prefetch_related("items") is often a better starting point. Django performs a separate batched lookup and associates the items with their orders in memory. SQLAlchemy offers similar choices with joined and select-in loading; the right option depends on the relationship and the shape of the result.

Don’t optimize the query counter in isolation

It is tempting to hear 21 queries and decide that one query must always be the goal. That can replace one problem with another. Joining several collection relationships may multiply result rows, duplicate parent data, consume more memory, and make pagination harder to reason about. A very broad eager load can fetch columns or relationships the response never uses.

I’d choose the loading plan by asking what the response actually needs. For each relationship, check whether it is one-to-one or many valued, whether the data is used for every returned row, and how many child records might exist. Then compare the query count with the rows and payload returned. A batched query that avoids hundreds of round trips can be a good trade, even if the request still uses more than one query.

Also look at how the relation is accessed. A prefetch cache only helps when the code uses the prefetched relation in a compatible way. Calling a new filtered query for each order can bypass the prefetched collection and bring the repeated work back. Query inspection should include the actual serializer or template behavior, not just the model declaration.

A small test can catch the regression

I’d write a focused test around the endpoint or serialization path and vary the number of orders while keeping the request shape the same. Try one, 20, and 100 rows, then check whether relationship queries stay bounded instead of increasing once per order. The exact total can vary because authentication, pagination counts, middleware, and other request work may add queries, so the useful signal is the slope of the repeated relationship loads.

For a Django unit test, assertNumQueries can make a narrow query expectation explicit. Keep that assertion close to the code path it protects, and avoid pinning a brittle endpoint-wide total if unrelated request behavior changes frequently. Pair the count check with response assertions so the optimization does not silently omit customer data or change ordering.

I’d also cover the cases that make a technically fast response wrong. Check orders with no optional related record, tenant or permission filtering, empty pages, and the exact fields the response promises. If the database enforces foreign keys, a required relation may behave differently from an optional one, but authorization still belongs in the query or policy layer; eager loading is not a substitute for access control.

The interview answer I’d keep in my back pocket

The concise version is that I would trace query execution through the full response path, identify the repeated lazy relationship lookup, and choose eager loading based on cardinality. I’d use a join for a single-valued relation or a batched prefetch for a collection, then verify both response correctness and query growth at different result sizes. I’d also check row multiplication, payload size, and memory before calling the fix complete.

That answer shows more than familiarity with select_related. It shows that you can find work hidden outside the view, choose a strategy based on data shape, and explain how you would know whether the trade-off helped.

The habit transfers nicely to other backend performance questions too, because the useful move is usually to trace the work the request actually performs rather than optimize the line that looks suspicious. If you want another practice prompt about scaling a service and making schema, indexing, and search trade-offs, try this related backend exercise.

For reference, Django’s select_related and prefetch_related documentation explains the single-valued join and batched-prefetch distinction, while SQLAlchemy’s relationship loading guide describes lazy loads and alternatives such as joined and select-in loading.

A saveable debugging checklist

• Trace from the list query through serializers, templates, and properties.
• Look for repeated SQL that changes only the related row’s ID.
• Choose a join for single-valued relations and a batch load for collections.
• Check that access patterns actually use the loaded relationship data.
• Compare query growth at 1, 20, and 100 results, holding the request path fixed.
• Validate response fields, empty relations, permissions, row counts, and memory cost.

If the endpoint still returns the same useful data while relationship queries stop growing one-for-one with the result list, you have fixed the hidden repetition and kept the trade-off visible.

~ Emily

Your script finishes on the sample file and now your output looks sensible, and you mentally move on to the next task.

But then someone sends the actual export.

Suddenly, the useful little script is competing with everything else on your laptop for memory. You add a chunk size, try again, and discover that Python can still run out of room while politely processing one chunk at a time.

Honestly, this is my favorite part of the problem to unpack, because the first answer sounds so reassuring. Read the file in smaller pieces and everything should be fine.

Except reading less at once does not help enough if you keep everything afterward.

Let’s build a small CSV pipeline around that distinction. By the end, you’ll have a practical approach for an oversized export and a much stronger answer when an interviewer changes the memory limit.

Before choosing a library, decide what survives each row

Imagine an orders export with four columns, and a task that sounds pleasantly boring. Keep paid orders, retain their IDs and amounts, and write one JSON object per line.

Nothing in that result requires remembering an earlier order. There is no global sort, no deduplication, and no running dictionary of customers.

That detail gives us permission to release each parsed row after writing its result. If someone adds a requirement to remove duplicate order IDs, we have a different memory problem, even though the input file hasn’t changed.

I prefer to sketch three places where data can accumulate before touching the implementation.

• Input includes parser buffers and the current row or chunk.
• Retained state includes anything kept between iterations, such as a set of IDs or a list of results.
• Output includes serialization and anything waiting to be written.

Those three buckets sit on top of the interpreter, libraries, and other process overhead. So a 512 MB process limit does not mean we can comfortably choose a 512 MB chunk.

For this task, the goal is to keep the working set tied to a bounded record size, rather than the total number of records. We still need enough disk space for the output, which is a separate budget.

Give the tiny script a clear contract

Let’s keep the exercise small enough to explain without turning it into a data platform.

• Input is a stable, local UTF-8 CSV with exactly order_id,status,amount_cents,note as its header.
• Each record has four fields. IDs must be nonempty, status must be paid, pending, or refunded, and amounts are nonnegative ASCII integers of at most 12 digits.
• Fields are limited to 100,000 characters. We assume a trusted producer also bounds total record size and field count; this is not a parser for arbitrary hostile uploads.
• Every valid paid record produces an output object containing only its ID and integer amount. Other valid statuses are skipped, input order is preserved, and duplicate IDs remain duplicates.
• Malformed input stops the run. A partially written file must not be presented as a completed export.

That last point is worth agreeing on early. Silently skipping bad rows can make a job look successful while quietly changing the answer.

We also need a real CSV parser. A note can contain a comma or a quoted newline, so neither splitting on commas nor treating every physical line as a complete record is enough. Python’s CSV documentation explains the parser behavior and recommends opening file objects with newline=''.

Let one row travel all the way to disk

For our example, order A17 has status paid and an amount of 1250 cents. Its note can disappear because the output does not need it. Order A18, with status pending, produces no output record.

Here’s a reference implementation for that contract. It uses the standard library and writes to a new partial file before promoting the finished result.

import csv
import json
from pathlib import Path
HEADER = ['order_id', 'status', 'amount_cents', 'note']
STATUSES = {'paid', 'pending', 'refunded'}
def export_paid(source, destination):
    source = Path(source)
    destination = Path(destination)
    partial = destination.with_name(destination.name + '.partial')
    if source.resolve() in {destination.resolve(), partial.resolve()}:
        raise ValueError('Input and output paths must differ')
    if destination.exists():
        raise FileExistsError(destination)
    old_limit = csv.field_size_limit(100_000)
    try:
        with source.open(encoding='utf-8', newline='') as src:
            reader = csv.reader(src, strict=True)
            if next(reader, None) != HEADER:
                raise ValueError('Unexpected or missing header')
            with partial.open('x', encoding='utf-8', newline='\n') as out:
                for record_no, row in enumerate(reader, start=2):
                    if len(row) != 4:
                        raise ValueError(f'Wrong field count at record {record_no}')
                    order_id, status, amount, _note = row
                    valid_amount = (
                        1 <= len(amount) <= 12
                        and amount.isascii()
                        and amount.isdigit()
                    )
                    if not order_id or status not in STATUSES or not valid_amount:
                        raise ValueError(f'Invalid values at record {record_no}')
                    if status == 'paid':
                        result = {'order_id': order_id, 'amount_cents': int(amount)}
                        out.write(json.dumps(result, ensure_ascii=False) + '\n')
        partial.replace(destination)
    finally:
        csv.field_size_limit(old_limit)
export_paid('orders.csv', 'paid-orders.jsonl')

Notice what never appears here. We don’t call list(reader), append every result to a list, or wait until the end to serialize the entire export.

Each iteration replaces the previous row. The output file uses ordinary bounded buffering, and JSON serialization briefly creates a string for the current result. With the stated record size assumptions, memory does not need to grow in proportion to the file’s row count.

The record number in an error is a logical record number, with the header counted as record one. It is not necessarily the physical line number, because quoted fields can span lines.

One practical boundary matters here, too. This example assumes one process owns these output paths. Checking whether a destination exists is not a concurrency lock, and replace can overwrite a destination created by another process in the meantime. Concurrent jobs need unique run paths and a coordinated completion step. CSV field size setting is also process-wide, so don’t change it casually inside competing threads.

Chunking is useful until the chunks start collecting

Maybe pandas is already the right tool for your transformation. Wonderful, keep it. Just inspect what happens between reading a chunk and asking for the next one.

This pattern is the trap.

pieces = []
for chunk in pd.read_csv('orders.csv', chunksize=50_000):
    pieces.append(transform(chunk))
result = pd.concat(pieces)

Assuming the transformed chunks remain substantial, the list keeps their data alive. Concatenating them may require additional allocation. We’ve delayed the big memory bill instead of removing it.

For an independent per-row transformation, write each processed chunk to the output and release it before continuing. Read only the columns you need, choose suitable types, and leave room for temporary arrays created by the transformation. The pandas scaling guide covers these options.

Also, 50,000 rows is a row count, not a memory guarantee. Fifty thousand short numeric records and fifty thousand long text records are very different guests at the same party.

Measure peak process memory on representative data, including unusually wide records and the actual transformation. A DataFrame’s own memory estimate is useful, but it doesn’t describe every allocation in the process.

When the follow-up changes, change the plan

Now imagine the interviewer asks for total spend per customer.

A dictionary feels natural, and it may be perfectly reasonable when the number of distinct customers fits the budget. But its size follows distinct keys, which could grow almost as fast as the file.

We can still read sequentially while using too much memory. The input path is streaming; the retained state is growing.

If that state cannot fit, move the aggregation into a disk-backed database or a processing engine that can spill intermediate work. A hand built partitioning approach needs bounded partitions and a plan for skew, because one oversized partition can bring the same problem back.

A global sort changes the task again. Sorted chunks alone are not a globally sorted result; an external sorting plan needs sorted runs, a merge, and enough temporary disk space.

You don’t need to bring distributed computing into every large file conversation. Start with the required result, estimate the state it demands, and choose the smallest approach that fits the actual limits.

Make the answer checkable before making it bigger

Before scaling this example, I want a tiny fixture whose expected output I can explain without trusting the script. These are the cases to include.

1. Mix paid and pending orders. Only paid records should appear, in their original order.
2. Put a comma and a quoted newline inside a note. Each should remain part of one parsed field and must not create extra output records.
3. Include a paid order with zero cents. Zero is valid under our contract, so it must survive.
4. Add a duplicate paid ID. Both records should remain because deduplication was never requested.
5. Try an invalid amount, a wrong field count, and a field over the size limit. Each should fail the run instead of yielding a supposedly complete destination.
6. Use a header only file. It should produce an empty completed output like a completely empty file should fail the header check.

Those are proposed checks for this reference implementation than just reported test results or a benchmark.

Next, increase the amount of representative data while tracking elapsed time, peak memory, output size, and whether the output remains correct. If memory climbs with every batch, look for references that keep previous data alive before reaching for a larger machine.

Leave the next person a trustworthy output

Our partial file is deliberately boring. If processing fails halfway through, the final destination is never created by this run. Inspect the error and the leftover partial file, then remove or archive that partial file before restarting from the beginning with the same unchanged source.

That is a simple restart policy, not resumable processing. Renaming after success also isn’t a promise of crash durability or exactly once delivery to another system.

For very expensive jobs, design checkpoints around stable input identity and valid parser boundaries, with a clear policy for already-written output. A random byte offset can land inside a quoted CSV record or a multibyte character, so saving the last offset is not automatically a safe resume strategy.

For a nice transfer exercise, try the large-log-file section of this practice question. Focus on Part 3 and explain how your approach changes when you need a timestamp range or the end of a huge log, rather than a complete CSV transformation.

Try answering that before looking for another tool. You’ll quickly discover whether you’re choosing an access pattern or just naming a library.

Keep this explanation in your interview notes

I’ll first clarify whether the output needs information from previous records. For this filter-and-transform task, I can parse one bounded record, validate it, write its result, and release it. I’ll budget input buffers, retained state, serialization, and process overhead. If a new requirement introduces global grouping or sorting, I’ll account for that state separately and use disk-backed work when it cannot fit.

That answer gives the interviewer something concrete to explore, and it gives you a design you can actually defend.

Small samples are lovely for checking logic.
They just can’t tell you what your program will keep when the real file arrives.

Save the decision sheet for your next export, and follow along if you like turning ordinary engineering problems into interview practice that feels useful at work, too.

You can prepare a beautiful interview story about disagreeing with your team and still leave out the part that makes it interesting.

You spotted a problem, explained your reasoning, and brought a sensible proposal to the meeting. Then someone chose another approach, and your carefully rehearsed answer skips straight to the project shipping.

Hang on, because there is an entire working relationship hiding inside that jump.

Did you help the chosen approach succeed? Did you keep raising the same objection in every meeting? Could your teammate trust you to review their design fairly after your own proposal had been rejected?

I find that stretch of the story much more revealing than a polished account of how everybody eventually agreed with you. It gives us something concrete to practise, especially if Amazon is on your interview list.

And honestly, it is a useful question even when nobody is interviewing you.

Your disagreement story needs a second half

Amazon’s Have Backbone; Disagree and Commit principle describes respectfully challenging a decision and then committing once the decision has been made. Its SDE II preparation guidance also asks candidates to explain their actions, decisions, and specific evidence through structured examples.

That gives us a sensible preparation target. Pick a real disagreement, explain why it mattered, and stay in the story long enough to show what you did afterwards.

Winning the discussion can produce a useful example too. But if all your examples end with everyone adopting your recommendation, try finding one where you had to support a reasonable choice you initially opposed.

You do not need a dramatic confrontation with a director. A disagreement about a rollout, a dependency, a deadline, or who should own an operational task can give you plenty to discuss.

Let’s give both engineers a reasonable argument

Imagine a team adding scheduled reminders to an existing product. For this fictional example, the first release is a one-week pilot capped at 500 reminders per day, with an agreed target of sending each reminder within five minutes of its scheduled time.

These details are invented for practice. This is neither an Amazon system-design question nor an experience to borrow for your own interview.

Our engineer recommends a managed queue. They want dedicated delivery infrastructure and room to grow without increasing polling work on the application database.

A teammate recommends extending an existing database-backed worker for the limited pilot. The team already knows its deployment and monitoring setup, while a new queue would introduce configuration, permissions, and operational work before launch.

Both proposals still need careful handling of retries, duplicate processing, and failures. Choosing a managed queue does not magically settle those application-level responsibilities.

Notice how different this sounds from a story where one person cared about reliability and everyone else just wanted to ship. That version is tempting because it makes the narrator look wonderfully responsible. It also makes the disagreement suspiciously easy.

Before practising your answer, try describing the other person’s reasoning in language they would recognize as fair.

In our example, the trade off is whether the pilot justifies introducing new infrastructure now. Our engineer can explain the polling risk, ask about the expected workload, and offer evidence without pretending the teammate has forgotten how software works.

Now stay in the room after your proposal loses

Suppose the technical lead chooses the existing worker for the capped pilot, with a review after one week. Our engineer still prefers the queue as a possible longer-term direction.

This is where an answer needs observable actions. Saying that you respected the decision leaves the listener guessing what happened on Monday morning.

For the engineer in our example, meaningful follow-through could include these steps.

• Confirm the boundary of the decision. Record the pilot cap, delivery target, responsible owner, and review date so a limited experiment does not silently become an unlimited rollout.
• Own part of the implementation. Take responsibility for the retry path and duplicate-handling behavior, and get those choices reviewed.
• Exercise the failure paths. Help check what happens when the worker restarts or a delivery request is retried, including uncertainty about whether an external provider already accepted it.
• Make the outcome visible. Track delivery delay and failed work so the team can evaluate the pilot against the conditions it actually agreed to.

These are illustrative contributions, not a complete production design. In your answer, choose the work you really owned and be ready to explain it.

There is also a smaller, slightly uncomfortable detail worth remembering. Helping a teammate means giving their approach a fair review. If you scrutinize their implementation only to collect material for the next argument, the technical contribution can become tangled up with a personal scoreboard.

I prefer an answer that makes the working relationship visible. Perhaps you paired on a risky part, improved the rollout checklist, or raised a missing assumption before it became a launch problem.

Those details give the interviewer something better to explore than whether you sounded agreeable in the meeting.

Committing still leaves room for new evidence

Now change one detail in the example. During the pilot, delivery delay repeatedly exceeds the five-minute target.

That observation deserves investigation. It does not prove that your preferred queue would have prevented the problem. The cause could be a slow provider, a worker bottleneck, contention, or an assumption neither proposal addressed.

Our engineer can bring the measured delay, affected workload, and investigation results back to the team. If the pilot is missing its agreed target, the team can contain the impact and reconsider the plan with better information.

Before launch, it helps to agree on a trigger that is precise enough to use. For example, define how delivery delay is measured, what breach requires a review, who responds, and whether expansion pauses while the problem is investigated. The exact thresholds belong to the real service and its users.

In a practice answer, explain what changed between your original objection and the later discussion. New evidence gives the team a reason to revisit the decision; repeating your preference gives them very little new information.

Commitment also does not erase an obligation to raise a serious security, privacy, or safety concern through the appropriate process. Keep the example grounded in a legitimate engineering trade-off.

Here is how the answer could sound

Below is a fictional practice answer based on the reminder example. The successful pilot outcome is part of the illustration, separate from the missed-target variation above.

We were preparing a one-week reminder pilot capped at 500 reminders per day. I recommended a managed queue because I was concerned about polling load as usage grew. My teammate preferred our existing database worker because we already knew how to deploy and operate it, and the pilot was limited.

I brought the workload assumptions and reliability concerns to the design review. Our lead chose the existing worker with a five-minute delivery target and a review after the pilot. I documented those boundaries, took ownership of retry and duplicate-handling behavior, and worked with my teammate on restart checks and delivery-delay monitoring.

In the pilot, the reminders we tracked met the agreed delivery target, and the restart checks recovered pending work. That supported continuing within the tested scope; we had not established that the same approach would handle a larger rollout. I learned to separate the requirements of a bounded pilot from the infrastructure we might need later. Next time, I would agree on the decision criteria earlier so we could compare both options against the same constraints.

Read that answer aloud and you can hear several places an interviewer might interrupt. Which workload assumptions? What happened if delivery succeeded but the worker crashed before recording it? How was delay measured? What did the teammate own?

Good preparation gives you those details underneath the short answer. You do not have to deliver the entire design review before the interviewer has a chance to speak.

Try your own story before polishing the wording

Start with a real example and use the disagreement section of this leadership principle practice prompt. Give your answer once without reading the guidance, then return to the follow-ups and notice which details you cannot explain yet.

That small gap between recognizing a good answer and producing your own is where the useful practice starts.

For a focused rehearsal, make a rough note with five headings.

1. Shared goal. What were both of you trying to achieve, and what constrained the decision?
2. Disagreement. What did you recommend, what did the other person recommend, and what evidence did each of you have?
3. Decision. Who decided, what scope did they approve, and which concerns remained open?
4. Your next actions. What did you personally build, review, communicate, or monitor after the decision?
5. Outcome and reflection. What happened, how do you know, and what would you approach differently now?

Keep the note scruffy enough that you can change it. We are trying to remember an actual event, and a beautifully formatted paragraph can become surprisingly difficult to question.

Then ask a friend to interrupt your answer with the five questions in the image. If you practise alone, record the answer and respond to each follow-up without editing your first attempt.

Listen for missing substance. Maybe you describe the disagreement clearly but cannot name anything you did after the meeting. Maybe the team’s result has swallowed your contribution. Perhaps your explanation of the other option sounds dismissive when you hear it back.

Write down the gap, recover the details you can honestly support, and try again. If the story does not contain the behavior the question asks about, choose another real example rather than stretching this one to fit.

You can be disappointed and still be dependable

I like this question because it leaves room for a very ordinary human reaction. You cared about your idea, you put work into it, and you were disappointed when the team went another way.

Your answer does not need to airbrush that feeling out of the story. It needs to show how you handled it and what your teammates could rely on you to do next.

Before your next practice session, find the sentence where the decision was made. Then look at everything that comes after it.

That is where your answer can show the kind of engineer people want beside them when the meeting ends and the work begins.

~ Emily

Being allowed to use AI in a coding interview sounds like the moment you can finally relax. Then you remember that someone will be watching you decide what to ask it, which changes to accept, and what to do when the result looks convincing but behaves strangely.

Honestly, that last part deserves its own practice session.

Imagine reaching minute forty with a beautifully reorganized project and no clear explanation for why a refund request landed in the wrong queue. Your assistant has been very busy. Unfortunately, so has the clock.

I prefer a preparation plan that makes this situation visible while there is still time to learn from it. So let’s spend an hour making one small change, following the consequences, and leaving enough evidence that another engineer can understand what happened.

First, what DoorDash actually described

In its March 2026 engineering interview announcement, DoorDash described replacing traditional coding rounds with a 60-minute AI-assisted working session. Candidates work in their own development environment, use integrated AI tools, and contribute to a supplied project.

Its evaluation includes getting oriented, checking AI output, debugging, controlling scope, and explaining tradeoffs. Completing every task is less important than demonstrating good judgment, although speed still matters.

That gives us something much more useful to prepare for than a vague instruction to get better at prompting.

I am trying to write to focus on that engineering format.

Follow the instructions for your particular round and confirm tool permissions with your recruiter, because permission in one session does not automatically apply to the rest of an interview loop.

The schedule below is my proposed rehearsal, not DoorDash’s official agenda or a question I claim to have received. Our support-ticket example is an original practice project.

Give yourself a project small enough to understand

Before starting the timer, choose a tiny API project that already accepts a support message, assigns a queue, and saves the ticket. Use a tutorial project you can run locally, or ask an assistant to scaffold this practice app in advance. Keep the feature change for the timed session.

We only need three pieces here.

• An endpoint accepts a nonempty text message.
• A classifier chooses delivery, billing, or general support.
• A storage function records the ticket and its assigned queue.

For this exercise, an in memory store is enough. Existing validation rejects empty or whitespace only messages. We are routing requests to human teams, so nothing in this app actually issues a refund.

Let the starting classifier be deliberately ordinary Python.

def classify(text):
    message = text.casefold()
    if 'order' in message:
        return 'delivery'
    if 'payment' in message:
        return 'billing'
    return 'general'

Our new requirement is to send messages containing refund to a dedicated refunds queue. That rule should take priority even when the same message contains order or payment. Everything else keeps its existing behavior.

Yes, the rule is intentionally simple. The interesting work is discovering where it belongs, handling overlapping conditions, and checking that the saved result matches the decision.

Minutes 0–8 are for finding your way through one request

Start the project and send a message that already works. For example, Where is my order? should reach the delivery queue under the starting rules.

Follow that message from the endpoint into the classifier and then into storage. Read the actual functions involved, including any validation around them. Keep a tiny map beside your editor so you can return to it when the assistant suggests changing six other files.

AI can help locate the route and its callers, but open the referenced code yourself. A confident description of a function is still only a description until you connect it to the running application.

If setup fails, capture the exact error and explain the next thing you are checking. Silently spending ten minutes asking the assistant to try again makes your reasoning difficult to follow.

By the end of this stage, aim to explain where the message enters, where its queue is chosen, and where that choice is stored. You do not need a guided tour of every folder.

Minutes 8–15 are for deciding what counts as finished

Before requesting code, write the behavior you expect. For our deliberately limited keyword exercise, these examples make the priority rule concrete.

• refund goes to refunds.
• payment refund also goes to refunds, because the new rule wins.
• REFUND my order goes to refunds, regardless of capitalization.
• payment failed still goes to billing.
• Where is my order? still goes to delivery.
• hello still goes to general, while blank input remains invalid at the endpoint.

Notice how much work that second example does. Without it, two people can implement different rule orders and both believe they understood the requirement.

Also decide what stays outside this change. No new dashboard, no automatic money movement, and no database migration unless the existing schema genuinely requires a new allowed queue value.

I like having one sentence that keeps the session honest. We are adding a routing rule and preserving the surrounding behavior. It gives you something specific to return to when a helpful suggestion starts growing into a renovation.

Minutes 15–35 are for a change you can review

Now give the assistant the relevant context and a bounded request. This is a reusable prompt for our practice project.

Inspect the ticket route, classifier, storage call, and existing checks. Add a refunds queue for any message containing refund, ignoring case, and give that rule priority over order and payment. Preserve existing validation and all other routing behavior. Explain which files need changing before editing. Keep the patch focused and flag any schema restriction that prevents saving the new queue.

Then read the diff as it arrives. If the assistant changes the response shape, removes validation, or introduces a new abstraction, ask what requirement made that necessary. Sometimes there is a good reason, and sometimes your little routing task has acquired a framework.

Watch what happens when the new rule comes last

Suppose the proposed code inserts the refund condition just before the final fallback.

if 'payment' in message:
    return 'billing'
if 'refund' in message:
    return 'refunds'

That looks reasonable when you try refund by itself. But payment refund hits the earlier return and never reaches the new condition.

Under our agreed requirement, the fix is to put the refund check before both existing routing checks. Now the implementation expresses the priority we decided on, rather than depending on whichever order the assistant happened to produce.

Explain that reasoning while you work. You can say that the first matching branch returns immediately, so a message containing two keywords needs an explicit precedence rule. That is more informative than narrating every keystroke or reading the assistant’s explanation aloud.

Before moving on, search for places that assume there are exactly three queues. A classifier can return the right string while a schema, enum, or storage check rejects it. The request path you traced earlier helps you find that boundary.

Minutes 35–50 are for checking the behavior independently

Run the existing checks and add the examples you wrote before implementation. Keep their expected answers tied to the requirement, even if the assistant suggests changing them to match its output.

For the classifier, the overlapping keyword example gives us a compact check.

assert classify('payment refund') == 'refunds'
assert classify('REFUND my order') == 'refunds'
assert classify('payment failed') == 'billing'

Then go one level wider. Submit a refund message through the endpoint and inspect the stored ticket. Confirm that the response and persisted queue agree. A unit check of the classifier cannot establish that the endpoint calls it correctly or saves its result.

Try a blank message through that same endpoint, too. It should still be rejected under our starting contract. Sending it directly to the classifier would check a different boundary, because validation belongs to the route in this example.

Keep one awkward message in your pocket

Try I do not want a refund. Our substring rule sends it to refunds, even though a person reads the intent differently.

This is where you can show judgment without turning the session into an NLP project. The implementation follows the simplified rule, but the rule has a product limitation. Explain that distinction and ask whether the requirement should change.

If it should, agree on the revised behavior before editing again. If it should not, record the limitation and keep the scope intact. In production, a structured category field or a more carefully evaluated classifier might be appropriate, but neither appears for free because we noticed an awkward sentence.

After this rehearsal, a different task can reveal whether the review habit transfers. Try an AI-assisted cache review exercise and choose a case where two plausible implementations produce different answers. The point is to practice defending the expected behavior again.

Minutes 50–60 are for an explanation someone else can use

Protect a little time for the handoff. Show a normal request, an overlapping-keyword request, and the saved result. Keep the relevant code visible so your explanation has something concrete behind it.

Then write four short notes that describe the change, the evidence, the tradeoff, and the next step. For this practice project, the handoff can fit on a notebook page.

If part of the task is unfinished, describe exactly what remains and how you know. Perhaps the classifier checks pass but storage still rejects the new queue value. That is a useful, specific state of the work; saying that the feature is basically done hides the remaining problem.

Leave a clear account of what AI contributed and which decisions you made. Owning the result includes being able to explain the boundaries of your evidence.

Make the next rehearsal slightly less comfortable

For another attempt, ask a friend to choose one small change without giving you the implementation. They might introduce a new queue, change rule precedence, or require a reason code alongside the chosen queue.

Keep the same hour, but stop trying to reproduce the exact solution from this article. Your useful practice is the sequence of decisions that gets you from an unfamiliar request to behavior you can demonstrate.

Afterward, look for where your time actually went. Did you understand the project before editing? Did the patch grow beyond the requirement? Could you explain a failing example before asking for a fix? Choose the weakest moment and rehearse that part again.

I find this a much more useful ambition than sounding impressive every minute. You can be curious, ask for clarification, catch a mistake, and still leave someone confident that you understand the work.

AI may help you write the change quickly. Your job is to leave the next engineer with a change they can trust for clearly stated reasons.

~ Emily

My November 2026 shortlist, with the examples, traps and follow ups that make each question worth your time.

You can recognize an interview question, write a clean solution, and still solve a different problem from the one you were given.

Let tasks run in any order and a scheduling problem becomes a counting exercise. Keep their original order and that same shortcut can give the wrong answer. Both versions look reassuringly familiar until one little sentence changes the whole job.

That is what I want my November practice to catch. I’m choosing ten questions that make us read carefully, explain a decision and handle the follow-up without starting from zero. Your evenings are too valuable for another enormous bookmark folder.

I’ve added small practice examples and review prompts, and called out where the original report leaves details open.

Start with the section closest to your next round. Questions 1–3 cover algorithms, 4–7 cover practical implementation, and 8–10 cover system design and database reasoning. Each gives you something specific to explain when you finish.

1. Waymo: can you schedule the tasks in a different order?

Coding practice · Source page dated October 30

Run unit length tasks on one worker while leaving a cooldown between two tasks of the same type. You may reorder the tasks, and you want the earliest possible finish.

Try AAABBB with cooldown 2. One valid schedule is A B idle A B idle A B, which takes eight time units. Cooldown starts when a task finishes, so consecutive starts of the same type must be at least three units apart.

For n tasks, count the most frequent type. If its frequency is f, and m types share that frequency, the answer for nonempty input is max(n, (f − 1) × (cooldown + 1) + m). Empty input returns zero.

Before trusting the formula, explain the gaps forced by the busiest types and why enough other tasks can fill them. Counting the fixed alphabet takes O(n) time and constant extra space.

Then change one rule. Preserve the original order of AAABBB. It now takes 14 units. Track the next allowed start for each type and simulate; the eight-unit schedule is no longer available.

Try the full scheduling question yourself.

2. Lyft: can your smallest window handle repeated letters?

Coding practice · Source page dated October 28

Find the shortest substring of s containing every character required by t, including repeated copies. Matching is case-sensitive, and ties go to the earliest window. Return an empty string when no window qualifies.

For s = BAXAB and t = AAB, the answer is BAXA. Both four-character windows qualify, but the earlier one wins. Returning AB would quietly lose an A.

Grow a window until its character counts meet the requirements, then shrink it while it remains valid. Keep the best start and length, taking the final substring once. Moving both pointers forward gives O(|s| + |t|) time; the fixed letter alphabet bounds counting space.

Your check is the moment a count falls below its requirement. Explain why removing one surplus A can be harmless while removing the next A invalidates the window. Update the answer only for a strictly shorter window to preserve the earlier tie.

For a useful follow-up, ask whether the same constant-space claim survives arbitrary Unicode input. Your character model suddenly matters.

Work through the complete substring problem.

3. Google: can you reuse the answer when the tree root moves?

Coding stretch · Source page dated October 29

Given a tree with positive edge weights, return the sum of distances from every node to every other node. Running a traversal from each starting node works, but takes O(n²) time.

Use the path 0 —2— 1 —3— 2. The distance sums are [7, 5, 8]. We can check all three by hand before reaching for a larger example.

Root the tree once, compute subtree sizes and the root’s distance sum, then move the answer across each edge. For an edge of weight w from parent u to child v, let s be the child’s subtree size.

answer[v] = answer[u] + w × (n − 2s)

Those s nodes get w closer; the other n − s nodes get w farther away. Two traversals give O(n) time and O(n) space. Use a sufficiently wide integer type and avoid recursion-depth trouble on a long chain.

Finish by explaining the formula without looking at it. Then try a star-shaped tree. The practice payoff is understanding what changes across one edge, so you can derive the expression again.

Try the weighted-tree question from scratch.

4. Perplexity: can you build a file system without being too helpful?

Practical coding · Source page dated October 29

Build an in-memory file system with directory creation, listing, file appends and file reads. The important restriction is that mkdir creates only the final directory. Missing parents stay missing.

After mkdir(/notes), calling mkdir(/notes/november/day1) must fail because /notes/november does not exist. Automatically creating it feels helpful, but violates this version’s contract.

Agree on the error behavior and what ls returns for a file before writing methods. The report leaves those details open. For practice, choose explicit exceptions for invalid paths and a one-element filename list when listing a file.

Represent files and directories distinctly, and share one path resolver across operations. Resolve and validate the parent before changing state. Directory listing costs O(k log k) for sorting k names, in addition to walking the path.

Check the state after a failed operation. That unsuccessful call must leave no half-created directories behind. Then add recursive directory creation as a deliberate new feature and decide whether its failures should also be atomic.

Build the file system from this prompt.

5. Palantir: can you learn the code before trying to improve it?

Code comprehension · Source page dated October 30

Read an unfamiliar SQL-like query class, explain how it runs, then solve tasks using its mapping and accumulation operations. The original class was not included in the report, so any replacement implementation is a practice stand-in.

Use a small query API with lazy where and select steps, plus a terminal fold action. Trace one row through it before touching the implementation.

Suppose completed orders have amounts 12 and 8, while a cancelled order has amount 100. Filtering completed rows before summing must return 20. If you first project only amount and then filter by status, that later operation no longer has the field it needs.

Explain when work actually happens. Building a lazy query records steps; executing an action consumes rows. For a fold, say what the accumulator represents after every row and why the empty result has a sensible initial value.

Then inspect whether running the same query twice reuses a list or consumes an exhausted iterator. This is a lovely way to practice learning unfamiliar code without turning the session into a rewrite of somebody else’s library.

Explore the query-class exercise in full.

6. Anthropic: can your crawler stop at the right time?

Concurrency practice · Source page dated October 26

Starting from one URL, discover reachable pages on exactly the same hostname. Remove fragments before deduplicating, handle cycles, and fetch each eligible page at most once. Build the single-threaded version before adding workers.

Two workers can discover the same link simultaneously. Checking the visited set and claiming that URL must happen atomically, keeping only the individual set operations safe does not protect the whole decision.

Keep network calls outside the shared state lock. Then account for outstanding work, because an empty queue can coexist with a running fetch that will discover five more pages. Completion needs both no queued work and no active work, observed through coordinated state.

Make your check deterministic. Use a fake parser that pauses one worker while the queue becomes empty, then releases another link. The crawler must wait and discover it. Afterwards, inject a fetch error and explain how work accounting still reaches completion. A hanging fetch additionally needs a timeout or cancellation policy.

Practice the crawler and its concurrency follow-up.

7. MongoDB: can a waiting writer ever get its turn?

Concurrency stretch · Source page dated October 26

Implement a read-write lock using a mutex and condition variables. Several readers may enter together, but a writer must be alone. For the first version, exclude reentrancy, upgrades and timed acquisition.

Track active readers, an active-writer flag and waiting writers under one mutex. Condition waits belong inside loops that recheck the admission rule, as Python’s condition-variable documentation explains.

Now imagine one reader enters, a writer starts waiting, and new readers keep arriving before the current readers finish. Exclusion can remain correct while that writer waits forever.

Block new readers once a writer is waiting, so existing readers can drain. This addresses reader-induced writer starvation, but does not by itself guarantee FIFO service among writers. Specify fair queueing if every waiter needs eventual admission, and assume holders eventually release.

Draw the arrival sequence before writing more code. Show R1 entering, W1 waiting, R2 waiting, R1 leaving, then W1 entering. Follow up by asking whether sustained writer traffic can now starve readers. Fairness needs its own argument.

Try the complete read-write lock exercise.

8. Instacart: does your inventory system actually own the inventory?

System design · Source page dated October 30

Design an inventory system for grocery delivery. The original report names the topic without a detailed specification, so the ownership and freshness assumptions below are our practice framing.

Start by separating a warehouse you control from a partner supermarket whose shelves you only observe. Reserving the last unit makes sense in the first case. A reservation in your database cannot prevent an in-store shopper from taking the same item in the second.

For partner stores, track product and store identity alongside each stock signal’s source, observation time and arrival time. Keep enough history to resolve stale or conflicting reports according to an explicit policy.

Try this disagreement. A store file observed at 9:00 says five units remain. A shopper reports none at 9:20. The old file arrives at 9:30. Explain why arrival order alone should not make the item confidently available again.

Then discuss what the customer sees while confidence is low, how replacements work and how you measure shown availability against actual finds. If the exercise switches to an owned warehouse, add atomic reservations and releases. That change deserves a different design.

Work through the full inventory design prompt.

9. eBay: can you make your own system asynchronous?

System design and project discussion · Source page dated October 28

Explain a system you have genuinely worked on, then change one synchronous request into asynchronous work. This is where your project experience becomes part of the technical problem.

Choose a slow operation whose result the caller can wait for. An export is a useful practice example. The initial response can return an accepted job ID, while a separate status endpoint eventually returns success, failure or a download location.

Trace acceptance, durable job creation, dispatch, worker execution and result retrieval. Each arrow needs an answer when the process stops halfway through.

Pause at the gap between saving the job and sending its message. If the process crashes there, how does the job become discoverable again? A transactional outbox is one option, a durable job table that workers poll is another. Pick one and explain it.

Then deliver the same work twice. Show how repeated execution avoids duplicating its side effects, and how permanent failures reach the caller. Async changes the interaction and absorbs bursts, it does not erase work, guarantee faster completion or make an overloaded dependency healthy.

Practice the asynchronous system design question.

10. MongoDB, can you explain what happens after SQL leaves your editor?

Database discussion · Source page dated October 26

Walk through a query’s lifecycle inside a disk based database, then explain the buffer pool and optimizer. A relational engine is a useful practice model, but settle the engine and scope first.

Follow parsing and name resolution into planning, execution, page access and returned rows. Explain how cached pages avoid some reads, why a page in active use cannot simply be evicted, and what changes when a page is dirty.

For a write-ahead-logged engine, the relevant log records must be durable before the corresponding dirty data page reaches disk. The recovery story belongs in the explanation.

Now give the optimizer an unpleasant surprise. It expects ten matching rows but gets a million. Explain how that estimate can make a nested-loop plan expensive, then compare estimates with actual rows using a controlled example.

PostgreSQL’s EXPLAIN examples give you real plans to read. Remember that EXPLAIN ANALYZE executes the statement. Use a safe local dataset for practice, and investigate statistics and data distribution before prescribing another index.

Explore the complete database internals question.

Make this a November plan you can actually finish

Please don’t turn ten questions into ten unfinished side projects. Choose the lane that matches your interview format, then give one question enough attention to expose a specific gap.

• For a coding screen, begin with #1 and #2. Try #3 when you can already explain a traversal and want a harder reuse-of-work problem.
• For practical coding, start with #4 or #5, then add #6. Save #7 for rounds that explicitly expect concurrency fundamentals.
• For backend or design rounds, work through #8 and #9, then use #10 to deepen the storage and query discussion.

Give the first attempt a boundary. Spend about 35 minutes implementing a small version or tracing a design, then ten minutes explaining its weakest assumption. These are practice timeboxes, so a larger concurrency exercise can span several sessions.

Keep a tiny note with four lines after each attempt. Record the requirement you nearly missed, the example that exposed it, the change you made and the follow-up you still cannot answer. That last line chooses your next session for you.

Come back from an empty file or blank diagram a day or two later. Change one meaningful rule before reattempting it. Reordering becomes fixed order, a missing parent becomes an error, or a quiet workload becomes two callers arriving together.

That is the progress I want from this list. By the end of November, I want the familiar looking problem to make me curious enough to read the sentence that changes everything.

If you’re following along, tell me which number deserves the next full walkthrough. I’ll keep these notes focused on the examples and decisions you can actually use in an interview.

~ Emily

Imagine asking Claude to fix a bug, getting a tidy patch, and watching the tests turn green. You are already thinking about the next task when somebody asks which test failed before the fix.

You scroll back through the conversation, and suddenly the reassuring summary is doing a lot of work.

Maybe the fix is correct.
Maybe the test never exercised the broken behavior.
Maybe both the implementation and its new test agree on the same wrong assumption.

Passing result cannot tell those stories apart by itself.

I prefer a slightly awkward question before getting comfortable with an AI generated repair. If we put the original mistake back into a disposable local copy, does the same test complain again?

That question gives us something more useful than another request for the model to double-check its answer. It gives us a small experiment we can understand.

In its account of code review, the Claude Code team describes giving agents runnable checks and asking for their output before human review. My favorite place to start is smaller than a whole review system, with one bug and a test whose expected answer comes from the requirement.

Let’s lose one support ticket without raising an error

Here is an invented, deliberately small example. A support dashboard lists tickets from oldest to newest, and the user presses Next to continue where the previous page ended.

For this exercise, the dataset stays fixed while we paginate. Each ticket has an integer creation time and a unique integer ID. Sort by creation time first, then ID, and return only tickets strictly after the last one already shown. The page size is a positive integer.

Our entire dataset fits on a sticky note. Ticket 41 was created at time 100, ticket 42 was also created at time 100, and ticket 43 was created at time 101.

With one ticket per page, the reader should see 41, then 42, then 43. No advanced algorithm is hiding here, which is exactly why this makes a useful review exercise.

# Each row is (created_second, ticket_id).
def next_page(rows, after, limit):
    ordered = sorted(rows)
    if after is not None:
        ordered = [
            row for row in ordered
            if row[0] > after[0]
        ]
    return ordered[:limit]

After the first page, our cursor is (100, 41). The function keeps only tickets with a creation time greater than 100, so it jumps straight to ticket 43. Ticket 42 quietly disappears from the walk through the results.

Nothing crashed, and the remaining ticket looks perfectly reasonable. Checking that the response contains one ticket would let this bug stroll right past us.

Write the expected answer before admiring the patch

Before asking an agent to repair anything, preserve the smallest example that demonstrates the problem. In this case, we can justify the answer by hand because ticket 42 follows ticket 41 in the agreed ordering.

def test_next_page_keeps_a_ticket_with_the_same_time():
    rows = [(100, 41), (100, 42), (101, 43)]
    result = next_page(rows, after=(100, 41), limit=1)
    assert result == [(100, 42)]

Following the original code, the actual result is [(101, 43)], so this assertion should fail. That failure matters because it connects our test to the complaint. These are hand-traced outcomes for the example, rather than a claimed test run.

Notice where the expected answer came from. We worked it out from the ordering rule. We did not call another helper from the implementation to calculate what the result should be, because that helper might carry the same mistake.

If the test passes on the original code, pause before applying the patch. Check that it imports the intended function, that the fixture actually contains a tie, and that the test runner discovered the test. If it fails because a dependency is missing, we still have setup work to do before we have evidence about pagination.

A failure needs to fail for the right reason

An assertion showing ticket 43 where ticket 42 belongs is useful. An import error, a timeout from an unavailable database, or a test skipped by configuration does not demonstrate this particular bug.

Honestly, this is where a confident AI summary can make a messy situation feel cleaner than it is. Keep the actual command and output so somebody else can distinguish a reproduced defect from an environment problem.

Now repair the rule that lost the ticket

Our ordering uses two fields, but the filter remembers only one. The correction is to compare the complete cursor in the same order used for sorting.

def next_page(rows, after, limit):
    ordered = sorted(rows)
    if after is not None:
        ordered = [row for row in ordered if row > after]
    return ordered[:limit]

Python compares these tuples in order. A later creation time qualifies immediately. When the creation times match, a larger ID qualifies. That makes (100, 42) greater than (100, 41), so ticket 42 stays in the result.

Changing the time comparison from > to >= looks tempting, but it lets ticket 41 back into the next page. We need to include the other tickets at the same time while excluding the ticket already shown.

Keep this repair narrow enough to explain. Renaming every variable, introducing a pagination class, and reorganizing the repository would make it harder to see which change addressed the missing ticket.

The small function is a teaching model that sorts an in-memory list. A real database implementation needs a matching order, cursor predicate and suitable index. With ascending non-null fields, its filter would express created_at > cursor_time OR (created_at = cursor_time AND id > cursor_id). It also needs defined behavior when records change between requests, which our fixed-dataset example deliberately leaves outside its contract.

Bring the old mistake back while the test stays still

Once the focused test passes on the patch, make a disposable local copy containing both the patch and the test. Restore only the old timestamp only comparison there, leaving the test and its expected answer unchanged.

The same assertion should fail again, with ticket 43 replacing ticket 42. Now we have checked that this test can distinguish the repaired behavior from the original defect.

This is a small mutation check, meaning we deliberately put a known mistake into the code to see whether the test detects it. Keep that experiment away from production, and retain the clean patched version separately.

If the restored bug still passes, investigate before celebrating. The test might be importing a different copy, a mock might be bypassing the comparison, or the assertion might be too weak to notice which ticket returned.

One successful mutation check does not prove the whole program correct. It tells us something narrower and genuinely useful, which is that this test detects this mistake under these conditions.

Give the fix a few uncomfortable neighbors

Our first case protects ticket 42, but the repair also has to preserve the next page, the end of the list and the page-size rule. These nearby cases make it harder for a special-case patch to masquerade as a general correction.

In a real test file, keep the input data fixed and write the expected results explicitly. Here is one compact way to express those cases using pytest parametrization.

import pytest
@pytest.mark.parametrize(
    ("after", "limit", "expected"),
    [
        ((100, 41), 1, [(100, 42)]),
        ((100, 42), 1, [(101, 43)]),
        ((101, 43), 1, []),
        ((100, 41), 2, [(100, 42), (101, 43)]),
        (None, 1, [(100, 41)]),
    ],
)
def test_page_boundaries(after, limit, expected):
    rows = [(101, 43), (100, 42), (100, 41)]
    assert next_page(rows, after, limit) == expected

I deliberately shuffled the input so the test also asks the function to honor its ordering rule. For a larger practice exercise, walk all the pages and compare the collected IDs with a hand-written expected list. That catches omissions and duplicates across page boundaries.

Then run the relevant existing suite. A focused regression test can protect our missing ticket while missing a broken route, a changed response field or an authorization check elsewhere.

Match the check to the place where users get hurt

For a database backed endpoint, add coverage that reaches the actual query and cursor encoding. For a frontend race, control the order in which requests resolve and check what appears on screen. For a permission bug, exercise the relevant user boundary instead of checking only a helper that returns true.

Choose the next check by asking which layer could still break the promised behavior. That is a better guide than trying to make an arbitrary test-count number look impressive.

Give your agent a job that ends with evidence

You can ask an AI assistant to help construct the reproduction, but review the expected answer before letting it repair the implementation. Otherwise, the assistant can accidentally write a test that politely agrees with its own patch.

For this kind of bug, a useful request can stay wonderfully plain.

Reproduce the reported failure before changing application code.
Explain why the expected answer follows from the requirement.
Run the focused test on the original implementation and show
the command, the failure, and why it matches the report.
Make a focused repair without weakening the assertion.
If a test expectation needs changing, explain the contract
conflict first instead of silently updating it.
Check the patched code with the same test and nearby cases.
In a disposable local copy, restore the original defect
and check that the regression test detects it again.
Report what ran, what did not run, and what remains uncertain.

This request does not make an agent infallible. It makes its work easier to inspect, and it gives you specific places to intervene when a result is missing or an assumption is wrong.

If the project cannot run locally, preserve the reproduction and explain the blocker. A reasoned expectation can still help a reviewer, but label it as unexecuted. There is no need to dress up a prediction as a green test result.

Use your next practice session to leave a better trail

For a follow-up, try an existing API debugging exercise and choose one failing behavior before touching the rest. That exercise covers a different movie-search API, so our ticket-pagination patch is not its answer. The reusable part is the evidence trail.

Keep one note with the requirement, the smallest reproduction, the cause, the focused change and the checks you actually ran. During an interview, that note becomes a clear explanation of your decisions. During a review, it gives the next engineer somewhere useful to start.

Some real bugs are too intermittent to reproduce on demand. For those, capture the relevant timing, versions and observations, then use controlled scheduling, replay or monitoring where possible. Be honest about the uncertainty instead of promising that one neat unit test settles the incident.

What I want from an AI fix is a trail I can follow from the complaint to the corrected behavior. Once I understand that trail, I can explain the change without rereading the assistant’s entire conversation.

Before accepting your next patch, try keeping the test still while bringing the mistake back. If it catches the bug again, you have earned a much more useful kind of confidence.

~ Emily

Imagine Maya ordering an oat latte on her way to work. Her payment goes through, but the café’s tablet never receives the order. She arrives at the counter holding a receipt for a drink nobody has started.

Now the retry button looks less like a convenience and more like a very expensive question. Does tapping it recover her coffee, create another order, or charge her again?

I love a system design problem that gets interesting before we have even finished ordering breakfast. Everyone understands the customer’s expectation, which makes it much harder to hide a missing guarantee behind an impressive architecture diagram.

This walkthrough uses a Salesforce coffee-ordering question. We’ll build a concrete design, follow Maya’s order through it, then interrupt the flow at the moments that matter. Maya and the café are our worked example; the implementation choices below are a proposed solution.

First, decide when the barista is allowed to start

Our service lets customers choose a store, customize drinks, pay online or at the counter, and follow preparation status. Staff need a store-specific queue and a way to claim and complete orders.

Before choosing a database, settle two business rules. Can preparation begin before payment, and can a customer cancel once someone starts making the drink? These decisions change the allowed state transitions.

For this version, payment must be confirmed before an order enters the preparation queue. A staff member can record payment collected at the counter. Customers may cancel until preparation begins; later problems go through a staff-assisted resolution. Those are our explicit assumptions, and another café could choose differently.

Start with immediate pickup in one region. Delivery, inventory forecasting and loyalty points can wait. Scheduled pickup will be a follow-up, because placing a drink in tomorrow morning’s queue is different from asking someone to make it now.

Three guarantees keep the rest of the discussion focused. Retrying one checkout must not create another order, two staff members must not both win the same claim, and a successful payment must eventually produce either fulfillment or a recorded resolution such as a refund.

Draw one order’s journey before adding more services

A small ordering application backed by a relational database is enough for the initial design. Separate modules handle menus, orders and payments; they do not need separate deployments just to earn their own boxes.

Maya’s request reaches the Order API, which validates the selected store and drink options, calculates the price and saves order #42. The payment integration works with an external provider. Staff and customer screens read order state through authorized endpoints.

Whenever an important state change needs downstream delivery, save an event in an outbox table in the same database transaction. A relay publishes committed events for screen updates and notifications. This is the transactional outbox pattern, which addresses the gap between committing a change and sending its notification.

The relay can publish twice if it crashes after sending but before recording delivery. Consumers therefore track event IDs or use versioned upserts. The durable order remains queryable even when a notification is delayed.

A compact API surface makes the responsibilities easier to explain.

GET  /stores/{store_id}/menu
POST /orders                 # checkout key + item choices
GET  /orders/{order_id}       # current authorized view
POST /orders/{order_id}/claim # staff, expected version
POST /orders/{order_id}/ready # staff, expected version
POST /orders/{order_id}/cancel
POST /payments/webhook       # verified provider events

Derive customer identity from authentication and check staff access to the order’s store. A guessed order ID must never be enough to read someone’s purchase or move their drink into preparation.

Save the receipt, not a pointer to today’s menu

Suppose Maya’s drink totals 650 cents. Ten minutes later, someone edits the menu price. Her existing order must still explain the 650 cents she approved.

Keep an Orders record with customer, store, currency, total, fulfillment status and version. OrderItems hold snapshots of the chosen drink, modifiers, quantity and charged amounts. PaymentAttempts link to the order and retain the provider reference, attempt identity, amount, currency and payment status.

Calculate totals on the server, using integer minor units for money. If the submitted menu version is stale and the price changed, return the updated quote for confirmation instead of quietly charging more.

Each checkout carries a client-generated key. Enforce uniqueness on customer_id plus checkout_key in the database, and save a canonical request fingerprint with it. The same key and same request return the existing order; the same key with different drink choices is rejected.

Create that identity record and the order atomically. Otherwise, two simultaneous requests can both pass an application-level lookup and insert two lattes. Keep the mapping for the supported retry window, and make a genuinely new purchase use a new key.

Payment and preparation need different state

A single paid flag becomes awkward as soon as refunds enter the conversation. Money can have arrived while preparation is cancelled, and the refund may still be pending. Both facts deserve to survive.

In this design, fulfillment moves from AWAITING_PAYMENT to QUEUED, then PREPARING, READY and COLLECTED. Cancellation is allowed from the first two states. Payment has its own lifecycle, including PENDING, SUCCEEDED and the refund states.

A verified successful payment queues an order only if it is still eligible. If cancellation has already won, preserve CANCELLED and arrange the refund instead. A late success callback is not permission to resurrect the order.

Serialize these decisions on the order row. Record the payment result, decide the fulfillment transition, and save any required event or refund intent in one transaction. The external charge or refund itself still happens outside that transaction.

Two baristas tap the same order

Both tablets can display order #42 as QUEUED. That is fine until both people press Start and both screens confidently report success.

Make the claim a conditional database update. Here is the essential PostgreSQL shape, with authorization checked before it runs.

UPDATE orders
SET fulfillment_status = 'PREPARING',
    claimed_by = :staff_id,
    version = version + 1
WHERE id = :order_id
  AND store_id = :authorized_store_id
  AND fulfillment_status = 'QUEUED'
  AND version = :expected_version
RETURNING id, version;

Only a caller that receives a returned row owns the claim. A competing caller rereads the order and updates its screen. The cancellation path uses the same row and a compatible conditional transition, so it cannot approve cancellation after a claim has already committed.

This is where the software meets the counter. Staff must wait for claim confirmation before starting. If that confirmation is lost, reread the owner before taking action. A database constraint cannot stop two humans who both decide to make the drink anyway.

Now lose the payment confirmation

We have reached Maya’s original problem. The provider captured her payment, but our application never received the event that would move the order forward.

A timeout tells us that we did not receive an answer. It does not prove that the payment failed. Starting a fresh charge at this point can make the customer’s morning considerably worse.

Before contacting the provider, persist a payment attempt with a stable identity and provider idempotency key. Save the provider’s payment reference when it becomes available. An uncertain retry resumes that same attempt using the provider’s documented idempotency behavior, rather than inventing a new payment.

Provider idempotency has a retention policy, so it is not a forever guarantee. Once safe replay is no longer available, unresolved attempts need provider lookup or manual reconciliation instead of another blind charge.

Webhooks handle the normal asynchronous path. Verify their signatures, durably record accepted events, deduplicate repeated event IDs, and process them through the order’s transition rules. Stripe’s webhook documentation explicitly covers duplicates and events arriving out of order, so delivery order is an unsafe foundation for payment state.

A reconciliation worker covers the missing-event path. It scans attempts that have remained unresolved beyond a chosen age, retrieves the existing payment from the provider, and checks its order association, account, amount and currency.

If P9 succeeded for Maya’s 650-cent order, the same transition handler records success and queues eligible order #42. If another worker or webhook already applied that result, it becomes a no-op. Old notifications must not regress a confirmed success to pending.

If #42 is cancelled, persist refund work using a stable refund identity and track that refund until the provider confirms its outcome. Never display refunded merely because a request was sent.

Notice the two recovery problems we have separated. Reconciliation discovers what happened outside our database. The outbox delivers a decision already committed inside it. Each has a different piece of missing information to recover.

A disconnected café needs an honest promise

Imagine the store tablet loses connectivity during the morning rush. Keeping the ordering button green does not make the barista aware of newly paid orders.

For the initial cloud dependent design, use store connectivity signals and a staff-controlled pause to stop accepting new pickup orders when the store is unavailable. Heartbeats can be stale, so orders accepted just before detection still need an explicit cancellation or fulfillment resolution.

After reconnecting, fetch the current store queue from the database. Versioned updates prevent old messages from moving the screen backward. Customers can use server-sent events for status, with a fresh snapshot on reconnect and polling as a fallback. A notification is a convenience; the current order record is the recovery path.

If offline operation is mandatory, the design needs a durable local queue, stable operation IDs and clear ownership of which orders the store may execute. Staff also need a reconnect reconciliation process. That is a substantial extension, so name the requirement before promising uninterrupted service.

Scheduled pickup changes when work becomes eligible

For an 8:15 pickup, store the requested time with a time zone and admit the order only if the store has capacity. A scheduler releases it near the preparation window using a conditional transition and an event saved atomically with that change.

Duplicate scheduler deliveries must not enqueue it twice. Cancellation before release must prevent preparation, and the payment authorization or capture policy needs to match how far ahead customers can order. Putting tomorrow’s latte straight into today’s queue is wonderfully efficient at producing cold coffee.

Scale the part that actually needs scaling

For a sizing exercise, assume 120 orders per second across 600 stores and 30 read requests per order. That gives roughly 3,600 reads per second, with large differences between quiet stores and busy ones. These are planning assumptions, not measured Salesforce traffic.

Cache menus by store and menu version, index active orders by store and status, and keep customer-status delivery separate from payment processing. Partitioning by store can come later if measurements justify it; a hot flagship location still needs its own capacity controls. If 20,000 customers are watching active orders, budget for those concurrent connections independently of order-write throughput.

Watch the age of unresolved payments, committed queue entries not acknowledged by a store, unpublished outbox events and pending refunds. An average API latency graph can look lovely while Maya is still standing at an empty pickup shelf.

Let the schema answer the SQL follow-up

The question also includes spend-related SQL. Before writing a query, define the word spend. An order total is not proof that money was collected, and joining orders to their items can multiply a payment amount.

For this exercise, use a payment ledger containing one deduplicated row per provider capture or refund. Store customer_id, currency, kind, amount_minor and posted_at. Define last month’s spend as USD captures minus USD refunds posted during a specified half-open time window. Enforce uniqueness on the provider account plus capture or refund object ID, so repeated notifications cannot add the same money movement again.

SELECT customer_id,
       SUM(CASE WHEN kind = 'CAPTURE'
                THEN amount_minor
                ELSE -amount_minor END) AS net_spend
FROM payment_ledger
WHERE currency = 'USD'
  AND kind IN ('CAPTURE', 'REFUND')
  AND posted_at >= :month_start
  AND posted_at < :next_month_start
GROUP BY customer_id
HAVING SUM(CASE WHEN kind = 'CAPTURE'
                THEN amount_minor
                ELSE -amount_minor END) > 0
ORDER BY net_spend DESC, customer_id
LIMIT 1;

Supply the boundaries in the agreed reporting time zone, converted to the storage convention. This query picks one winner deterministically on ties. Returning all tied winners requires ranking the grouped totals instead. Refunds posted this month count this month under our definition, even if the original purchase was earlier.

For users who have never paid anything, use a different definition and check for the absence of any positive capture. A fully refunded customer has paid before, so they do not belong in this result.

SELECT u.id
FROM users AS u
WHERE NOT EXISTS (
    SELECT 1
    FROM payment_ledger AS p
    WHERE p.customer_id = u.id
      AND p.kind = 'CAPTURE'
      AND p.amount_minor > 0
);

This also includes users with no orders at all. If the interviewer means lifetime net spend of zero, say so and change the aggregation. Similar-sounding questions can describe different customer groups.

Walk one latte through the whole answer

Bring the explanation back to Maya. Her checkout identifies one order, its receipt preserves the agreed price, and its payment attempt identifies one external operation. A lost confirmation leaves work to reconcile. A confirmed payment makes an eligible order claimable, and one barista receives permission to prepare it.

Practice the walkthrough by changing one event at a time. Lose the checkout response, deliver the webhook twice, cancel just before preparation, then disconnect the store. For each change, name the durable record and the decision that still has to be made.

That is the part of this question worth taking into another interview. A convincing answer can follow one customer through uncertainty and explain how the system reaches a correct outcome. By then, every important arrow on the diagram has earned its place, and Maya finally has a reason to trust the pickup screen.

~ Emily

You can draw a second database in 3 seconds. Explaining what happens when both copies contain writes the other one has never seen takes a little longer.

That gap is where system design starts getting interesting for me. The diagram looks reassuring, everyone agrees that redundancy is sensible, and then someone asks how the system actually comes back. Suddenly, the arrows need a much better explanation.

Before my next system design interview, I want some practice with that uncomfortable middle. Public incident reports are lovely material for it because they show the assumptions that disappeared under real conditions, along with the decisions engineers made afterward.

I picked three reports from AWS, Cloudflare and GitHub. These are historical incidents, and their dates matter. We’re reading them for the reasoning they make visible, rather than treating them as breaking news or claiming these companies asked an interview question.

Bring one notebook page. You do not need to memorize every service name to get something useful from this.

Read for the missing assumption

My preferred starting point is a five-line note. As you read, record what changed, what assumption stopped holding, how the failure spread, what complicated recovery, and what evidence could show users were okay again.

That last line is easy to skip because a report usually tells us how the incident ended. In an interview, though, we have to explain what we would observe before declaring our own system healthy.

Keep reported facts separate from your proposed safeguards. If you think a version check or a rollout gate would help, write it as a design idea and explain its limits. Knowing the ending of an incident does not make the response obvious to the people living through it.

1. AWS shows why an old operation can become dangerous later

During the October 19–20, 2025 DynamoDB incident in Northern Virginia, a delayed DNS updater applied an older plan after another updater had installed a newer one. Cleanup then deleted the older plan, which was now active. The regional endpoint lost its IP addresses, and the resulting inconsistent state needed manual intervention.

The freshness check had happened before the delay, so it no longer protected the eventual update. Follow that sequence in AWS’s post-event summary.

My takeaway is to examine the gap between checking permission to change state and actually changing it. Redundant workers still need a safe way to disagree about which operation is current.

Try a smaller version in your notebook

Imagine a fictional settings service with two workers. One prepares revision 12 and pauses. Another commits revision 13. When the first resumes, where does your design reject its outdated update?

Put that decision at the storage boundary, using an atomic conditional write against the expected revision. A separate read followed by an unconditional write leaves another gap for a competing worker. Then consider cleanup separately, because a version can be old without being safe to delete if something still uses it.

You now have a concrete concurrency discussion to practice, including what a rejected worker should reread before retrying.

2. Cloudflare makes an ordinary internal file worth questioning

On November 18, 2025, a database permissions change caused duplicate entries in a Bot Management feature file. The enlarged file spread across Cloudflare’s network and exceeded a limit in the software consuming it, causing failures. Recovery included stopping the bad file’s generation and distribution, inserting a known good file and restarting the core proxy.

Start with the cause and recovery sections of Cloudflare’s incident report. This was not a cyberattack.

I take away a useful boundary question. What prevents internally generated data from carrying the same mistake to every instance?

Give your imaginary service a bad configuration

For a separate practice scenario, imagine a recommendation API receiving a generated rules file. Write down what the consumer validates before activation, including schema, maximum size and supported version. Decide what happens when validation fails.

Keeping the last accepted version may preserve service, but stale rules can have consequences too. State which rules may become stale and which require the service to stop or degrade. Then choose a limited rollout group and a user-facing error signal that would halt expansion.

That is a much more useful answer than adding a box labelled validation and hoping the diagram takes care of the details.

3. GitHub separates a repaired connection from a recovered service

GitHub’s October 21, 2018 network interruption lasted 43 seconds. It triggered failover, while databases in different locations accumulated writes missing from the other side. Switching back safely became difficult, and applications also struggled with the changed latency. Service degradation lasted 24 hours and 11 minutes.

Read the timeline in GitHub’s post-incident analysis, especially the decisions made to preserve data. The report describes degraded service, not a complete shutdown of every GitHub feature.

My takeaway is that a working network link does not settle which data is authoritative or make the application ready to use it.

Make recovery prove something

In a fictional two-region order service, suppose a customer received confirmation just before connectivity failed. Which durable record lets you prove that order still exists after failover?

Choose whether your design prioritizes accepting new orders or pausing writes when authority is uncertain. Explain the customer consequence either way. Then name what prevents the old writer from continuing after a new writer takes over.

A leadership decision needs enforcement where writes happen. Merely telling a disconnected process that it has lost leadership is a fairly optimistic recovery plan.

Turn your notes into a decision you can defend

This is where I’d stop reading for a moment. Collecting three more links is tempting, but the useful output is one decision you can explain under pressure.

Use the worksheet as a reading aid, then build a small scenario of your own. Keep it narrow enough that you can trace an individual request instead of narrating the whole internet.

For example, imagine a fictional receipt service whose email provider is unavailable. Your first response might be to retry failed sends, which is a reasonable beginning. Now make that answer specific.

• Set a retry limit and explain what happens after it is reached, so unsuccessful work has an owner and a next step.
• Consider a timeout after the provider has accepted the email. Explain whether a repeated request can create a duplicate and what evidence could resolve the uncertainty.
• Limit recovery traffic so the accumulated backlog does not overwhelm the provider when it returns.
• Choose a recovery check that includes the age of the oldest pending receipt, because successful new requests can hide unfinished older work.

Those choices form a connected explanation. You have described the failure, the uncertainty it creates, the response, and the evidence you need afterward.

Notice how a generic plan to add retries becomes a discussion about ownership, duplicate effects and backlog recovery. You can get there without claiming you have operated a service at any particular company’s scale.

Give your safeguard an awkward follow-up

Every safeguard has a dependency or a cost, so choose one and press on it. If recovery needs an administrator to flip a switch, can that administrator still sign in during the incident? If your fallback serves older data, how old can it get before the response becomes unacceptable?

I prefer writing down the condition that would change my decision. For example, a stale product description and a stale account permission deserve very different answers. Naming that difference shows what your fallback actually guarantees.

You are allowed to leave an open question in the notebook. Mark what you need to learn next instead of filling the gap with a reassuring technology name.

A 20 min practice session you can actually finish

After an initial read of one report, try this routine with the report available for reference. It is a practice suggestion, not a validated assessment or a claim that twenty minutes covers distributed systems.

1. Spend the first five minutes filling the worksheet with short notes. Mark anything you inferred rather than found in the report.
2. Use the next five minutes to redraw the failure path with only the components needed to explain it. Include the state that changes, not just the machines.
3. Spend five minutes changing one assumption in your own scenario. A delayed worker resumes, a dependency stays unavailable, or a successful response gets lost. Choose a response and name its cost.
4. Use the final five minutes to explain your decision aloud and write a concrete recovery check. If you get stuck, record the exact uncertainty for your next session.

Later, pick a fresh practice question, choose one of its dependencies and apply the same method. The goal is to carry the reasoning into an unfamiliar design rather than reproduce the incident’s architecture.

Save the completed page with your notes. Next time you revisit it, cover your answer and see whether you can still explain why that particular safeguard belongs there.

Bring the reasoning back to your diagram

I still want a clear architecture diagram. I also want every important arrow to survive a conversation about delays, conflicting state and recovery.

That is why these reports belong in my preparation pile. They give us something concrete to think with, and the worksheet turns that reading into a habit we can repeat.

Before opening another resource, choose one box in your own design and explain how you would know its users had recovered. If the answer gets uncomfortable, you have found a useful place to practice.