System Design & Distributed Systems: Senior / Lead Engineer Interview Roadmap

Debarshi Raj Basumatary

14 min read

·

Oct 7, 2026

--

Listen

Share

A comprehensive 120-hour curriculum for scalable, reliable and production-grade architecture

Press enter or click to view image in full size

1. Purpose and Target

This curriculum is the system-design companion to the Spring Boot Senior/Lead roadmap. It is designed to develop the ability to take ambiguous business requirements, translate them into technical requirements, estimate scale, choose architecture and data models, reason about distributed-system trade-offs, and defend the design under interviewer pressure.

The target is not to memorize architecture diagrams. A senior/lead engineer should be able to explain why a component exists, what happens when it fails, where state lives, how the system scales, what consistency guarantees exist, and how the system is operated.

Target outcome

• Decompose ambiguous requirements into functional and non-functional requirements.
• Perform back-of-the-envelope capacity and traffic estimation.
• Design APIs, domain boundaries and data models.
• Understand DNS, HTTP, TLS, load balancing and reverse proxies.
• Choose SQL, NoSQL, cache, object storage and search technologies based on workload.
• Reason about replication, partitioning, sharding and consistency.
• Understand CAP, PACELC, quorum systems and consistency models.
• Design queues, streams and event-driven architectures.
• Handle retries, timeouts, idempotency, deduplication and backpressure.
• Design reliable distributed workflows and transactions.
• Apply caching, rate limiting, distributed locking and coordination.
• Design for observability, disaster recovery and operational excellence.
• Communicate architecture clearly using diagrams, APIs, schemas and trade-off analysis.
• Solve senior/lead system-design interviews within a realistic time limit.

2. Curriculum at a Glance

• 1. System Design Method — Hours: 6 · Primary outcome: Structured interview approach and requirement decomposition
• 2. Networking + Web Foundations — Hours: 8 · Primary outcome: Understand the infrastructure beneath services
• 3. Scalability + Load Balancing — Hours: 8 · Primary outcome: Scale stateless and stateful workloads
• 4. Data Storage + SQL — Hours: 10 · Primary outcome: Model data and reason about database performance
• 5. NoSQL + Distributed Data — Hours: 8 · Primary outcome: Select and scale non-relational stores
• 6. Caching — Hours: 6 · Primary outcome: Design multi-layer caching and invalidation
• 7. Messaging + Streaming — Hours: 10 · Primary outcome: Queues, Kafka-style streams and delivery semantics
• 8. Distributed Systems Theory — Hours: 12 · Primary outcome: Consistency, CAP, consensus and coordination
• 9. Reliability + Resilience — Hours: 10 · Primary outcome: Build systems that survive failures
• 10. Distributed Transactions — Hours: 8 · Primary outcome: Saga, outbox, idempotency and workflow design
• 11. Search + Storage + Specialized Systems — Hours: 6 · Primary outcome: Search, object storage, files and time-series workloads
• 12. Security Architecture — Hours: 6 · Primary outcome: Identity, authorization and service-to-service security
• 13. Observability + Operations — Hours: 6 · Primary outcome: Operate systems using SLOs and telemetry
• 14. Cloud + Deployment Architecture — Hours: 6 · Primary outcome: Containers, Kubernetes and multi-region design
• 15. Advanced Architecture — Hours: 8 · Primary outcome: Event-driven, CQRS, event sourcing and platform thinking
• 16. Interview Case Studies + Mocks — Hours: 6 · Primary outcome: End-to-end senior/lead interview readiness
• TOTAL — Hours: 120 · Primary outcome: Senior/lead system-design capability

3. The Core System-Design Mental Model

For every design, continuously reason across six dimensions:

• Correctness — What must never happen? What invariants must hold?
• Scale — How many users, requests/sec, records and bytes?
• Latency — What is the p50/p95/p99 target? Which calls are on the critical path?
• Availability — What happens when dependencies fail? What availability is actually required?
• Consistency — Which data must be strongly consistent and which can be eventually consistent?
• Operations — How will we detect, debug, deploy, recover and evolve the system?

A good architecture is a set of explicit trade-offs, not a collection of fashionable technologies.

4. Phase 1 — System Design Method (6 hours)

The interview framework

1. Clarify the problem and define scope.
2. Separate functional requirements from non-functional requirements.
3. Estimate traffic, storage and bandwidth.
4. Define APIs and major domain entities.
5. Choose the high-level architecture.
6. Choose storage based on access patterns.
7. Design caching and asynchronous flows.
8. Address consistency, concurrency and failure.
9. Address security and observability.
10. Identify bottlenecks and scaling strategies.
11. State trade-offs and alternatives.
12. Summarize the final architecture.

Requirements checklist

• Users and actors
• Core use cases
• Read/write ratio
• Latency targets
• Availability target
• Data retention
• Geographic scope
• Consistency requirements
• Security/compliance constraints
• Expected growth

Back-of-the-envelope estimation

• Requests per second
• Peak vs average traffic
• Storage per record
• Daily/monthly/yearly storage growth
• Bandwidth
• Cache memory
• Replication multiplier
• Partition count
• Approximate database capacity

Rule of thumb: use estimates to constrain architecture decisions; do not pretend they are exact production measurements.

5. Phase 2 — Networking + Web Foundations (8 hours)

Topics

• DNS resolution
• TCP connection establishment
• TLS handshake
• HTTP/1.1, HTTP/2 and HTTP/3 concepts
• Keep-alive and connection reuse
• Request/response lifecycle
• Reverse proxies
• CDNs
• Load balancers
• L4 vs L7 load balancing
• NAT and basic network boundaries
• WebSockets and server-sent events
• gRPC vs REST
• Timeouts and connection pools

Interview questions

• What happens after a user enters a URL?
• Why does HTTP/2 improve performance?
• L4 vs L7 load balancer?
• When would you choose WebSockets?
• REST vs gRPC?
• Where should TLS terminate?
• Why are connection pools important?

6. Phase 3 — Scalability + Load Balancing (8 hours)

Scaling fundamentals

• Vertical scaling
• Horizontal scaling
• Stateless application design
• Load-balancing algorithms
• Health checks
• Sticky sessions
• Session externalization
• Autoscaling
• Queue-based load leveling
• Hotspot detection

Scaling pattern

Client → CDN → Load Balancer → stateless application fleet → cache/database/message broker.

Senior questions

• Why is statelessness useful?
• When are sticky sessions justified?
• What happens if one partition/key becomes hot?
• How do you scale a write-heavy service?
• How do you protect a database when application instances scale rapidly?

7. Phase 4 — Data Storage + SQL (10 hours)

Relational modeling

• Entity and relationship modeling
• Normalization and denormalization
• Primary and foreign keys
• Indexes
• Composite indexes
• Covering indexes
• Query plans
• Joins
• Transactions
• Isolation
• Optimistic and pessimistic locking
• Read replicas
• Connection pooling

Database scaling

• Read replicas
• Vertical scaling
• Partitioning
• Sharding
• Archival
• Materialized views
• CQRS read models

Data modeling rule

Start from access patterns and invariants, not from a technology choice. Ask: what queries must be fast, what must be transactional, and what can be eventually consistent?

8. Phase 5 — NoSQL + Distributed Data (8 hours)

Categories

• Key-value — Simple high-volume lookups
• Document — Flexible aggregate-shaped records
• Wide-column — Large distributed workloads with predictable access patterns
• Graph — Relationship-heavy traversals
• Time-series — Timestamped metrics/events

Topics

• Access-pattern-driven modeling
• Denormalization
• Partition keys
• Hot partitions
• Replication
• Quorum reads/writes
• Eventual consistency
• Multi-model architecture
• Polyglot persistence

Interview challenge

Choose between PostgreSQL, Dynamo-style key-value storage, document storage and Redis for four different workloads. Defend each choice using access patterns and failure behavior.

9. Phase 6 — Caching (6 hours)

Caching layers

Browser → CDN → reverse proxy → application cache → distributed cache → database.

Patterns

• Cache-aside
• Read-through
• Write-through
• Write-behind
• Refresh-ahead
• TTL
• Explicit invalidation
• Negative caching

Failure modes

• Cache stampede
• Cache penetration
• Cache avalanche
• Hot keys
• Stale data
• Cache inconsistency
• Cold cache after deployment

Interview question

Design caching for a product catalog with 10 million products, high read traffic and frequent price updates. Explain invalidation and failure behavior.

10. Phase 7 — Messaging + Streaming (10 hours)

Queues vs streams

• Primary model — Queue: Work distribution · Stream: Ordered event log
• Consumption — Queue: Usually one worker handles a message · Stream: Multiple consumers can independently read
• Replay — Queue: Usually limited · Stream: Core capability in log-based systems
• Typical use — Queue: Background jobs · Stream: Events, analytics, integration

Topics

• Producer/consumer model
• Kafka-style partitions
• Consumer groups
• Offsets
• Ordering
• Retention
• Replication
• At-most-once delivery
• At-least-once delivery
• Exactly-once concepts
• Idempotent consumers
• Retries
• Dead-letter queues/topics
• Backpressure
• Batching
• Event schemas and compatibility

Senior patterns

• Transactional outbox
• Change data capture
• Event-driven integration
• Work queues
• Delayed jobs
• Priority queues
• Fan-out
• Competing consumers

11. Phase 8 — Distributed Systems Theory (12 hours)

Consistency

• Strong consistency
• Eventual consistency
• Read-your-writes consistency
• Monotonic reads
• Causal consistency
• Session consistency

CAP theorem

Understand CAP precisely: during a network partition, a distributed system cannot simultaneously guarantee both strong consistency and availability for every operation. Do not reduce CAP to the slogan ‘pick two.’

PACELC

When there is a partition, choose between availability and consistency; else, choose between latency and consistency.

Replication

• Leader/follower replication
• Multi-leader replication
• Leaderless replication
• Synchronous vs asynchronous replication
• Replication lag
• Quorum reads and writes
• Conflict resolution

Consensus and coordination

• Why consensus exists
• Raft concepts
• Leader election
• Quorum
• Failure detection
• Split-brain
• Distributed locks
• Fencing tokens
• Logical clocks
• Lamport clocks
• Vector clocks — conceptual understanding

Senior interview questions

• What does CAP actually say?
• Why can asynchronous replication lose recent writes?
• What is split-brain?
• Why do distributed locks need expiry/fencing considerations?
• Why is consensus difficult?
• When is eventual consistency acceptable?

12. Phase 9 — Reliability + Resilience (10 hours)

Failure taxonomy

• Process crash
• Network timeout
• Packet loss
• Dependency overload
• Database outage
• Cache outage
• Message duplication
• Partial failure
• Region failure
• Corrupt or inconsistent data

Resilience patterns

• Timeouts
• Retries with exponential backoff
• Jitter
• Circuit breakers
• Bulkheads
• Rate limiting
• Load shedding
• Backpressure
• Graceful degradation
• Fallbacks
• Health checks
• Dead-letter handling

Critical rule

Never add retries blindly. A retry can amplify an outage. Always ask whether the operation is idempotent, whether the dependency is already overloaded, and whether the latency budget permits another attempt.

Reliability exercise

A payment dependency has intermittent 2-second latency and 5% errors. Design timeout, retry, circuit breaker, idempotency and user-facing fallback behavior.

13. Phase 10 — Distributed Transactions (8 hours)

Why distributed transactions are hard

A local database transaction provides atomicity inside one transactional boundary. Once a workflow crosses independent services/databases, atomic commit becomes difficult and expensive.

Patterns

• Two-phase commit — understand conceptually and its trade-offs
• Saga choreography
• Saga orchestration
• Compensating transactions
• Transactional outbox
• Idempotency keys
• Deduplication
• Workflow engines
• Exactly-once business effect vs exactly-once message delivery

Payment example

Order created → payment requested → payment authorized → inventory reserved → order confirmed. For each step, define the state machine, retry behavior, compensation, idempotency key and failure recovery.

14. Phase 11 — Search, Object Storage + Specialized Systems (6 hours)

Search

• Inverted indexes
• Tokenization and analyzers
• Relevance scoring
• Sharding and replicas
• Indexing pipelines
• Search consistency
• When search engines should not replace the primary database

Object storage

• Blob/object storage
• Multipart upload
• Presigned URLs
• Metadata
• CDN integration
• Lifecycle policies
• Durability vs availability

Specialized workloads

• Time-series databases
• Geospatial systems
• Analytics warehouses
• Data lakes
• When to introduce a specialized store

15. Phase 12 — Security Architecture (6 hours)

Topics

• Authentication vs authorization
• OAuth2 and OIDC
• JWT
• API keys
• mTLS
• Service identities
• Secrets management
• Key rotation
• Least privilege
• Network segmentation
• Zero-trust concepts
• Audit logging
• Data encryption at rest and in transit
• PII and sensitive-data handling

Architecture exercise

Design authentication for browser users, mobile clients, internal services and third-party integrations. Explain where identity is established and how downstream services authorize requests.

16. Phase 13 — Observability + Operations (6 hours)

Three pillars

• Logs
• Metrics
• Traces

Operational concepts

• SLI, SLO and SLA
• Error budgets
• p50/p95/p99 latency
• Throughput
• Saturation
• RED method
• USE method
• Distributed tracing
• Correlation IDs
• Alert design
• Runbooks
• Incident response
• Postmortems

Production scenario

p99 latency doubled while average latency stayed stable. Explain why percentile latency is useful and how you would use traces and dependency metrics to isolate the problem.

17. Phase 14 — Cloud + Deployment Architecture (6 hours)

Topics

• Containers
• Docker
• Container networking
• Kubernetes primitives
• Deployment vs StatefulSet concepts
• Services and ingress
• Config and secrets
• Horizontal Pod Autoscaling
• Rolling deployments
• Blue/green deployments
• Canary releases
• Health/readiness probes
• Infrastructure as code concepts
• Multi-AZ deployment
• Multi-region deployment

Cloud architecture

Think in capabilities rather than provider-specific names: compute, network, object storage, relational DB, distributed cache, queue/stream, secrets, observability and CDN.

18. Phase 15 — Advanced Architecture (8 hours)

Event-driven architecture

• Domain events
• Integration events
• Event contracts
• Schema evolution
• Event choreography
• Event replay
• Consumer versioning

CQRS

Separate write and read models when their access patterns, scaling needs or consistency requirements materially differ. Do not use CQRS merely because it is fashionable.

Event sourcing

• Event log as source of truth
• Aggregate reconstruction
• Snapshots
• Event schema evolution
• Replay
• Operational complexity

Architecture evolution

• Monolith
• Modular monolith
• Service extraction
• Strangler pattern
• Shared database risks
• Service ownership
• Platform engineering concepts

19. Phase 16 — Case Studies

• URL Shortener — ID generation, redirects, cache, database scaling, hot keys, analytics
• Rate Limiter — Token bucket, sliding window, Redis, distributed state, fairness
• Notification System — Kafka/queue, fan-out, retries, DLQ, user preferences
• E-commerce — Catalog, cart, order, inventory, payment, consistency
• Payment System — Idempotency, state machine, retries, reconciliation, audit
• Ticket Booking — Concurrency, locks, reservations, expiration, overselling
• Food Delivery — Orders, drivers, location, real-time updates, events
• Ride Sharing — Geospatial indexing, matching, real-time state
• Chat System — WebSockets, message ordering, delivery status, fan-out
• Video Platform — Object storage, CDN, transcoding, metadata, streaming
• Social Feed — Fan-out on write/read, ranking, caching, hot users
• Search Platform — Ingestion, indexing, shards, replicas, relevance
• Distributed Scheduler — Leases, leader election, retries, duplicate execution
• Metrics Platform — High-cardinality data, ingestion, aggregation, retention
• Cloud File Storage — Object storage, metadata, sync, conflict resolution

20. How to Solve a 45-Minute System Design Interview

• 0–5 min — Clarify requirements and scope.
• 5–10 min — Estimate scale and identify key constraints.
• 10–15 min — Define APIs and core data model.
• 15–25 min — Draw high-level architecture and explain request flow.
• 25–32 min — Deep dive into the hardest subsystem.
• 32–38 min — Failure, consistency, scaling and security.
• 38–43 min — Observability, operations and trade-offs.
• 43–45 min — Summarize decisions and remaining risks.

What interviewers are evaluating

• Requirement clarification
• Structured thinking
• Scale awareness
• Correctness
• Trade-off quality
• Failure-mode reasoning
• Communication
• Ability to go deep rather than list technologies
• Ability to revise the design when requirements change

21. Architecture Trade-Off Matrix

• Communication — Option A: REST · Option B: Async events · Typical trade-off: Simplicity/latency vs decoupling/resilience
• Storage — Option A: SQL · Option B: NoSQL · Typical trade-off: Transactions/query flexibility vs distributed scale/access-pattern specialization
• Cache — Option A: Local · Option B: Distributed · Typical trade-off: Latency/isolation vs shared state
• Consistency — Option A: Strong · Option B: Eventual · Typical trade-off: Correctness/coordination vs availability/latency
• Deployment — Option A: Monolith · Option B: Microservices · Typical trade-off: Operational simplicity vs independent scaling/ownership
• Messaging — Option A: Queue · Option B: Stream · Typical trade-off: Work distribution vs replayable event log
• Replication — Option A: Sync · Option B: Async · Typical trade-off: Consistency vs latency/availability
• Scaling — Option A: Vertical · Option B: Horizontal · Typical trade-off: Simplicity vs elasticity/fault isolation

22. Senior/Lead Interview Question Bank

Requirements and estimation

• How do you start a system design problem?
• How do you estimate peak QPS?
• How do you estimate storage for five years?
• Which requirements can you safely defer?

Scalability

• How do you make a service horizontally scalable?
• What creates hotspots?
• How do you scale a database write path?
• When does caching stop helping?

Distributed systems

• Explain CAP precisely.
• What is quorum?
• Why does replication lag matter?
• What is split-brain?
• Why is consensus necessary?

Reliability

• Why can retries make an outage worse?
• How do circuit breakers work?
• What is bulkheading?
• How do you design graceful degradation?

Data

• SQL vs NoSQL?
• Read replicas vs sharding?
• When would you denormalize?
• How do you prevent hot partitions?

Messaging

• At-least-once vs at-most-once?
• How do you make consumers idempotent?
• Why use an outbox?
• How do you handle poison messages?

Architecture

• When should a monolith become microservices?
• How do you choose service boundaries?
• How do you avoid distributed-monolith architecture?
• When should you use synchronous calls vs events?

23. Common Design Mistakes

• Jumping to technologies before clarifying requirements.
• Using microservices for every problem.
• Saying ‘use Kafka’ without explaining why.
• Ignoring database access patterns.
• Ignoring failure and partial failure.
• Adding retries without idempotency.
• Assuming exactly-once delivery solves business duplication.
• Treating CAP as ‘pick any two.’
• Ignoring hot keys and hot partitions.
• Designing for average traffic instead of peak traffic.
• Ignoring operational complexity.
• Using cache without an invalidation strategy.
• Using distributed locks without understanding lease expiry and fencing.
• Ignoring security until the end.
• Drawing architecture boxes without explaining data flow and state.

24. Final Readiness Rubric

• Foundation — Can explain basic client/server, APIs, databases, caching and queues.
• Intermediate — Can design common web systems and explain basic scaling.
• Senior-ready — Can reason about consistency, partitioning, messaging, resilience, data modeling and operational behavior.
• Lead-ready — Can set boundaries, quantify trade-offs, handle failure, evolve architecture and communicate decisions across teams.

Senior readiness test

• Solve a URL shortener in 30 minutes.
• Solve a ticket-booking system in 45 minutes.
• Explain CAP, replication and consistency without notes.
• Design an idempotent payment workflow.
• Design a Kafka-based order pipeline with outbox and DLQ.
• Explain how the system behaves when Redis, database, Kafka or a region fails.
• Estimate traffic and storage before selecting infrastructure.
• Defend SQL vs NoSQL and monolith vs microservices choices.
• Identify the first bottleneck and the next scaling step.
• Explain how you would observe and operate the system.

25. 4-Week Intensive Interview Plan

• Week 1 — Focus: Method, networking, scalability, databases · Deliverables: 5 estimation drills + 3 designs
• Week 2 — Focus: NoSQL, caching, messaging, consistency · Deliverables: Kafka/outbox design + 4 designs
• Week 3 — Focus: Reliability, distributed transactions, security, cloud · Deliverables: Failure-mode matrix + 3 designs
• Week 4 — Focus: Advanced architecture + mocks · Deliverables: 7 timed mocks + architecture review

Weekly practice rule

At least 60% of study time should involve drawing, designing, calculating, debugging hypothetical failures or defending decisions. Reading alone will not build interview skill.

26. 12-Week Deep Curriculum

• 1 — Primary theme: System design method + estimation · Core deliverable: Requirement/estimation worksheet
• 2 — Primary theme: Networking + web · Core deliverable: Request lifecycle diagram
• 3 — Primary theme: Scaling + load balancing · Core deliverable: Horizontally scalable API architecture
• 4 — Primary theme: SQL + data modeling · Core deliverable: Database schema + query analysis
• 5 — Primary theme: NoSQL + distributed storage · Core deliverable: NoSQL design comparison
• 6 — Primary theme: Caching · Core deliverable: Multi-layer cache design
• 7 — Primary theme: Queues + Kafka · Core deliverable: Event-driven order flow
• 8 — Primary theme: Consistency + CAP + replication · Core deliverable: Consistency decision matrix
• 9 — Primary theme: Reliability + resilience · Core deliverable: Failure-mode and recovery plan
• 10 — Primary theme: Distributed transactions · Core deliverable: Saga + outbox implementation design
• 11 — Primary theme: Security + observability + cloud · Core deliverable: Production architecture
• 12 — Primary theme: Mock interviews · Core deliverable: 10 timed designs + final review

27. The Lead Engineer Standard

A lead-level answer should go beyond drawing Client → API → Database. You should be able to explain the state owned by each component, the invariants that must hold, the expected load, the failure modes, the consistency model, the recovery path, the security boundary, the operational signals, and the trade-offs behind each major decision.

The strongest interview habit is to repeatedly ask: What happens when this component is slow? What happens when it is unavailable? What happens when a message is duplicated? What happens when traffic is 10× larger? What happens when a region disappears? What data can be stale? What must be correct? How do we know the system is healthy?

28. Final Checklist

☐ Requirement decomposition

☐ Functional vs non-functional requirements

☐ Capacity estimation

☐ Latency and percentile reasoning

☐ DNS / HTTP / TLS / networking

☐ CDN / reverse proxy / load balancer

☐ Horizontal scaling and statelessness

☐ SQL data modeling and indexes

☐ Replication / partitioning / sharding

☐ NoSQL access-pattern design

☐ Caching and invalidation

☐ Queues and streams

☐ Kafka fundamentals

☐ Delivery semantics and idempotency

☐ CAP / PACELC

☐ Consistency models

☐ Quorums and replication

☐ Consensus concepts

☐ Distributed locking

☐ Timeouts / retries / circuit breakers

☐ Rate limiting / backpressure / load shedding

☐ Saga / outbox / distributed workflows

☐ Search / object storage

☐ Security architecture

☐ Observability / SLOs

☐ Cloud / Kubernetes concepts

☐ Multi-AZ / multi-region design

☐ CQRS / event sourcing concepts

☐ Architecture evolution

☐ Timed system-design interviews

29. Definition of Done

You are ready for senior/lead system-design interviews when you can consistently produce a coherent design under time pressure, quantify the major constraints, choose appropriate data and communication patterns, identify failure modes, explain consistency, design for observability and security, and defend trade-offs when the interviewer changes a requirement.

The objective is architectural judgment: knowing not only what can be built, but what should be built, why, at what scale, with what guarantees, and at what operational cost.