What Production Taught Snitch: Building a Resilient Data Platform

SNITCH Threads

9 min read

·

Sep 21, 2026

--

Listen

Share

Part 3 of 3 — The honest bit. Production incidents, hard lessons, and the principles that outlasted every tool choice.

The Pattern We Didn’t See Coming

Looking back at everything that broke, a pattern emerges. It took us a while to see it, but once we did, it changed how we think about architecture:

Almost every problem we encountered was solved by increasing separation between components — never by adding complexity.

Not by bigger limits. Not by more monitoring. Not by adding features. By pulling things further apart.

This final part covers the failures, the fixes, and the lessons that matter more than any specific tool choice.

Press enter or click to view image in full size

The Problems

1. Too Many Runtimes, One Container

The gravitational pull toward fewer, bigger containers is real.

Early on, the temptation is obvious: install Airbyte, Airflow, dbt, and custom connectors all in one container. Shared runtime, fewer images to manage, one deployment.

We nearly did this.

Then the problems arrived:

• Airbyte wanted Python 3.9. dbt needed 3.11. Airflow had different opinions.
• Upgrading dbt broke Airflow’s dependencies. Upgrading Airflow broke a connector.
• A security patch for one tool meant rebuilding and redeploying everything.
• Debugging failures meant archaeology: which component’s dependency caused this?

Shared runtimes don’t scale. They rot.

Fix: One runtime image per component, each with its own release lifecycle.

Before (Coupled):

• Single container with Airbyte + Airflow + dbt + connectors
• Dependency conflicts everywhere
• Painful, risky upgrades
• Tight coupling

After (Separated):

• dbt-1.9.8 (rarely changes)
• airflow-3.0.2 (changes independently)
• source-shopify-0.1.2 (versioned separately)
• airbyte-0.45.1 (isolated runtime)

Upgrading dbt now touches nothing else. Upgrading Airflow doesn’t cascade through six other components. Each component evolves at its own speed.

2. “Works on My Machine” Images

This one costs real operational time.

A Docker image built locally on your laptop cannot be reliably deployed across a Kubernetes cluster. Every node needs to run the exact same artifact. Local builds don’t guarantee this.

We learned this the hard way. Image A worked in dev. Deployed to staging: different node, different behavior. Deployed to prod: mysterious failures, impossible to reproduce, impossible to rollback.

Fix: Amazon ECR as the single registry. Single source of truth.

The flow:

• Developer builds image locally
• Push to ECR (versioned tag)
• All environments pull the same artifact
• Reproducible across clusters
• Rollback = deploy previous tag

Nothing deploys unless it’s been built, versioned, and published to ECR. Every deployment references a known tag. Every rollback is just reverting to a previous tag.

This sounds like bureaucracy. It’s actually freedom — the freedom to roll back instantly, with confidence that the previous version is exactly what was running before.

3. The Temptation to Run dbt Inside Airflow

This is the mistake we most nearly made.

And it’s the one we’d warn every team about.

Installing dbt on Airflow worker nodes looks convenient for about a week. “Let’s just put dbt here, it’s simpler, everything’s in one place.”

Then:

• You upgrade dbt, and suddenly Airflow can’t import its own modules
• You upgrade Airflow, and dbt’s dependencies break
• One worker has a different version than another
• A transformation failure cascades into a scheduler failure
• You can’t upgrade either tool without risking the other

Two runtimes sharing one environment is how you get coupled systems that break together under load.

Fix: A dedicated dbt runtime image, launched by KubernetesPodOperator as a throwaway pod per execution.

Key principle: Orchestration and transformation never share a runtime.

Airflow lives on long-running worker nodes. dbt lives in ephemeral pods that are created, used once, destroyed.

They evolve independently. They fail independently. They scale independently.

4. Shared Execution Environments Rot

Even with a dedicated dbt runtime, a long-running shared dbt environment accumulates state.

Think about it: after 100 dbt runs, the environment has:

• 100 sets of dependency changes
• 100 sets of cached artifacts
• 100 runs’ worth of state mixed together
• Debugging is archaeology (which run left which artifact?)

The second run behaves differently than the first. The 50th run is different from the 25th. Reproducibility evaporates.

Fix: Ephemeral pods. Create pod → run dbt → destroy pod.

Pod Lifecycle:

• Created (fresh, clean, known state)
• dbt run (executes in isolation)
• Pod destroyed (no trace left behind)
• Next execution starts from clean state

Every execution starts from a known clean state. There is nothing to accumulate, nothing to rot, nothing to mysteriously affect the next run.

The first execution and the 1000th execution behave identically because they’re both starting from the same clean image.

5. Baking SQL Into Images Slows Everyone Down

Our first instinct was wrong: package the dbt project inside the runtime image.

Seemed logical. The code is version controlled, the image is versioned, everything’s reproducible.

Then you make a one-line SQL fix. What should take 30 seconds now takes:

1. Edit SQL
2. Commit to Git
3. Rebuild Docker image (5 minutes)
4. Push to ECR (2 minutes)
5. Redeploy to Kubernetes (3 minutes)
6. Wait for pod rollout (2 minutes)
7. Test the change (finally)

A simple fix became a 12-minute deployment.

And analysts got slower. The feedback loop broke.

Fix: The git-sync init container.

Separation:

• Runtime image (stable, rarely changes) — dbt binary, Python, dependencies, utilities
• Project code (changes constantly) — Git repository, cloned fresh at pod startup

Every dbt execution:

1. Pod creates
2. git-sync init container runs first, clones latest from Git
3. dbt container starts, finds fresh project
4. Transformation runs
5. Pod terminates

SQL changes ship at Git speed. No image rebuilds, no redeployments, no waiting. Analyst merges to main, next dbt run picks it up.

The runtime stabilizes. The project accelerates.

6. Letting the Frontend Touch Airflow

This couples user-facing applications to an internal orchestration tool.

It’s tempting: the UI needs to trigger a transformation, Airflow has a REST API, why not just call it directly?

Because then:

• Every UI change that talks to Airflow is a cross-team negotiation
• Airflow API changes break the UI
• User-facing code depends on orchestration internals
• You can’t upgrade Airflow without coordinating with frontend teams
• You can’t change the orchestrator without rewriting every UI

Leaking orchestration concepts into user interfaces couples things that should be independent.

Fix: The backend mediation layer.

The flow:

• User UI → Backend API (business concepts)
• Backend API → Airflow REST API (orchestration concepts)
• Airflow REST API → DAG execution

Frontends speak business APIs: “run this transformation.” Only the backend speaks Airflow.

Now:

• Airflow is an implementation detail
• The UI never changes when Airflow changes
• You can swap Airflow for a different orchestrator without touching the UI
• User-facing applications remain stable while infrastructure evolves

7. Admin Tools on the Internet

Airflow and Airbyte UIs are powerful. They have no business being publicly reachable.

An unsecured Airflow instance on the internet is an invitation to chaos. Someone could trigger any DAG, modify configurations, view secrets.

Fix: Internal-only networking.

Architecture:

• Internet → ALB (Application Load Balancer)
• ALB → Backend API
• Backend → Airflow (internal only)
• Backend → Airbyte (internal only)

Only the backend is public. Airflow and Airbyte are reachable through the backend, never directly.

Administrative interfaces sit behind IP-based restrictions: office networks, approved CI/CD runners, authorized machines only.

Principle: Expose only what must be exposed, and put the backend in front of everything else.

8. Airbyte Started Dying in Production — The OOM Saga

This was our biggest production incident. And it deserves the longest explanation.

The Symptom

Our Airbyte deployment started as a standard Helm install with near-default resources. In development it was flawless.

Then production arrived.

Long-running synchronizations — our large catalogue syncs were the canonical example — processed far larger datasets, ran for far longer, consumed memory continuously.

Connector pods started restarting. Kubernetes reported OOMKilled (out of memory).

Jobs died before completion, retried, and died again.

At first it looked like flaky connectors. It wasn’t. It was infrastructure.

The Naive Fix (That Didn’t Work)

Our first instinct was everyone’s first instinct: increase the memory limit.

Jobs survived longer… and then failures returned at a higher memory ceiling.

Raising limits had delayed the failures without explaining them. We were turning knobs instead of understanding the problem.

That was the moment we stopped guessing and started investigating.

The Real Insight: Airbyte Is Not One Application

This is the crucial realization:

Airbyte is a distributed system. We’d been treating it as a single monolithic service.

It actually consists of:

• Server — Handles API requests, lightweight
• Temporal — Workflow engine, moderate resources
• Bootloader — Startup only, minimal resources
• Connector pods — Source & destination connectors (the real resource hogs)

Each component has a completely different resource profile:

• Server: Lightweight. Idle most of the time.
• Temporal: Workflow engine. Mostly idle.
• Bootloader: Only exists at startup.
• Connector pods: Download, buffer, serialize, write. Memory grows continuously.

We’d been giving every component identical memory allocations. That meant starving the component that needed it while over-provisioning five that didn’t.

The Real Fix: Three Parts

Part 1 — Per-Component Tuning

We profiled each component individually:

• Server: 50 MB observed, low CPU, 128 MB allocated
• Temporal: 100 MB observed, medium CPU, 256 MB allocated
• Bootloader: 30 MB observed, spike at start, 64 MB allocated
• Source connector: Variable, medium-high CPU, 2048 MB allocated
• Dest connector: Variable, high CPU, 2048 MB allocated

Size each one from observed utilization, not estimates.

Part 2 — Requests vs Limits

The most important Kubernetes lesson: these answer different questions.

• Requests — “I need at least this much space” (determines scheduling)
• Limits — “I can’t have more than this much space” (caps consumption)

Balancing both is critical:

• High request, low limit → pod fails to schedule OR gets evicted
• Low request, high limit → pod scheduled wrong, kills node
• Balanced → scheduled correctly AND constrained appropriately

We’d been focusing on limits. Requests are where the real work happens.

Part 3 — The Node Layer

Some failures weren’t pod-level at all.

Multiple large connector pods competing for the same nodes caused memory pressure, cascading evictions, and scheduling failures. No pod-level tuning fixes node exhaustion.

This is where the Cluster Autoscaler (from Part 2) earned its keep:

• Big connector pod tries to schedule
• Insufficient capacity on current nodes
• Autoscaler detects pending pod
• Launches new worker node
• Pod scheduled onto new node
• Sync completes
• Node deleted (no longer needed)

Workload sizing and infrastructure sizing had to evolve together.

The End State

No heroic over-allocation anywhere. Just a deployment where each component gets what it actually needs, and the infrastructure grows only when work demands it.

The pattern:

• Small syncs → use existing capacity → no scaling
• Large sync arrives → autoscaler launches node → pod runs → node deleted

Predictable, observable, efficient.

The Lessons That Survived

Looking back, the tools mattered less than the principles.

These ten lessons are the ones that earned their place:

1. Design a Platform, Not a Pipeline

Ask: Can another team reuse this? Can it support more data sources?

Optimize for reusability, not one-off solutions.

2. Separation of Responsibilities

• Airbyte — Ingestion only
• Redshift — Storage only
• dbt — Transformations only
• Airflow — Orchestration only
• Backend — APIs only

Violate this boundary, everything breaks together.

3. Separate Runtime From Project Code

• Runtimes — version them (change rarely)
• Code — git-sync it (changes daily)

Fast development + reproducibility.

4. Version Everything

• Runtime images
• Connector images
• Artifacts
• Projects

Upgrades predictable. Rollbacks trivial.

5. Keep Business Logic Out of Orchestration

Airflow schedules. dbt transforms. Backend exposes APIs.

SQL in DAGs = coupled failure points.

6. Secure by Default

• Keep tooling internal
• Mediate through backend
• Restrict admin access

Network architecture, not application complexity.

7. Production Is a Different Planet

Dev hides problems. Production exposes them.

Budget for post-launch tuning phase.

8. Understand the Workload First

OOMKilled? Raising limits is a painkiller, not a cure.

Profile. Decompose. Size deliberately.

9. Build Incrementally

Infrastructure → Ingestion → Orchestration → Transformation → Integration.

Architecture matures instead of big-bang design.

10. Document Decisions, Not Just Code

Reasoning outlasts implementation.

Write down the why as you go.

What We’d Keep, Starting Over

With full hindsight, six decisions survive unchanged:

• ✅ Independent, containerized components on EKS
• ✅ Versioned images in ECR (nothing mutable in production)
• ✅ All transformations centralized in dbt
• ✅ Airflow strictly as orchestrator
• ✅ Ephemeral pods + git-sync for execution
• ✅ Backend as the only public face

If we rebuilt from scratch tomorrow, these stay exactly the same.

Everything else is replaceable.

Previous: Part 2 — From Zero to a Working Pipeline

Start here: Part 1 — Why We Built Our Own Data Platform

This series is based on real engineering work. We’ve avoided throughput, cost, and performance numbers — the architecture and reasoning transfer to other companies; specific metrics don’t.

This article is based on real systems built at SNITCH, an Indian e-commerce fashion platform.