Every bank transfer we process depends on an external system we do not control: the bank. Banks fail, banks get slow, and each one has its own limits. The hard part is not calling a bank, it is making sure one misbehaving bank does not take down transfers to every other bank. This article walks through the four architectures we went through to get there, and why each one was not enough.

The workload

Some numbers to set the scale:

• 3 to 4.5 million transfers / day
• 400 RPS at peak

Correctness and latency both matter. This is money movement and a stalled transfer is a real person waiting.

The failures we design around

Banks are unreliable dependencies. They can fail in four different ways, and each one requires a different response:

• Total & sudden downtime
  Connection refused or 5xx, bank’s endpoint unreachable.
• Tail latency
  The bank accepts the request but hangs. Worst case, a call sits open until our 150s timeout fires.
• Different rate limits
  Every bank and every transfer type tolerates a different request rate. There is no single global limit that fits all of them.
• Burst traffic
  On peak hours or peak days, throughput to a bank spikes to 10 to 15x normal.

The evolution at a glance

We did not arrive at per-lane isolation in one step. Each version fixed the previous bottleneck and exposed the next one.

v1.0 — Synchronous, end to end

The first version was fully synchronous. A customer request travelled straight through to the bank and back on the same blocking call chain, so the customer waited for the bank to respond.

What didn’t work

• Customer latency was the bank’s latency
  The user experience was only ever as fast as the slowest bank.
• A slow bank took down the whole app
  When a bank got slow, request threads piled up waiting on it. Once the thread pool was exhausted, the entire application went down, not just transfers to that one bank.

• Two kinds of traffic, no separation
  We noticed our traffic split into two profiles that were competing for the same threads:
  • Priority
  Sporadic and light, from normal user requests. Needs to finish ASAP.
  • Bulk
  dense and in bursts, from merchant’s scheduled requests or merchant file uploads. These kind of transactions can afford to take time.

v2.0 — Asynchronous, two topics (priority and bulk)

We moved processing off the request path and made it asynchronous, backed by a queue. This version used two Kafka topics, one for priority and one for bulk.

What improved

• Bulk traffic no longer starved priority traffic.

What didn’t work

• Banks share a topic, so a busy bank chokes a quiet one
  Traffic per bank varies a lot. A high-traffic bank hogs the consumer and delays transfers to low-traffic banks on the same topic. And it shifts, bank A is hot today, bank B tomorrow.
• No per-bank throughput tuning
  If we know bank A can handle 100 RPS, we would like to consume its transfers faster. But with every bank in the same queue, speeding up consumption would also overrun banks with lower limits. One knob for everyone.

v3.0 — Asynchronous, a dedicated lane per-bank & per-transfer type

The final version isolates every dimension. We built a standalone generic consumer service that calls a processor service, which calls the bank. Every bank × transfer type × priority/bulk combination gets both its own Kafka topic and its own dedicated consumer service — a fully independent lane.

What changed from the previous version

• From a single service working as consumer, now we have multiple deployments of the same generic service working as consumer for each topics
• To make the most of each consumer, we also have the capability to configure consumer concurrency

For example, Bank A — Intrabank — Priority topic has 36 partitions, the consumer setup would be:

• Pod count: 2, for minimal resource usage
• Consumer concurrency thread count: 18

Usually we only increase consumer concurrency when needed to scale up, because we still have a lot of unused CPU & memory space on each pod.

This is the bulkhead pattern taken to its conclusion:

• Full isolation per lane.
  No bank and/or transfer type can cause problems in another lane. In case of a specific bank’s transfer type having issue, it is fully isolated on its lane

• Fine-grained control.
  We tune throughput independently for every bank, every transfer type, and priority vs bulk. When we have increased traffic for a specific bank’s transfer type, we can scale up only for that lane to accommodate the increased traffic

Results

The clearest metric is queue wait time, how long a transfer sits before a consumer picks it up. Before, that number blew up under exactly the conditions where it mattered most. Now, it is flat.

Takeaway

Isolation has a granularity, and you have to push it down to where failures actually happen.

• Going async (v2.0) stopped bulk from starving priority, but banks still shared the same consumers.
• Only a dedicated lane per bank, transfer type, and priority type contained failure completely.

If you call unreliable dependencies through shared capacity, you have coupled their fate. Bulkhead them at the granularity where they actually fail.

Per-Bank Consumer Lanes: How We Solved Head-of-Line Blocking with the Bulkhead Pattern was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Situation:

For every search keystroke, RTT latency was higher than expected.

Open GoFood, tap on search, and start typing. Suggestions show up as you type (“nas”, “nasi”, “nasi goreng”). That’s type-ahead, and for a feature like this, speed is critical.

However, we noticed that our search type-ahead requests were experiencing significant latency (p95 of about 500ms), even though the core recommendation model needed only about 30ms to return suggestions.

Investigations:

To pinpoint the bottlenecks, we profiled and broke down the full API lifecycle into granular stages. We then ran comparative benchmarks across both low-end and flagship devices to understand which components of the Round-Trip Time (RTT) were bound by network constraints versus local device hardware.

Our investigation highlighted two major areas contributing to the delay:

• Connection Acquisition: Significant device overhead on lower-end hardware just to initiate the request.
• Network RTT: Substantial time spent on the end-to-end network round trip, largely driven by auth token resolution and internal routing redirections.
• Both these stages together contributed to the HTTP latency

This told us where to push.

• Device overhead is real, but it’s local to each phone. There’s only so much app-side work you can shave off.
• The round trip was the opposite: everyone paid for it, on every keystroke, and it was the single biggest chunk. Fix it, and search gets quicker for the whole user base at once.

The Solution:

Long-Lived MQTT Connections

• Reused Connections: Eliminated per-keystroke HTTP handshake overhead by leveraging a single, long-lived connection already open in the app. For this we used MQTT
• Swappable Transport Layer: Decoupled the UI by implementing a swappable transport abstraction. The UI consumes the same data model, whether delivered via MQTT or the legacy HTTP fallback, ensuring safe feature-flagged rollouts and clean A/B testing.
• Simplified Request structure for Courier (MQTT): We chose MQTT because we already use Courier, our in-house MQTT client, and it offered several additional advantages:
• Persistent, Low-Overhead Connection: Swaps noisy, per-keystroke HTTP handshakes for a single, long-lived pub/sub channel — drastically cutting network RTT and connection overhead.
• Bypassed Auth & Routing: Eliminating HTTP removes the need to resolve authentication tokens and handle internal routing redirections on every single letter typed.
• Stripped-Down, Direct Payloads: Removing bulky HTTP headers and dynamic components shrinks download sizes and allows the backend model to consume request data directly with zero preprocessing.
• Minimized Server Latency: The combination of lean payloads, skipped auth checks, and instant pub/sub message delivery maximizes backend response speeds.

The Task: Make it instant without breaking it

Our goal was simple: minimise round-trip latency for instant suggestions. However, execution required meeting several critical guardrails:

• Strict Request-Query Matching: Out-of-order or stale responses (matching a previous keystroke) must be discarded immediately to prevent a confusing UI state.
• Controlled Rollout/Kill Switch: Feature-flagged deployment to a small user subset with real-time telemetry monitoring and instant rollback capability.
• Graceful Degradation: Silent error handling for edge cases, like intermittent network drops or app backgrounding — failures must never surface to the user.
• Empirical Metrics: Rely strictly on hard, before-and-after performance telemetry rather than subjective perception.

Implementation details:

1. MQTT Connection setup: Courier MQTT connection is created when the user opens the Gojek app. For our particular use case, when a user enters the Food Search screen, we create a subscription to a pre-defined topic to listen to server updates.
2. The Request & HTTP Fallback: As user types, the app publishes the search query as a lightweight message over the open MQTT channel.
3. If the channel isn’t connected when the user types, it executes a silent fallback to HTTP, so the user never sees a broken keystroke.
4. Response Compression: The responses from the server are fully compressed to shrink the data footprint and shave extra milliseconds over the air.
5. Deduplication (Latest Query Wins): Network responses can land out of order. Each request carries a unique ID, and the client instantly drops any stale messages — ensuring results for “piz” never overwrite the results for “pizza”.
6. Built-in Timeout Guard: If nothing comes back from the backend within a tight, reasonable window, a timeout triggers a graceful UI state instead of hanging on an endless spinner.
7. End-to-End Telemetry: Every event — from query sent, to response received, to stale response discarded, is logged to mathematically track and prove latency improvements.
8. Foreground-Only Lifecycle: The app actively manages the MQTT connection only when in the foreground. The channel is immediately torn down in the background to prevent unused network overhead.
9. Flag-Controlled Rollout: A remote configuration flag determines the routing path. If active for a user, it tries the new MQTT path; otherwise, it seamlessly defaults to the old, proven HTTP flow.

The architecture at a glance

Here’s the old flow, redrawn from scratch on every keystroke:

And the new one: a single open channel, a per-keystroke choice of path, HTTP as a backstop, and a guard against out-of-order replies.

The Results:

Spike benchmark (5,000 runs per path)

The short version: the real-time MQTT path was about 40% faster on average, faster at both p50 and p90, and its worst case was nowhere near as bad (roughly 1.2 s against 6 s).

Benchmark Insights:

• Consistent Performance: The latency reduction held at p90, proving the optimisation wasn’t inflated by a few outliers.
• Network Resilience: A significantly tighter tail suggests the real-time MQTT path handles degraded network conditions much better than HTTP.
• Test artifacts: The extreme maximum spikes on both paths were caused by OS scheduling overhead within the test framework, and are expected to stabilise on production devices.
• We later did more spike with 10,000 runs for each path to validate the above results. And the patterns held.

🚀 Production Rollout Update (50% Traffic Experiment)

We have successfully run the Courier experiment for a full week (with 5 days at 50% traffic). Based on strong performance and latency wins, we will be rolling out to 100% traffic.

Below are the results

📊 Performance Metrics

💡 Secondary Improvements & User Behaviour

The substantial reduction in latency has directly improved the user experience, leading to higher engagement with the component:

• Increased Engagement: Autocompletion clicks increased by +4.83%.
• Behavioural Shift: Searches originating from suggestions grew by +4.23%
• (shifting from 40% — → 41.72% of total searches).

Lessons learned and what’s next

• Target the Biggest Bottleneck: We optimised the network round trip because it was the most significant, device-independent lag. This fixed the experience universally — especially for users on low-end hardware who were already bottlenecked by local device overhead.
• Persistent Connection Trade-offs: Shifting away from per-keystroke HTTP handshakes doesn’t remove infrastructure complexity; it just moves it. You stop worrying about request overhead and start managing connection lifecycles (handling drops, reconnections, and app backgrounding/foregrounding states).
• Async Request Ordering: Once you move to an async stream, you have to manage out-of-order responses. Tagging payloads and enforcing a “latest query wins” policy to drop stale results is mandatory to keep the UI state accurate.
• Abstracted Transport Layers: Keeping the transport layer swappable was our safety net. Because the UI component didn’t care how data arrived, we could safely hide the feature behind a flag, fall back to HTTP instantly, and run clean production A/B tests.

What’s next: A smarter fallback that retries over HTTP when a real-time attempt actually fails, not just when the channel is down, plus keeping an eye on the production numbers to confirm the benchmark holds up at full scale.

Type-ahead is invisible when it works and annoying when it doesn’t. Getting it off a per-keystroke request and onto an open connection made the fast path the normal path.

“Reducing latency by 40% didn’t just speed up the app — it fundamentally changed how users interact with it.”

Making Search Feel Instant: Moving GoFood Type-Ahead from HTTP to MQTT was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cross-origin iframes in WebView mini apps can’t touch the native bridge same-origin policy blocks them. Most teams work around this with a relay through the main frame. We didn’t. Routing directly to native instead of through JavaScript meant security validation couldn’t be tampered with, and iframes stopped depending on a main frame that might navigate away or crash.

Every GoTo mini app runs inside a WebView. Some of those mini apps embed third-party iframes — a payment checkout here, a partner widget there. And when those iframes need to call a native API, the browser quietly blocks them. For a long time, we lived with a workaround. Then the workaround became the problem.

The problem

When a native application injects a JavaScript bridge object into a WebView, that object lives in the main frame’s window. Every iframe has its own isolated execution context, and cross-origin iframes cannot reach across into the main frame’s properties. A mini app embedding a payment checkout from a partner domain cannot simply call parent.NativeBridge.call() the browser blocks it immediately.

Three factors combine to create this constraint. Each iframe maintains its own JavaScript execution context with an independent window object. The Native bridge injection targets only the main frame — WKUserScript on iOS and @JavascriptInterface on Android. And SOP prevents cross-origin frames from accessing each other’s properties altogether.

Two approaches considered

We evaluated two architectural paths before settling on our final design.

Main frame relay

The first approach routes everything through the main frame. The iframe sends requests via postMessage, the main frame invokes the native bridge, and the response is relayed back. It is straightforward to implement and requires no changes to the native layer.

The problem is coupling. If the main frame navigates away or crashes, iframe bridge access is lost. With multiple iframes, the main frame becomes a bottleneck. Worse, security validation lives in JavaScript — code that can be tampered with — rather than in native code where it cannot.

Isolated iframe bridge (chosen approach)

The second approach establishes direct communication between the native layer and each iframe. The SDK in the iframe sends requests through the standard bridge channel; native sends responses back via postMessage. Security validation happens in native code. Each iframe gets its own permission set. Partners do not need to implement any custom relay logic.

Architecture

The system has three distinct layers inside the native bridge, followed by a postMessage dispatch mechanism and per-frame SDK instances.

End-to-end call flow

A complete bridge call from an iframe happens in two phases: configuration at launch, and the call itself at runtime.

During configuration, the host app fetches mini app metadata from the backend. This includes the main frame permission list and an array of iframe entries, each specifying an origin, its own permission set, and navigation whitelists. These are transformed into a WebKit configuration and passed to the SDK before any content loads. The bridge does not accept runtime configuration changes.

Security model

The three layers each address a distinct threat vector rather than providing redundant checks of the same thing.

The configuration lives on the backend and is fetched at launch time, not embedded in client code. The bridge does not accept runtime configuration changes the permission and whitelist setup is fixed for the lifetime of the session.

Platform differences

The architectural pattern is identical across Android and iOS. Implementation details differ because the platforms expose different APIs for frame context.

Android uses WebViewCompat.addWebMessageListener() from the AndroidX WebKit library, which provides sourceOrigin and isMainFrame in the callback. The legacy @JavascriptInterface is retained for backward compatibility but does not carry origin information, so the JS SDK uses two separate call paths depending on whether postMessage mode is active.

iOS uses WKScriptMessage, which inherently provides WKFrameInfo with the request URL. No API migration was needed on iOS, the same bridge path handles both main frame and iframe calls, and the frame info is available directly in the delegate method.

Engineering challenges

Thread-safe frame context correlation

Bridge calls span multiple threads, incoming messages arrive on the WebView’s callback thread, consent checks may involve background threads, and handler execution happens on the main thread. We use a FIFO queue with atomic operations: a snapshot is stored at bridge entry and consumed exactly once when the handler executes. This guarantees sequential calls are correctly paired with their frame contexts.

Response routing to cross-origin iframes

Native can only run JavaScript via evaluateJavaScript in the main frame context. We inject a small dispatch script that runs in the main frame, iterates direct child iframes, matches the target hostname against each iframe’s src attribute, and calls contentWindow.postMessage() on the matching frame. We traverse only depth-1 iframes, as nested iframes inherit their parent’s configuration.

Concurrent access to event-based handlers

Some handlers, compass, accelerometer, gyroscope, continuously stream data and cannot support concurrent consumers. We enforce a single-session policy: when a handler already has an active session, subsequent registration attempts receive an error. The SDK propagates this to the caller. This keeps resource management simple and predictable.

Backward compatibility

Existing mini apps use a direct callback mechanism where native calls a global window.gpCoreCallback function. Changing this to postMessage would break all existing implementations. The bridge SDK maintains both paths simultaneously. Legacy callbacks continue to work for main frame calls; the postMessage path handles iframe calls. A configuration flag enables postMessage mode automatically when iframe definitions are present, mini apps without iframes see no behavior change at all.

Decision summary

DecisionRationaleAlternative rejectedDirect iframe bridgeNo coupling between frames; security validation in native code where it cannot be bypassedMain frame relay — coupled, unscalable, security logic lives in JSPostMessage for response deliveryOnly viable mechanism for cross-origin frame communicationevaluateJavaScript — cannot target cross-origin framesFIFO queue for origin trackingGuarantees sequential ordering; one-time consumption prevents replayCall ID mapping — adds protocol complexity with no additional security benefitDepth-1 iframe traversalCovers the common case; nested iframes inherit their parent’s configurationFull tree traversal — performance cost for an uncommon scenarioDual callback and postMessage pathsZero migration burden for existing implementationsBreaking change — requires coordinated cross-team migration

Takeaways

Key principles

Frame context must be captured at the earliest possible point in the call flow, before any async operations.

Permission sets must be independently configurable per frame. Inheriting permissions from the main frame defeats the purpose of scoping.

Response routing to cross-origin frames requires a dispatch mechanism that runs in the main frame context — there is no shortcut around this.

The FIFO queue pattern is a simple and effective solution for correlating asynchronous call contexts without adding protocol complexity.

Thanks for reading!

Let’s connect:
🔗 LinkedIn: [https://www.linkedin.com/in/tushar-nikam-dev/]
🐦 X: [https://twitter.com/champ_96k]

Follow for more insights on tech, AI, and product building.

Beyond the Main Frame: Architecture and Implementation of Secure iFrame-to-Native Communication in… was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is how we stumbled upon an artificial ceiling in our gRPC client configurations, the assumptions we challenged, and the tradeoffs we navigated to break through it.

The Hidden Ceiling in Our Legacy Setup

To set the scene, our architecture relies heavily on Istio and Envoy sidecars. When a service makes a call, it uses a fully qualified domain name (k8s FQDN) that resolves to a single service IP address. In this setup, there is a strict 1:1 mapping between a gRPC channel and the underlying TCP connection.

Everything was running smoothly until we started hitting unexplained limits during throughput spikes. Digging into the legacy client code, we found the culprit: a fixed connection pool operating with an HTTP/1.1 mindset. When the application spun up, it created a fixed set of TCP connections. Each outgoing request would “borrow” a connection from the pool, blocking it entirely until a response was received. If a sudden burst of requests arrived and the pool was busy, new requests had to wait in line. If they waited too long, they failed.

We were using gRPC, which is built on HTTP/2, but completely ignoring its superpower: multiplexing. By blocking whole channels per request, we were causing self-inflicted latency and throughput starvation.

Evaluating the Fix: Two Divergent Paths

We knew we had to embrace HTTP/2 multiplexing, which allows multiple concurrent requests to flow over a single connection simultaneously. But how exactly should we architect this interaction with the sidecar? We debated two main approaches.

Path 1: The HTTP/2 Purist (Single Connection)

The most theoretically sound approach in HTTP/2 is to use a single, heavily multiplexed TCP connection. Why pool connections when one pipe can handle it all? Before committing, we had to ask ourselves three crucial questions:

• Will a single connection be reliable enough? Yes. Because this TCP connection is entirely local, established on the loopback interface (127.0.0.1) between the application container and the sidecar on the same Kubernetes node. It is incredibly stable and supports massive bandwidth. The dreaded TCP head-of-line blocking isn’t a practical concern over localhost.
• Will a single connection hit any multiplexing limits? Servers enforce a limit on concurrent streams. Envoy defaults this to 1,024, but Istio configurations typically override this to a massive number (roughly 2 billion). We weren’t going to hit a queueing ceiling.
• What about Envoy proxy concurrency? Here is where the theory hit a wall. Envoy’s threading model is highly opinionated: it assigns each downstream connection to exactly one worker thread.

If our client uses only a single TCP connection, the sidecar handles all of that client’s outbound traffic on a single thread. Effectively, our proxy’s outbound processing becomes limited to the clock speed of a single CPU core.

Path 2: The Pragmatic Middle Ground (Non-Blocking Small Pool)

To solve Envoy’s single-thread bottleneck, we looked at a hybrid approach. Instead of one massive connection, we maintain a very small pool (say, 5 to 10 connections). Crucially, unlike our legacy setup, we don’t block them.

We utilise HTTP/2 multiplexing across all of them, distributing the requests using a random load-balancing strategy. By doing this, we trick the Envoy sidecar into distributing the traffic across multiple worker threads, unlocking multi-core concurrency for outbound requests.

Putting it to the Test

We spun up a test environment, pushing the Istio proxy to its absolute limits to see how the single connection (Path 1) fared against the small pool (Path 2) under heavy synthetic load (using an Envoy concurrency setting of 2). The measured latency is internal processing time in Istio proxy.

The data paints a clear picture. Under normal or moderate loads (sub-500 RPS per pod), both approaches perform beautifully. But as we artificially scaled the throughput into the thousands, Path 1’s single-thread constraint caused latency to skyrocket. Because Path 2 could utilize both of the proxy’s worker threads, its average latency was quite literally cut in half.

The Takeaway

In software engineering, everything is a tradeoff. The pure, single-connection approach is incredibly simple and solves the blocking issue for 99% of micro-services, as very few individual pods actually sustain thousands of outgoing RPS without scaling out first.

However, for high-throughput gateway nodes or extremely dense aggregators, that Envoy threading model constraint is a silent killer. In these contexts, Path 2, the pragmatic, non-blocking multiplexed pool is the preferred choice. It balances the architectural simplicity of HTTP/2 multiplexing with the raw reality of how proxy sidecars manage CPU threads under the hood.

References

1. gRPC docs: Performance Best Practices
2. gRFC: MAX_CONCURRENT_STREAMS Connection Scaling
3. Envoy docs: Life of a Request
4. Envoy blogs: Threading model
5. IETF RFC 9113: HTTP/2
6. TCP head of line blocking

The Sidecar Bottleneck: Rethinking gRPC Connection Pooling was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

When approaching any problem, organizations naturally gravitate toward defining the technical solution — moving methodically through the stages of the Software Development LifeCycle (SDLC). While the nature of problems may vary across business contexts and technical domains, one constant remains: chaos. This chaos stems from a fundamental human instinct — different stakeholders approaching the same problem with different perspectives on how it should be solved.

Every initiative involves a diverse set of stakeholders — end users, implementation teams, execution teams, and supporting functions. Each group views the problem through its own lens, shaped by priorities, constraints, and success metrics. As a result, alignment on the “right” solution or approach is rarely immediate. What emerges instead is fragmentation — multiple interpretations, competing priorities, and misaligned expectations.

This fragmentation often evolves into organizational chaos, becoming a significant barrier to delivering initiatives effectively while staying aligned with broader business goals.

At GoTo, the challenge is amplified by a fast-moving, high-autonomy environment where multiple business units operate with tightly coupled dependencies. Rapid product growth has led to a complex web of services and siloed datastores. In this setup, even small changes create ripple effects across teams. With work happening in real time across shared channels, differing perspectives and shifting priorities are inevitable. The result is a constant state of controlled chaos — where misalignment on ownership, effort, or business impact can slow execution and dilute outcomes.

The TPM Compass: Two Case Studies in Structural Navigation

At GoTo, Technical Program Managers (TPMs) are the people who help bring structure to complex technology programs, especially when initiatives span multiple business units, systems, and stakeholder groups. Their role is not just to manage technical efforts, but to drive cross-functional programs where alignment is often the hardest problem to solve.

In this complexity, TPMs act as a compass — bringing structure to ambiguity, aligning diverse perspectives, and ensuring that execution stays true to the program’s goals. In fast-moving environments, misalignment can quickly lead to missed timelines, fragmented outcomes, or diluted impact. TPMs mitigate this by creating clarity on priorities, dependencies, ownership, and sequencing — before and during execution.

At GoTo, this means going beyond coordination. TPMs actively shape how teams collaborate, anticipate points of conflict, and put mechanisms in place to ensure that progress across one group does not come at the cost of another.

The following case studies highlight how TPMs at GoTo navigated large-scale, high-stakes challenges — turning potential chaos into structured, predictable execution.

Navigating the Digital Jungle: One Night Cloud Cutover

On 26 May 2025, GoTo successfully completed the cloud migration of its ODS systems to Tencent Cloud — one of the most complex infrastructure transformations undertaken within the organization.

The Challenge: High-Stakes Migration at Scale

This was a high-stakes operation involving a planned 5-hour shutdown of Gojek services. Over 600 engineers across Indonesia, India, China and Singapore collaborated to migrate nearly 500 services and more than 1,000 datastores. With minimal margin for error, success depended not just on technical execution, but on precise coordination across hundreds of interdependent tasks.

The TPM Approach: Orchestrating Precision and Alignment

At the core of this effort was a well-defined orchestration layer, ensuring seamless alignment across all participating teams.

While engineering teams defined their individual execution steps, TPMs owned the end-to-end coordination of the migration. They designed and managed a detailed runbook that aligned all teams on sequencing, dependencies, and timing — ensuring that no step blocked another.

The work began several weeks before the cutover. TPMs drove multiple integration drills to surface gaps, resolve cross-team dependencies, and iteratively refine the execution plan. These drills helped teams work through complex scenarios with greater clarity, enabling teams to align on timelines, validate assumptions, and commit to well-defined execution expectations.

During the actual cutover, TPMs facilitated real-time coordination and decision-making — monitoring progress, managing handoffs, and ensuring seamless traffic switching and service recovery on the new cloud environment.

The Outcome: Synchronized Execution at Scale

The result was a highly coordinated and predictable execution. The migration was completed one hour ahead of schedule, with Gojek services restored smoothly, ready to serve millions of users at the start of the new day.

This initiative demonstrated how TPMs transform large-scale, high-risk programs into structured, synchronized outcomes by bringing clarity, alignment, and control to even the most complex systems.

Breaking Barriers: How TPM Drove a Cross-Entity Program

In a high-stakes environment involving external partners, language barriers, and asymmetric information sharing, the role of a TPM shifts from “tracking tasks” to “defining the path forward.”

The Challenge: Asymmetric Visibility and Information Silos

Recently, a program required teams at GoTo to integrate with a third-party entity. Unlike standard programs focused solely on Product and Engineering, this initiative involved a complex web of stakeholders, including Legal, Finance & Accounting, and Driver Care Operations.

Operating with an external partner introduced a “transparency paradox”: We required full visibility into their progress to ensure success, yet were restricted in the proprietary data we could share from our end. When these constraints were combined with things literally getting “lost in translation,” the risk of misalignment was high.

The Approach: TPM as the Strategic Bridge

Following a rigorous initial workshop, it was clear that TPM had to wear multiple hats to transition the program from planning to high-velocity execution. The following pillars were established for our ways of working.

• Unified Documentation & Visualization: Comprehensive Program Boards were built, that visualized tasks across all departments. By ensuring every flow, decision, and timeline was documented and acknowledged by both parties, we created a “single source of truth” that kept stakeholders updated in real-time.
• Cross-Functional Orchestration: Focused cadences were orchestrated specifically to close open questions between GoTo and the partner. Beyond the technical build, TPM also helped with the GTM (Go-To-Market) strategy with non-technical teams, ensuring Legal and Finance were aligned on payout flows and reconciliation before the first line of code was deployed.
• Precision Execution via Runbooks: Given the sensitivity of the flows, a detailed Deployment & Validation Runbook was created. This provided a step-by-step sequence for engineering teams on both sides to follow, including real-time validation checkpoints. Having an appointed PIC for each step gave the organization the confidence to move forward with a high-risk launch.

Results: From Chaos to BAU

The execution of all outlined tasks and steps leading to a successful launch helped establish a new Business As Usual (BAU) framework. By defining clear owners, Standard Operating Procedures (SOPs), and structured communication loops, we transformed a chaotic, multi-entity space into a repeatable, organized system.

Ultimately, this program proved that when complexity is high and communication is strained, a TPM’s greatest value is in building the structure that makes collaboration possible.

The Road Ahead for TPMs: From Coordination to Intelligent Orchestration

As our systems grow more interdependent, the role of a TPM is shifting. At a high-scale organization like GoTo, TPMs don’t just manage projects, they manage a network of dependencies.

They are moving away from mere coordination and toward intelligent orchestration at scale. This requires a fundamental mindset shift and, more importantly, their toolkit.

1. The Shift to Pre-Execution Rigor

In large-scale initiatives, such as the massive cloud migration, success is often won or lost before the first line of code is moved. TPMs are now doubling down on pre-execution planning and rigor to ensure that the “chaos” is controlled. This involves:

• Ownership Clarity: Establishing ironclad accountability models so no service falls through the cracks.
• Measurable Success: Defining “done” and “successful” upfront with quantifiable metrics.
• Drills: Running structured simulations to find breaking points early.
• Resilience by Design: Building fallback and recovery strategies as a core part of the plan, rather than an afterthought.

2. The AI Accelerator

One of the most promising accelerators in this journey is the adoption of AI-driven capabilities. In an environment where decisions are scattered across hundreds of threads and documents, AI acts as a force multiplier.

By leveraging AI to augment our daily operations, we can:

• Synthesize Signal from Noise: Summarize cross-team discussions and surface hidden dependencies that manual tracking might miss.
• Proactive Risk Detection: Flag execution plan risks before they become blockers.
• Automated Transparency: Ensure stakeholders are informed at desired frequency without manual intervention, maximizing clarity while minimizing the effort

While chaos is an inherent byproduct of any large-scale system especially within a diverse ecosystem like the GoTo group, the goal of a TPM isn’t to eliminate it. Instead, they aim to continuously evolve how we navigate and structure that entropy.

TPM — The Cultural Architect

In the complex web of a digital ecosystem, team interests will inevitably collide. The Fintech team wants ironclad security; the Growth team wants frictionless onboarding; the Platform team wants to deprecate the legacy database everyone is still using. If a TPM waits for agreement — where everyone is “happy” with the decision — the project will stall in a loop of endless stakeholder reviews.

Alignment, however, is different from agreement. It means that while a team might not agree that a specific path is the absolute best, they understand why it was chosen, they believe the process was fair, and they are committed to making it successful.

Ultimately, the future of the TPM role is as much cultural as it is technical. They are facilitators of alignment over agreement. In an organization such as ours, teams will naturally have differing perspectives. Their job is to ensure that, despite these differences, every team can move forward cohesively with shared clarity on the final goals and the trade-offs required to get there.

As organizations continue to scale, this ability to orchestrate — rather than control — will become even more critical. And that is where TPMs will continue to evolve: not just as program managers, but as architects of clarity in an increasingly complex world.

Orchestrating Chaos — How Technical Program Managers (TPM) Enable Scalable Collaboration was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

A practitioner’s guide to building relationship-based access control for multi-tenant platforms - the architecture, the schema design, the pitfalls, and the hard-won lessons from production.

You know that moment when your access control system starts fighting you?

It starts small. Someone needs a permission that doesn’t fit neatly into any existing role. So you create a new one. Then another team needs a slight variation. Another role. Then someone asks: “Can we make it so Team Lead X can see reports for their team but not the team next door?” Suddenly, you’re staring at a database table with hundreds of roles, half of which no one remembers creating, and a check_permission function that's 400 lines of nested if-statements held together by comments like // TODO: fix this properly.

We’ve all been there.

This is the story of how we replaced that system with something fundamentally different - a Zanzibar-style, relationship-based access control system built on SpiceDB, Go, and PostgreSQL. Not a toy, but a production system handling multi-tenant organizational hierarchies, fine-grained resource permissions, ABAC conditions, and UI-level component visibility checks.

This article focuses on practical implementation details and trade-offs rather than providing a formal evaluation of every available system.

Part 1: Why RBAC Breaks (And When You’ll Notice)

Role-Based Access Control is a reliable, well-understood baseline for authorization. It works extremely well - until it doesn’t.

RBAC tends to break along three axes:

The Hierarchy Problem

Imagine a platform where each customer (tenant) has their own organizational tree:

An administrator at the Company A level should automatically have management permissions over every subsidiary, business unit, department, and team below them. In RBAC, you’d either:

• Assign them a role on every single entity (nightmare to maintain)
• Create a “Company A Admin” role with hardcoded access to everything below (breaks when the hierarchy changes)
• Build custom hierarchy-traversal logic in your application (now you’re maintaining a permissions engine)

None of these scale well.

The Granularity Problem

Users need different access to different parts of the same resource. User A can view a ticket but not the payment widget on it. User B can view the ticket and the payment widget but can’t close the ticket. User C can do everything - but only for tickets belonging to their team.

In RBAC, each unique combination of resource + action + scope becomes a new role. This is the classic role explosion problem. 50 resources × 5 actions × 10 organizational scopes = 2,500 roles. And you haven’t started on exceptions yet.

The Context Problem

“This resource can only be modified when its status is resolved.”

“This user can only access resources during business hours in their timezone.”

“This temporary promotion expires in two weeks.”

RBAC has no native vocabulary for conditions, time-bounds, or contextual attributes. These rules often end up embedded in application logic and scattered across microservices, each with its own slightly different interpretation of what ‘admin access’ means.

Part 2: Zanzibar - Thinking in Relationships

In 2019, Google published a paper describing Zanzibar - an authorization system used across many of its products, including Drive, YouTube, and Cloud. The core idea:

Much of modern access control can be modeled as a graph problem.

Instead of “Does this user have this role?”, you ask: “Does a path exist in the relationship graph from this user to this resource that grants this permission?”

Three building blocks:

1. Schema - The Type System

The schema defines entity types, their relationships, and how permissions are computed:

definition user {}

definition folder {
    relation parent: folder
    relation owner: user
    relation editor: user
    relation viewer: user

permission edit = owner + editor + parent->edit
    permission view = owner + editor + viewer + parent->view
}

definition document {
    relation parent_folder: folder
    relation owner: user

permission edit = owner + parent_folder->edit
    permission view = owner + parent_folder->view
}

This says: If you’re an owner of a folder, you can edit and view every document in it - and every document in every subfolder below it. Permission inheritance emerges from the graph structure, not from application code.

2. Relationship Tuples - The Data

Each tuple is a simple fact: resource#relation@subject.

folder:engineering#owner@user:alice
folder:backend#parent@folder:engineering
document:design-doc#parent_folder@folder:backend

Three tuples. From these, the system derives that Alice can edit and view design-doc - because she owns engineering, which is the parent of backend, which is the parent folder of design-doc.

3. Permission Checks - The Graph Traversal

Permission check: one query, one graph traversal, typically single-digit milliseconds

Part 3: Choosing a Zanzibar Implementation

Google never open-sourced Zanzibar itself, but several implementations exist. We evaluated three:

We also evaluated higher-level frameworks built on top of Zanzibar implementations, but ultimately rejected them. They abstract away the schema language, which means you lose features like permission exclusion (permission view = editor - banned), caveats for ABAC, bulk checks, and multi-level consistency control.

Our Benchmarks

Important: All benchmarks below were run on our internal Kubernetes setup with our production schema and K6 as the load testing tool. These results are directional, not definitive. Your numbers will differ based on schema complexity, tuple count, hardware, and query patterns. Always benchmark your own workload.

We tested with 1M and 10M relationship tuples, at 1,000 and 10,000 queries per second, sustained for 5 minutes each. Server configuration: 4 CPU cores, 4 GB memory. Database: 2 CPU, 4 GB memory (PostgreSQL).

1M Relationships, 1,000 QPS

1M Relationships, 10,000 QPS

10M Relationships, 10,000 QPS

Storage Footprint

Our takeaway: OpenFGA showed lower average and median latency in several scenarios. SpiceDB showed tighter tail latency (P90/P95) and significantly lower max latency under high load. We chose SpiceDB because consistent tail latency mattered more for our use case, and the native distributed caching plus ZedToken consistency guarantees aligned with our multi-node deployment model.

Part 4: Our Architecture

The system decomposes into four bounded domains, each with its own responsibilities, database schema, and API contract:

High-level architecture: the BFF is the enforcement point, delegating to IAM for auth and SpiceDB for permissions

The BFF (Backend-for-Frontend) is the primary enforcement point. Every request flows through it:

1. Authenticates - Validates the JWT, checks session status
2. Authorizes - looks up the permission mapping for the endpoint and calls SpiceDB
3. Filters - After getting the response, strips fields the user isn’t allowed to see

Authorised Request Flow

Sequence diagram: every request is checked against SpiceDB before reaching backend services

Domain 1: Authentication

Handles how users prove their identity. Completely decoupled from authorization.

• Password auth with Argon2id hashing (m=65536, t=3, p=1), lockout policies, and rotation
• OIDC via Authorization Code + PKCE (Google Workspace, Okta, Azure AD, any OIDC provider)
• MFA with TOTP (authenticator apps) and WebAuthn/FIDO2 (hardware keys, biometrics)
• Session management with short-lived JWT access tokens (~15 min) and rotating opaque refresh tokens (~7 days)
• Identity linking - one user, multiple authentication providers, single canonical identity

Password Login Flow

Password authentication flow with Argon2id verification and session creation

OIDC Flow (Authorization Code + PKCE)

OIDC Authorization Code + PKCE flow with id_token verification and identity upsert

Session Lifecycle

Session lifecycle: sessions are explicitly tracked with revocation support

Key design decisions:

• Personas vs. Roles: Personas drive authentication (how you prove who you are). Roles drive authorization (what you’re allowed to do). An “admin” persona might require MFA; an “admin” role grants management permissions. Don’t conflate them.
• Tenant-scoped IdPs: Each tenant configures their own identity providers. Tenant A uses Google Workspace, Tenant B uses Okta, Tenant C uses username/password. The auth service supports all simultaneously.

Domain 2: Organization Hierarchy (Tenants)

The structural backbone. Manages tenants, organizational units, and role definitions.

• Tenants - The top-level namespace. Each tenant is fully isolated.
• Organizational Units - A recursive tree within each tenant. Uses PostgreSQL LTREE for efficient hierarchical queries.
• Roles - Defined within a tenant. Each role is a named collection of permission relations, scoped to an organizational unit level.

Tenant Lifecycle

Tenant lifecycle: Created → Active → Deactivated → Archived

Org Unit Lifecycle

Org unit lifecycle with merge and split as first-class terminal states

Merge and split are first-class concepts. When two departments merge, the old units transition to a terminal “merged” state; a new unit inherits their members and permissions, preserving full audit history.

Domain 3: User Management

User management is not authentication. A user exists independently of how they log in. You can create a user, assign them roles, and configure their profile before they ever authenticate.

User lifecycle with suspension support for temporary disablement

• Users belong to tenants (strict isolation via tenant_id)
• Users are assigned roles within specific organizational units
• Role assignments can have expiration dates (interim promotions, temporary access)
• All mutations publish domain events for downstream consumers

Domain 4: Authorization

The SpiceDB integration layer. Policy meets enforcement.

• Schema management - Versioned SpiceDB schemas deployed via CI/CD
• Relationship synchronization - When tenants/units/users/roles change, SpiceDB tuples are written
• Permission checking - gRPC calls to SpiceDB with proper consistency levels
• Permission mapping - YAML configuration connecting endpoints and UI components to SpiceDB checks

Resource Creation Flow

Resource creation: SpiceDB tuples are created after the backend resource, with Kafka compensation on failure

Part 5: The Schema - Where Everything Comes Together

This is the most important part. Get the schema wrong and everything downstream suffers.

Modeling Roles as Entities

In most RBAC systems, roles are just strings in a database column. In our Zanzibar model, roles are first-class entities with their own relationships:

definition user/human {}

definition role/human {
    relation bound_user: user/human

// What permissions this role grants
    relation tenant_manager: role/human
    relation org_unit_manager: role/human
    relation user_manager: role/human
    relation role_assigner: role/human
    relation resource_viewer: role/human
    relation resource_editor: role/human
    relation widget_executor: role/human
    // ... more relations as needed

// Derived permissions
    permission tenant_manage = tenant_manager->bound_user
    permission org_unit_manage = org_unit_manager->bound_user
    permission resource_view = resource_viewer->bound_user
    permission resource_edit = resource_editor->bound_user
    permission widget_execute = widget_executor->bound_user
}

To create a role, assign a user, and grant it at an org unit:

// "team_lead" can manage org units and view resources
role/human:team_lead#org_unit_manager@role/human:team_lead
role/human:team_lead#resource_viewer@role/human:team_lead

// Assign Alice to the role
role/human:team_lead#bound_user@user/human:alice

// Grant the role at the engineering org unit
tenant/org_unit:engineering#granted@role/human:team_lead

The role is composable (mix any permissions), tenant-scoped (each tenant defines their own), and inspectable (query SpiceDB’s schema reflection API to list all permissions).

Permission Inheritance Through the Hierarchy

definition tenant/space {
    relation granted: role/human
    relation root_space: tenant/space

permission resource_view = granted->resource_view
                             + root_space->resource_view
    permission org_unit_manage = granted->org_unit_manage
                               + root_space->org_unit_manage
}

definition tenant/org_unit {
    relation granted: role/human
    relation parent_org_unit: tenant/org_unit
    relation tenant_space: tenant/space

permission resource_view = granted->resource_view
                             + parent_org_unit->resource_view
                             + tenant_space->resource_view
    permission org_unit_manage = granted->org_unit_manage
                               + parent_org_unit->org_unit_manage
                               + tenant_space->org_unit_manage
}

The + operator means union. Permissions flow through three paths:

1. Direct grant - A role granted directly on this org unit
2. Parent inheritance - A role granted on any ancestor org unit (walks up recursively)
3. Tenant-level - A role granted at the tenant space level

One tuple at the tenant space level cascades permissions to every org unit below

Resource-Level Access Control

definition resource/ticket {
    relation granted: role/human
    relation org_unit: tenant/org_unit
    relation owner: user/human
    relation excluded: user/human

permission view = owner
                    + granted->resource_view
                    + org_unit->resource_view
                    - excluded

permission edit = owner
                    + granted->resource_edit
                    + org_unit->resource_edit
                    - excluded

permission close = org_unit->org_unit_manage
                     - excluded
}

There are four key things to notice here:

1. Multiple permission paths: Own it, have a direct role grant, or inherit via org unit hierarchy.
2. Hierarchy integration: org_unit->resource_view traverses the entire ancestry chain.
3. Exclusion: The - excluded operator surgically revokes access for specific users despite broad grants.
4. Action-specific scoping: close requires org_unit_manage (managers only), while view is broader.

ABAC via Caveats

SpiceDB supports conditional permissions through caveats:

caveat ticket_is_resolved(ticket_status string) {
    ticket_status == "resolved"
}

definition resource/ticket {
    relation owner: user/human with ticket_is_resolved
    relation org_unit: tenant/org_unit

permission escalate = owner
                        + org_unit->granted_escalate
}

At check time:

// Allowed — ticket_status is "resolved":
resource/ticket:123#escalate@user/human:bob  with {"ticket_status": "resolved"}

// Denied — ticket_status is "new":
resource/ticket:123#escalate@user/human:bob  with {"ticket_status": "new"}

RBAC and ABAC, unified in a single model. No separate policy engine. No Rego. One schema, one check.

Part 6: The Implementation

Tech Stack

• Language: Go
• API: gRPC with Protocol Buffers (12 service definitions)
• Databases: PostgreSQL (operational data), SpiceDB backed by PostgreSQL (authorization)
• Caching: Redis
• Events: Kafka (audit logs, domain events, compensation events)
• Auth: JWT (ES256), opaque refresh tokens, Argon2id, WebAuthn/FIDO2, TOTP
• Infra: Kubernetes, Helm, Docker

The Relationship Manager

type RelationshipManager interface {
    WriteRelationship(ctx context.Context, tuples []Tuple) error
    DeleteRelationships(ctx context.Context, filter Filter) error
    AddResource(ctx context.Context, resource, orgUnit string) error
    RemoveResource(ctx context.Context, resource string) error
    AddOwner(ctx context.Context, resource, user string) error
    ChangeOwner(ctx context.Context, resource, oldUser, newUser string) error
    BindUserToRole(ctx context.Context, user, role, orgUnit string) error
    CheckBulkPermissions(ctx context.Context, items []CheckItem) ([]Result, error)
}

Every state change flows through here. Create a resource? AddResource + AddOwner. Assign a role? BindUserToRole. Transfer ownership? ChangeOwner.

Permission Mapping: Connecting APIs to SpiceDB

- name: get_ticket
  prefix: /api/v1/ticket_service
  path: /tickets/{ticket_id}
  method: GET
  authz:
    - name: get_ticket_permission
      permission: view
      subject_type: "user/human"       # from JWT or X-User-ID header
      object_type: "resource/ticket"
      object_id:
        value: ticket_id
        source: path_params            # extracted from URL

The BFF reads this mapping, extracts the ticket ID from the URL, the user ID from the JWT, and calls SpiceDB: resource/ticket:123#view@user/human:current_user_id. No authorization logic in backend services. One mapping file, one enforcement point.

Bulk UI Permission Checks

Bulk permission checks: 20 items in one round trip, SpiceDB evaluates in parallel

// Frontend usage
<RequiredPermission has={["can_edit_ticket"]}>
  <EditButton />
</RequiredPermission>

<RequiredPermission hasAny={["can_refund", "can_close"]}>
  <ActionMenu />
</RequiredPermission>

Components the user can’t access simply don’t render. No flickering, no “unauthorized” errors after clicking.

Part 7: PostgreSQL LTREE + SpiceDB - A Powerful Duo

The organizational hierarchy lives in two places, by design:

PostgreSQL (source of truth for what exists):

CREATE TABLE units (
    id              TEXT PRIMARY KEY,
    tenant_id       TEXT NOT NULL REFERENCES tenants(id),
    hierarchy_path  LTREE NOT NULL,
    name            TEXT NOT NULL,
    slug            TEXT NOT NULL,
    type            TEXT NOT NULL,  -- company, subsidiary, department, team
    status          TEXT NOT NULL CHECK (status IN (
        'created','active','deactivated','archived','merged','splitted'
    )),
    metadata        JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_units_hierarchy ON units USING GIST (hierarchy_path);

-- All descendants of engineering:
SELECT * FROM units WHERE hierarchy_path <@ 'acme.engineering';

-- Direct children only:
SELECT * FROM units WHERE hierarchy_path ~ 'acme.engineering.*{1}';

SpiceDB (source of truth for who can access what):

Database schema: tenants, units (with LTREE hierarchy), and roles (with JSONB permission relations)

PostgreSQL gives you fast subtree queries, ordering, and metadata. SpiceDB gives you fast permission inheritance. Each system does what it’s best at - you can’t SELECT * FROM org_units WHERE status = 'active' ORDER BY name on SpiceDB.

Part 8: Schema Versioning and Migration

SpiceDB schemas evolve over time. Our approach uses versioned directories:

schema/
  v0.0.0/schema.zed    -- Initial schema (base definitions)
  v0.0.1/schema.zed    -- Added note and log viewing permissions
  v0.0.2/schema.zed    -- Added dashboard and report viewer roles
  v0.0.3/schema.zed    -- Added resource listing permissions

Safe migrations (always backward-compatible):

• Adding new relations to existing definitions
• Adding new permissions
• Adding new subject types to existing relations
• Adding entirely new definitions

Contingent migrations (require data cleanup first):

• Removing relations (must delete all tuples using that relation first)
• Removing subject types from relations

Dangerous migrations (avoid in production):

• Renaming relations (there’s no rename - it’s delete + create)
• Changing permission expressions (can instantly grant or revoke access)

CI/CD pipeline for schema changes:

1. Diff preview - /v1/schema/diffschema to see what will change
2. Schema write - /v1/schema/write to apply
3. Verification - Diff again to confirm

Every schema change is deployed separately from application code - no coordinated big-bang releases.

Part 9: The Hard Lessons

1. Schema Design Is the Hardest Part

We spent more time iterating on the SpiceDB schema than writing the entire Go service. The schema is your authorization model’s type system. There’s no “we’ll fix it in the application layer.”

Our biggest debate: should UI widgets be relations on the role (flat, specific) or standalone resource entities (generic, flexible)?

// Option A: Widget as role relation (what we chose)
relation widget_payment_viewer: role/human
permission widget_payment_view = widget_payment_viewer->bound_user

// Option B: Widget as resource
definition resource/widget {
    relation granted: role/human
    permission view = granted->widget_view
    permission execute = granted->widget_execute
}

Option A: Schema is the source of truth, no separate registry needed, but every new widget requires a schema migration. Option B: New widgets are just tuple insertions, but you need a widget registry in your service layer. Pick based on your rate of change.

2. Dual-Write Consistency Is Non-Trivial

Creating a user and assigning a role means writing to PostgreSQL and SpiceDB. If one fails, you have inconsistency. Our approach: PostgreSQL first, SpiceDB immediately after, Kafka compensation events on failure, periodic reconciliation job.

3. Benchmark Your Actual Schema

Published benchmarks use simple schemas. Our schema has 80+ role relations, 4 hierarchy levels, and 5+ relationship traversals per check. Always benchmark with your actual schema, realistic tuple counts, and realistic query patterns.

4. Caveats Require Discipline

If the caller doesn’t provide context, SpiceDB returns CAVEATED (a maybe). We treat it as DENIED by default and monitor for unexpected occurrences.

5. Single Schema Per Instance

All tenants share one SpiceDB schema. This works if tenants differ in data (which roles exist) but not in structure (the permission model itself).

6. Don’t Forget Response Filtering

Authorization isn’t just “can they call this endpoint?” It’s also “which fields in the response are they allowed to see?” We strip forbidden fields at the BFF layer before sending to the frontend.

Part 10: The Numbers

• 12 gRPC services: authentication, authorization, user management, tenant management, roles, sessions, MFA, identity providers, personas, API resources, UI resources, email verification
• 16 domain modules in Go with clean model/service/store separation
• 80+ permission types modeled in SpiceDB
• 4 schema versions deployed via CI/CD (additive, backward-compatible)
• ~3.8ms median permission check latency at 1,000 QPS with 10M tuples
• Sub-5ms P95 at moderate load on 4 CPU / 4GB SpiceDB

Should You Build This?

Yes, if:

• You have multi-tenant customers with complex organizational hierarchies
• Your permission model includes inheritance (team → department → company → tenant)
• You need fine-grained access control on individual resources
• You need both role-based and attribute-based checks
• Your frontend needs to know what the user can see before rendering
• Audit and compliance matter

No, if:

• You have a single tenant with flat roles (admin, editor, viewer)
• Your authorization is “is the user logged in? they can do everything”
• You don’t need hierarchical permission inheritance
• You’re a small team and shipping speed matters more than permission granularity

The investment is real: schema design, dual-write consistency, SpiceDB operations, migration planning. But the payoff is an authorization system that scales with your organizational complexity instead of fighting against it.

Zanzibar-style authorization, when designed carefully, is one of the few approaches where the system becomes easier to reason about as complexity grows.

That’s the whole point.

Credits :

Engineers — Muhammad Gian Ansyori, Muhammad Fawwaz Naabigh (Caretech Gojek)

Project Lead and Management — Ravi Raj (Senior Engineering Manager — Gojek)

This article describes the architecture and implementation of a production IAM system built on SpiceDB, Go, PostgreSQL, and Kafka. The system handles multi-tenant organizational hierarchies, relationship-based access control, OIDC/password/MFA authentication, and UI-level permission checking.

,

We Built a Zanzibar-Style IAM System From Scratch. Here’s Everything We Learned. was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

You know the feeling. You buy something online, you hit the button, and then you just stare at the screen. You are waiting for some kind of sign that the system actually registered your order.

At Gojek, we serve over 170 million users. That means millions of people are staring at their phones, waiting for groceries on GoMart. In the beginning, our initial MVP for the post-booking flow was completely bare-bones. We just dropped users onto a map with a little dot representing a driver.

And then they would sit there. Watching a dot that refused to move.

It is a recipe for high anxiety. Users constantly wondered where their order was. People get nervous, assume something is broken, and ultimately cancel their orders. We had no live progress steppers, no clear driver location, and no exact ETA. The legacy design failed to highlight our biggest unique selling point: 15-minute delivery. It was time for a complete teardown.

Stage 1: Rebuilding the Foundation and Adding a Safety Net

Maria, our Product Designer, stepped in and completely re-architected the layout. She brought in Gojek’s clean “Aloha” design system, which uses a boxed architecture to organize complex data. Out went the ambiguous screens, and in came clear progress steppers to answer exactly where the order was.

She mapped out five different realities for the delivery:

- Packing
- On the way
- Early
- On-time
- Delayed.

Working closely with our UX writer Nadia to make the text sound like a real person was talking to you. A crisp green layout meant things were on time. A cool blue screen let you know it was raining outside. A warm orange screen offered a gentle apology if the driver was running late.

We also added a clear “Cancel Purchase” button with a ten-second timer right on that first screen. People often panic immediately after ordering because they realize they made a mistake. Giving them a ten-second countdown gave them a safe window to fix those mistakes before the order processed.

Stage 2: Replacing Spec Sheets for Real Motion

Getting the foundation right was great. But the way those screens moved into each other had to feel entirely natural. Complex movements require precise timing.

Usually, designers hand engineers a bunch of static spec sheets and hope for the best. As the designer leading the interaction and UX motion for this project, I decided to optimise that process. Instead, I swapped them for high-fidelity, interactive ProtoPie Recipes. I mapped out all the bottom-sheet interactions, the drag gestures, the flings, UX motion for maps and exactly how the top bar elements responded to a touch.

I handed the engineers a living prototype. They could see the exact physics and easing of every transition, making our collaboration much smoother. It worked so well that we officially adopted ProtoPie for all our functional motion requirements.

Stage 3: The Million-Dollar Question: Map or Animation?

Those initial changes; Maria’s architecture, the conversational copy, the ProtoPie interactions, and the safety net completely changed user behavior and led to an 8% overall reduction in order cancellations.

But we still had a problem with the map.

Showing a live map while an order is just sitting in a warehouse being packed is very expensive. Ballooning Google Maps API costs for 170 million users is a massive issue. However, looking at a stationary dot is stressful for users.

We had a hypothesis: we wanted to know if state-driven Lotties or a Live map would better reduce anxiety among users.

We ran an A/B test in our warehouse areas. We kept the map for half the users. For the other half, we replaced it with custom Lottie animations.

This is where the magic of collaboration really happened. Our incredibly talented illustrator, Beny Styawan crafted beautiful, warm illustrations featuring our characters and our friendly cat mascot. She set the scene, and then I gave it life. By animating those illustrations, I created a continuous, comforting loop of progress. Together, we transformed a sterile waiting screen into a story.

The Verdict: Motion as a Functional Tool

We wanted to see if a friendly animation could beat a GPS dot.

It did.

Lottie animations beat the live map variant, driving an additional 0.9% drop in cancellations. We saved massive Map API costs by replacing maps with Lottie animations. The data was so clear that we rolled the animations out to 100% of our millions of active users.

People often talk about motion design as if it is just decoration. But it is much more than that. Behind the mood boards and the technical physics of a ProtoPie recipe is a simple desire to care for the person on the other side of the screen.

It is a way to show empathy.

It tells the user that the system is working for them so they can relax. When we replace uncertainty with warmth and movement, we build real trust. That sense of delight is not just a “nice to have.” It drives the business forward.

Thoughtful motion impacts everything from engagement and clicks to long term retention and revenue. At the end of the day, design is about solving complex problems for the company while staying deeply human for the user. That is what it is really for.v

The team

Product designer: Maria Grace Sandra
Motion design: Jayant Mandan
Product Manager: Nayantara
UX Writer: Nadiya Amalina
Illustrator: Beny Styawan
Engineering (Tech) : Naman Jain, Deep Mistry
Engineering (Frontend): Jeffrey Immanuel, Rajat, Dhruvil Rameshbai
QA: Pandu Kusumadewi, Felicia Aurelia
Blog Cover Illustration: CA Team

Why We Killed the Live Map: Using Motion to Cure Post-Booking Anxiety was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

5 years in product design to motion design intern

It’s the year 2020. I had just jumped ship to an under-construction motion design team at Gojek after spending five years of my career in product design. I opened After Effects once and decided to quit. But Abhinit (OG head of design, Gojek), convinced me to stay with the best advice he ever gave me:

“Keep. At. It.”

Alright, let’s do this one more time.

Two more weeks of tinkering with After Effects and YouTube tutorials, and I had successfully failed at creating a basic button click animation. I decided I should either convince Abhinit to shut down the motion design team altogether, or quit Gojek myself, rather than risk my career and reputation as a good designer.

As I was plotting my escape, Aniket (my product designer friend) walked up and said, “Bro, we can’t figure out how to help our users feel less anxious while waiting for us to find them a driver. Wanna try solving it with motion?” My immediate thought: Oh no. Oh yes! Oh no. Wait - let’s go!

The excitement of having the opportunity to approach a design brief through a completely new lens, along with Jean’s (my motion design teammate and mentor) eerily optimistic confidence that I’d be able to pull this off (what was she thinking), made me shake hands with Aniket and set the project rolling.

But here’s the thing about being blissfully ignorant of your own limitations - it’s weirdly liberating. I didn’t know what I didn’t know about technical constraints, implementation complexity, or the Herculean effort our engineering team would need to put in. I just started sketching like an excited kid building a fantasy world.

The turning point came when we stopped worrying about getting the animation right, and started focusing on what our users were actually feeling.

The Problem: When Waiting Feels Like Forever

There was a certain segment of users who used to drop-off when the waiting period during the finding driver stage stretched beyond five seconds. We wanted to take a look at why this was happening and try to address the issue in the best way possible.

The Old Experience: The Endless Loop of Despair

We had a basic looping animation - a pulsing graphic that repeats the same 2-3 second cycle. Forever. With no changes. No updates. Just… loop. Loop. Loop.

What would be going on in a user’s mind?

• “Is anything actually happening, or are they just showing me a placeholder?” (Valid question when nothing changes for 30 seconds)
• “They’re not even trying to find me a driver.” (When the animation gives zero indication of progress)
• “Let me check other apps real quick…” (And there goes the booking)

This wasn’t something we could solve with competitive pricing or discount codes. This was about the psychological experience of waiting. The anxiety. The impatience. The sneaking suspicion that nothing’s actually happening behind the scenes.

We needed to make the waiting period feel active, transparent, and engaging.

The Design Process: Explorations, Rejections, and Reality Checks

I explored 5-10 different iterations before we landed on the final solution.

What We Rejected (And Why)

Too Playful/Complex: Some early ideas were envisioned to be visually stunning but communicated nothing. We had to remember: this isn’t about showing off. It’s about reducing anxiety and keeping users engaged.

Performance Killers: Complex animations could drain battery life and/or slow down the phone’s performance - a critical concern when someone’s desperately trying to book a ride and might already be at 10% battery.

API Cost Bombs: We explored showing actual real-time driver positions on the map. Cool idea, right? Turns out it would’ve required constant API calls, increased server costs, drained user batteries, and slowed down phone performance. All for information that didn’t actually help the user.

Engineering nightmare: Motion design lives or dies in implementation. We had a strong engineering team, but bandwidth is finite - and every animation decision carried a real cost. The question was never whether we could build it, but whether the impact for our consumers justified the effort. Finding that balance between design intention and engineering investment became one of the defining constraints of the project, and ultimately shaped the final animation as much as any creative decision did.

The balancing act was brutal: How do we make this engaging without being annoying? How do we show progress without lying to the user? How do we stay technically performant while being visually interesting?

The Insight: Three Stages of Hope

After multiple brainstorming sessions, diving into past data, and making sense of user behaviour patterns, we got on to something :

What if we broke down the waiting experience into three distinct stages, each with its own animation and messaging?

The logic was - when you’re waiting for something important, your anxiety isn’t linear. It evolves:

• 0–5 seconds: “Okay, this should be quick…”
• 5–15 seconds: “Hmm, taking a bit longer than expected…”
• 15+ seconds: “WHAT IS HAPPENING. ARE THEY EVEN TRYING?”

So we designed for that emotional journey.

Stage 1 (0 - 5 seconds): “We Got This”

Animation: An intentional, communicative animation that feels like diving into the map with the user - like we’re literally going on a hunt together to find the nearest driver.

Why it works: The majority of our users get a driver during this stage. For them, this is all they ever see - a confident, purposeful search that resolves quickly. No room for anxiety.

Stage 2 (5 - 15 seconds): “We’re Working On It”

This is where things get interesting.

Animation changes:

• Multiple driver markers appear on the map and start animating
• The pulse rate subtly increases (faster = more urgent search)
• The arrow inside the pickup pin starts animating upward - like it’s raising its hand to get attention
• Everything moves dynamically to show active searching, not stagnation

Why it works: The animation evolves. Users can see that something different is happening. We’re not just showing them the same loop - we’re visually communicating that the system is escalating its search.

Stage 3 (15+ seconds): “We’re Going Full Throttle”

Now we’re in edge-case territory. Only a small percentage of bookings reach this point, but we couldn’t abandon these users. They’re the ones who need us most.

Animation changes:

• The map zooms out - showing a wider search radius
• The pickup pin starts jumping up and down - like someone frantically waving their arms for attention
• The movement is inspired by human body language: desperation, urgency, determination

Why it works: We’re being honest (“yeah, this is taking a while”) while also showing empathy and effort. The jumping pin is a visual metaphor - it’s us, desperately trying to flag down any driver in the vicinity. It tells a story with minimal cognitive load.

The user might be frustrated, but at least they know we’re still fighting for them. Or be informed that this search might take longer than usual.

The Dream Team: Why This Worked

I can’t take credit for this alone. Not even close.

This project was a masterclass in cross-functional collaboration:

• The product and business team believed motion design could drive impact and gave us the runway to explore.
• The product designers helped break down the flow into stages and kept us grounded in user behaviour.
• UX Writers crafted messaging that was honest, reassuring, and concise.
• The Design System team and UI designers streamlined the UI to show only essential information during the wait
• The engineering team implemented the animations (Lottie/JSON + native code) flawlessly across platforms, ran spike tests to ensure performance, and went through multiple iterations for handcrafting the UI transitions with patience and precision.
• The QA team stress-testing every scenario to make sure the animations didn’t break under edge cases.
• Creative design and illustration teams for crafting the assets
• My motion team-mates and Abhinit who guided and gave me the confidence to pull this off.

Without their conviction that motion design could add value to this flow, none of this would have happened. As the pilot project driven by motion design at Gojek, they were the ones who helped convince stakeholders that investing bandwidth, effort and resources into this would bring results, even if it meant the sprint would get longer.

The Launch: Measuring What Matters

We ran a staged rollout : starting with a small percentage of users, then expanding based on results.

The Test

• Duration: Over a month of data collection
• Metrics: Cancellation rate during the “finding driver” stage
• Hypothesis: If we make the waiting period more transparent and engaging, it will help mitigate waiting anxiety and users will be less likely to cancel.

The Result

19% reduction in cancellations during the finding driver stage.

We didn’t change the pricing. We didn’t add discounts. We didn’t magically find more drivers. We just made the waiting experience better through motion design.

What This Proved: Motion Design Isn’t Just “Polish”

Before this project, motion design at Gojek was… let’s say, experimental. Nice-to-have. The cherry on top after the “real work” was done.

This project changed that narrative.

It proved that:

• Motion design can drive measurable business impact - not just “delight,” but actual revenue-affecting metrics.
• Motion design should be part of product development - not an afterthought, but a strategic tool in the design process.
• Motion design is worth the investment - the time, the effort, the engineering bandwidth

This project paved the way for motion design to grow into a full-fledged central team at Gojek. What started as a scrappy pilot became proof that this discipline deserves a seat at the table.

The Lessons: What I’d Tell My Past Self

Not knowing the rules is an advantage - use it while it lasts. I designed freely because I didn’t know what was “impossible,” and some of the best ideas came from that ignorance.

Animation that doesn’t communicate something earns its removal. Every motion in this project had a job. If it didn’t do that job, it didn’t ship.

Human metaphors cut through complexity fast. A pickup pin waving for attention works because desperation is universal. When system states behave like people, they stop needing explanation.

Constraints are just design briefs you didn’t write. Battery limits, performance budgets, API costs - each one pushed us toward something leaner and smarter than the original idea.

And nothing here was a solo effort. Engineers made it fast. Writers made it clear. Product made it count. QA made it solid. I just got to see it move.

Final Thoughts: Motion Design is Product Design

If there’s one thing I want you to take away from this, it’s this:

Motion isn’t about making things pretty. It’s about making things understandable and intuitive.

When a user is anxious, waiting, uncertain - motion can communicate what words alone cannot. It can show progress and build trust. It can turn a frustrating moment into a reassuring one.

At its core, this was never just about animation, it was about solving a very real user problem.

And sometimes, if you’re lucky and have an incredible team, it can reduce cancellations by 19%.

The team

Motion design lead: Binoy Cyriac
Motion design support team: Jean D’Souza, Avadh Dwivedi
Product designer: Aniket Yadav
Product Manager: Ifad Ardin
UX Writer: Alverta Shani
Illustrator: Beny Styawan
UI: Isfarial Reihan R
Engineering (Tech) : Anshuman Srivastava, Rajeev Singh
Engineering (Frontend): Aditya Ladwa (Android), Anshul Vyas (IOS)
QA: Prasanta Biswas
Blog cover illustration: Zhafira Millenia

About the Author: I’m a designer with 11 years of industry experience, including 8 years at Gojek. I embrace being a generalist - because why specialize when you can grow branches in multiple directions? Currently heading the creative and motion design team at Gojek, but always a humble student of design.

Let’s connect - https://www.linkedin.com/in/binoy-cyriac/

How a Revamped ‘Finding Driver’ Animation Reduced Cancellations by 19% for Gojek was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author: Shivangi Parashar

Introduction

At Gojek’s Observability platform (Lens), more than 50,000 alerts fire every day across 40+ engineering teams running 500+ microservices on Kubernetes/VM’s. Each one demands the same mechanical investigation: open Grafana, pull logs from Loki, check pod state in kubectl, and cross-reference recent deploys in the cloud. A senior engineer who owns the service can close the loop in 25–30 minutes. An on-call engineer at 3 AM, paged for a service they didn’t write, takes considerably longer.

The Math is simple: even after aggressive noise filtering, hundreds of alerts each day still need human review. At 25–30 minutes per alert across a 40+ team organization, that consumes a significant share of engineering capacity on work that is mechanical and repetitive, while often requiring limited human judgment.

Challenges

We built a system that does it in 1–2 mins.

What it is: a multi-agent AI platform that receives an incoming alert, autonomously investigates across Prometheus metrics, Loki logs, Tempo traces, Kubernetes cluster state, and deployment history — simultaneously, in parallel — and hands the on-call engineer a structured, evidence-backed root cause analysis via Lark before they have finished reading the alert title.

What it is not: a dashboard, a query assistant, or an auto-remediation tool. It never touches production. Its job is to compress the mechanical portion of incident investigation — the tab-switching, the context-gathering, the cross-referencing — to near zero. Human judgment, escalation, and remediation stay exactly where they belong.

This is how we built it.This is the full engineering story of how we built it.

The Architecture: Six Layers of Intelligence

Two feedback loops exist: Layer 6→3 when context is insufficient, and Layer 2→1 for human escalation. More on both shortly.

The entire workflow is a typed state machine, implemented as a LangGraph StateGraph. Every node reads from and writes to a shared ObservabilityState TypedDict. List fields use append semantics so parallel agents can write results without overwriting each other.

class ObservabilityState(TypedDict, total=False):
    user_query: str
    agents_to_call: List[str]
    intent: Dict[str, Any]         # {"parallel": bool, "confidence": float}
    time_range: str
    promql_results: List[Dict[str, Any]]
    logql_results:  List[Dict[str, Any]]
    k8s_results:    List[Dict[str, Any]]
    traceql_results: List[Dict[str, Any]]
    agent_summaries: Dict[str, Any]
    final_answer: str
    path: Annotated[List[str], add]  # append-only execution trace
    # Resolved dynamically per team/alert
    prometheus_datasource_uid: str
    loki_datasource_uid: str
    tempo_datasource_uid: str

The path field — typed with Annotated[List[str], add] — is critical for debugging. Every node appends its name. After a run you can reconstruct exactly which agents executed, in what order, and whether the feedback loop triggered.

Layer 1: Entry and Context Resolution

Alerts arrive from Grafana webhooks as a POST to /api/v1/alerts/analyze. The response is a Server-Sent Events (SSE) stream — the client gets a live feed of agent progress in real time, not a blocking wait for a final answer.

@router.post("/analyze")
async def analyze_alert(request: AlertAnalysisRequest) -> StreamingResponse:
    return StreamingResponse(
        _stream_analysis(request),
        media_type="text/event-stream"
    )

Events are typed: PROGRESS, AGENT_DATA, FINAL_ANSWER, and ERROR. The frontend renders each as it arrives. By the time the Correlation Agent finishes, the user has already watched each domain agent's findings stream in one by one — which dramatically improves perceived latency even when total wall-clock time is unchanged.

Context Resolution: The Missing Link

This is the layer most observability tools skip entirely, and where the system earns a large part of its intelligence.

An alert payload contains labels like alertname, severity, namespace, and sometimes a team or app label. But to actually investigate the alert, you need to know: which Grafana datasource has Prometheus data for this team? Which one has Loki logs? Which one has Tempo traces?

The Context Node resolves all of this before a single query runs:

async def _resolve_datasource(team_name: str):
    datasources = [
        ("Prometheus", f"{team_name.lower()}_tencent"),
        ("Loki",       f"{team_name.lower()}_logs"),
        ("Tempo",      f"{team_name.lower()}_traces"),
    ]
    for ds_type, ds_name in datasources:
        ds = await grafana_client.get_datasource_by_name(ds_name)
        if ds:
            resolved[ds_type] = ds.get("uid")

The resolved UIDs flow into ObservabilityState.It uses state["prometheus_datasource_uid"] — resolved specifically for the alerting team, seconds ago, from a live Grafana lookup.

Layer 2: The Orchestrator — LLM as a Router

The core architectural bet we made early: let the LLM decide which agents to invoke and in what order, instead of writing routing rules.

The naive approach would be: if "cpu" in query → promql_agent. We could have built this in an afternoon. We didn't, for good reason: it doesn't scale, it can't handle compound queries, and every new agent requires updating a growing decision tree.

Instead, the Router Node sends the user’s query plus a structured description of every available agent to the LLM, which returns:

{
  "agents": ["promql", "k8s"],
  "confidence": 0.92,
  "rationale": "CPU spike suggests Prometheus metrics; pod restarts suggest K8s state",
  "parallel": true,
  "time_range": "now-2h"
}

The parallel flag is the most operationally significant field. When the LLM decides two agents are independent — fetching metrics and checking pod state simultaneously — we run them with asyncio.gather():

async def execute_agents_node(state: ObservabilityState):
    if parallel_mode:
        tasks = [agent_funcs[name](state) for name in agents_to_call]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        # Merge results with explicit append semantics

For a query that triggers PromQL + LogQL + K8s simultaneously, parallel execution cuts wall-clock time from roughly 15 seconds to under 6 seconds.

When we added the TraceQL agent for distributed tracing, the router learned to invoke it from natural-language context alone. Zero changes to routing code.

Layer 3: The RAG Pipeline — Grounding LLM Queries in Reality

This is where most AI observability tools fail, and where we spent the most engineering effort.

The problem: LLMs hallucinate Prometheus metric names. A model generating PromQL will produce syntactically valid queries that reference metrics like container_cpu_usage instead of your actual metric container_cpu_usage_seconds_total{container!="POD"}. The query returns no data. The engineer gets a useless answer.

Worse: even correct metric names have environment-specific label values. A model might generate {namespace="payments"} when your actual label is {app_namespace="payments-v2"}. Same problem.

The solution is a multi-stage RAG pipeline that grounds the LLM in your actual metric catalog before it generates a single character of PromQL.

Stage 0: Offline Ingestion — Building the Metric Schema Index

We built a custom ingestion script that crawls your Prometheus environment and extracts a structured schema for every metric:

High-cardinality labels (like pod_name or instance) are flagged rather than fully enumerated — we don't want 10,000 pod names polluting the embedding. The output is two artifacts:

• metric_schema_structured.json — full schema with all metadata
• metric_schema_rag_docs.json — 22,457 embedding-ready text documents (~8MB)

Each document looks like this:

Metric: container_cpu_usage_seconds_total
Labels:
- container (examples: api-server, worker, sidecar)
- namespace (examples: payments, auth-service, data-pipeline)
- pod (HIGH CARDINALITY — do not match by value)
- cluster_name (examples: prod-cgk-01, prod-sin-01)

This format is deliberately human-readable. We embed the same format the LLM will see as context, which improves retrieval alignment significantly.

Why Qdrant: The Vector Database Decision

We evaluated Pinecone, Weaviate, Chroma, pgvector, and Qdrant. The decision came down to four factors:

1. Hybrid search out of the box. Qdrant supports combining dense vector similarity with BM25 full-text search via Reciprocal Rank Fusion (RRF) in a single query. For metric names, this matters enormously. A user asking for “HTTP error rate” matches semantically (dense) to http_requests_total{status=~"5.."}, but also needs BM25 to surface metrics that literally contain the string "error." Semantic search alone misses exact matches; keyword search alone misses synonyms.

query_response = self._client.query_points(
    collection_name=collection_name,
    prefetch=[
        Prefetch(
            query=query_vector,      # Dense: 384-dim BGE embedding
            using="text-dense",
            limit=limit,
        ),
        Prefetch(
            query=query_text,        # Sparse: BM25 over metric_name field
            using="metric_name",
            limit=limit,
        ),
    ],
    query=FusionQuery(fusion=Fusion.RRF),  # Reciprocal Rank Fusion
    score_threshold=score_threshold,
)

2. Filtering by datasource. We have two separate metric corpora: Lens metrics and Events Platform metrics. Each Qdrant point carries a datasource payload field. When the context node resolves that a query targets the Lens datasource, the PromQL agent scopes retrieval automatically:

rag_filter = Filter(
    must=[FieldCondition(key="datasource", match=MatchValue(value="lens"))]
)
results = await rag_helper.search_metrics(query=query, query_filter=rag_filter)

3. Performance at our scale. Our metric catalog has roughly 50,000 unique metric names across two datasources. Qdrant handles this with sub-50ms query latency at p95 for top-10 results. The BGE-small-en-v1.5 embedding model (384 dimensions) gives us a good trade-off between retrieval quality and index size.

4. Operational simplicity. Qdrant runs as a single binary or Docker container. For an internal platform, we don’t want to manage a distributed vector database cluster. Qdrant’s simplicity matches our operational constraints.

Stage 1: Online Enrichment — Live Label Values

RAG gives us the metric name and example label values from index time. But label values drift. A cluster added last month won’t appear in a schema indexed two months ago.

The PromQL Agent addresses this with live enrichment: after retrieval, it queries Prometheus directly for current label values on the retrieved metrics:

async def _enrich_with_live_labels(self, rag_results):
    for result in rag_results:
        metric_name = result["metadata"]["metric_name"]
        labels = result["metadata"].get("labels", [])

        if not labels:
            # RAG had no label info — discover from Prometheus
            labels = await self._discover_label_names(metric_name)

        if labels:
            live_values = await self._discover_label_values(metric_name, labels)
            result["metadata"]["label_values"] = live_values

_discover_label_values uses asyncio.Semaphore(3) to query Prometheus for up to 3 labels concurrently, with a cap of 10 values per label. The result: the LLM sees not just that a metric has a namespace label, but that right now, in your production environment, that label has values ["payments", "auth-service", "data-pipeline", "checkout"].

Stage 2: Validation — Cutting Hallucinations at the Knees

Even with grounded context, LLMs sometimes add labels that look plausible but don’t exist. We validate every generated query against live Prometheus data before executing it:

async def _check_label_exists(self, metric_name, label, op, value):
    check_expr = f'count({metric_name}{{{label}{op}"{value}"}})'
    result = await self.grafana_client.call_tool(
        "query_prometheus",
        {"expr": check_expr, "queryType": "instant", ...}
    )
    has_data = PromQLErrorHandler.check_has_data(result)
    return label, op, value, has_data

All matchers in a query are validated in parallel. Invalid ones are silently dropped and the query is rebuilt:

if valid:
    new_query = query.replace(matchers_str, ", ".join(valid))
else:
    new_query = re.sub(r"\{[^}]*\}", "", query)  # Strip all matchers

In practice, this catches wrong assumptions on roughly 20–30% of queries in environments with non-standard label naming conventions.

Layer 3: The Agent Fleet

Six specialized agents, each owning a data source domain:

PromQL Agent handles Prometheus metrics — CPU, memory, latency, request rates, error rates, throughput. It runs the full RAG pipeline described above and generates 3–4 related queries per request to build a multi-dimensional view of the system.

LogQL Agent handles Loki logs — error messages, stack traces, application events. It follows a similar LLM-driven query generation pattern adapted for LogQL syntax and log-volume constraints.

TraceQL Agent handles Tempo distributed traces — request latency percentiles, slow spans, error traces, end-to-end request flow. It connects to the same Grafana MCP, using the Tempo datasource UID resolved by the context node.

K8s Agent handles Kubernetes cluster state — pod status, recent events, resource limits and requests, deployment history, node conditions. Uses the Kubernetes MCP, which wraps kubectl operations.

GoCloud Agent handles our internal deployment platform — recent deploys, rollbacks, current running versions. Invoked asynchronously when the orchestrator suspects a deployment-related issue.

DMS Agent handles apps deployed in vms. Provides context on whether recent schema migrations or pipeline deployments correlate with the observed issue.

The critical design constraint across all agents: no agent takes action on production. Every agent is read-only. The platform investigates; humans decide.

Layer 4: MCP Integration

All data access goes through Model Context Protocol (MCP) servers — an open standard for connecting AI agents to external tools via a typed, discoverable interface.

We run five MCP servers:

Grafana MCP wraps the Grafana HTTP API. The PromQL, LogQL, and TraceQL agents all connect to this single server. It handles Prometheus queries, Loki queries, and Tempo queries as separate tools parameterized by datasource UID. This is a critical architectural point — it’s one Grafana MCP, not separate Prometheus MCP, Loki MCP, and Tempo MCP servers. The datasource UID on each call determines which backend is hit.

Kubernetes MCP wraps kubectl. The K8s agent uses it to fetch pod state, events, describe resources, and check recent deployments.

Arsenal MCP is our internal service registry. Used exclusively by the Context Node to resolve team ownership, application metadata, and datasource names from alert labels.

GoCloud MCP / DMS MCP are internal platform tools for deployment and pipeline metadata.

Layer 5: The Feedback Loop

After all agents complete, the Correlation Agent assembles their summaries and sends them to the LLM with a synthesis prompt:

sections = [
    f"User Question: {state['user_query']}",
    f"Metrics Analysis:\n{summaries['promql']['analysis']}",
    f"Logs Analysis:\n{summaries['logql']['analysis']}",
    f"Kubernetes Analysis:\n{summaries['k8s']['analysis']}",
    f"Deployment Context:\n{summaries['gocloud']['analysis']}",
]
prompt = CORRELATE_PROMPT_TEMPLATE.format(context="\n".join(sections))

After the Correlation Agent produces its answer, we pass it through a Sufficient Context validator. If the correlation is “I don’t have enough information to conclude” — which happens when agents returned no data, or datasource UIDs couldn’t be resolved, or the alert doesn’t match any known metric patterns — the graph loops back to the Orchestrator

The re-routing pass has the previous attempt’s results in state. The orchestrator can decide to reinvoke which agents

How We Reduced Hallucinations

How we reduce hallucinations :

1. RAG grounding — retrieve real metric names and label values from Qdrant before the LLM generates anything. Nothing gets invented from thin air.
2. Live label enrichment — after retrieval, query Prometheus for current label values so the LLM sees real production data, not stale index snapshots.
3. Post-generation validation — every label=value matcher in generated PromQL/LogQL is verified against live Prometheus before execution. Invalid ones are silently dropped.
4. DeepEval faithfulness gate — scores every agent response inline. If faithfulness < 0.7, Reflexion fires for that agent only. One pass restores quality in 78% of cases.

We Built an Observability Platform. Then We Had to Observe It.

Every LLM call flows through a token tracker that exports to our OTLP collector:

class TokenUsage:
    provider: str           # "gemini", "ollama", "openai"
    model: str
    prompt_tokens: int
    output_tokens: int
    total_tokens: int
    operation: str          # "query_generation", "synthesis", "routing"
    metadata: Dict[str, Any]

We export five metrics via OpenLIT:

llm.tokens.total      Counter   — total tokens across all operations
llm.tokens.prompt     Counter   — prompt tokens (primary cost driver)
llm.tokens.output     Counter   — output tokens
llm.calls.total       Counter   — LLM invocations per operation type
llm.tokens.per_call   Histogram — distribution of tokens per call

Breaking down by operation has been critical for cost optimization. We discovered that query_generation (the PromQL agent generating 3–4 queries) consumed 40% of total tokens, and that the RAG context was the primary driver. Tuning _MAX_LABELS_IN_CONTEXT = 10 and _MAX_SAMPLE_VALUES = 3 in the context formatter reduced prompt tokens for that operation by 35% with no measurable quality regression.

System Execution Metrics

Each analysis result is persisted with full execution metadata:

execution_metadata = {
    "analysis_time_seconds": elapsed,
    "agents_called": list(agents_to_call),
    "routing_path": state["path"],
    "feedback_loop_triggered": "re_route" in " ".join(state["path"]),
    "parallel_execution": state["intent"].get("parallel", False),
}

The metrics that matter most in production:

• Analysis latency p50/p95/p99 — broken down by agent count. Single-agent queries (pure Prometheus) run in ~4s. Multi-agent parallel runs average ~8s. Sequential multi-agent runs average ~14s.
• Feedback loop trigger rate — currently 8%, target is <5%. Improvement comes from better datasource coverage and more complete Arsenal registration.
• Agent data yield rate — what fraction of agent invocations return actual data versus “no results.” Low yield on a specific agent signals a misconfigured datasource.
• RAG recall quality — what fraction of retrieved metrics actually appear in the final generated PromQL queries. Currently tracking at ~72% inclusion rate, which is acceptable given the validation step drops some retrieved metrics as irrelevant.

Conclusion

The core insight that drove all of this: on-call engineers don’t have an information problem. They have a synthesis problem. The data is all there — in Prometheus, in Loki, in Tempo, in your K8s cluster. The work is correlating it across systems, under pressure, at 3 AM. That is exactly the kind of work that should be done by a machine.

We built the system.

https://vimeo.com/1186878395

How We Built an AI Brain for Our Observability Stack At Gojek was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine standing in front of a giant gate. On the other side are millions of users: drivers, consumers, and merchants, waiting for that gate to open. Once you unlock it, there’s no pulling anything back. No undo button. Every mobile release carries that quiet pressure.

At Gojek, releasing an app is never just about submitting a build to the Play Store or App Store. It’s the point where weeks of engineering decisions finally meet real-world usage — across different platforms, regions, and user expectations.

As our ecosystem scaled across Driver, Consumer, and Merchant apps, releases naturally became more complex. More teams were contributing, more features were landing every week, and more dependencies had to line up at the right time.

Recently, with our move toward weekly mobile releases, that complexity has become even more visible.

Shipping more frequently brought speed, but it also forced us to look closely at our fundamentals — ownership, planning, automation, and post-release visibility.

As a key part of the Release Experience Team, I’ve collected lessons from this journey. Here’s everything, what helped us make mobile releases more predictable, safer, and easier to operate at scale.

The challenge

Mobile releases are different from backend deployments in one important way: once an app version is out, you live with it.

Users don’t update instantly. Issues can’t be patched in minutes. Rollbacks are slow and painful. At scale, even a small mistake can impact millions of users.

As teams grow, some common patterns start to appear:

• Releases depend on a few people who hold all the context
• With multiple products involved, teams are unsure what exactly is going out in each version
• Last‑minute changes sneak in with good intentions
• Monitoring starts only after users complain
• Each release feels stressful, even when nothing goes wrong.

Agree here?

Well, weekly releases simply amplify these problems.

What we learned is simple: speed alone doesn’t fix release pain, process does.

Making releases predictable with planning

One of the first things we learned is that releases can fail long before release day. They fail when teams don’t have clarity. For a long time, our release planning was in spreadsheets — detailed ones with freeze dates, testing milestones, rollout stages, and formulas doing impressive math behind the scenes. That worked well. But as apps and teams scaled, spreadsheets stopped scaling with us. Today, release planning happens through a dedicated in-house release management portal that brings visibility across platforms and regions. It helps track:

• Release timelines and milestones
• Testing and regression progress
• Bug trends and Automation Stability

Explaining more about this portal will take up another entire blog, which we will be dropping soon. For now, let’s realize that the first step towards a successful release is the planning of the release itself.

From “When should we release this feature?” to “What are we releasing today?”

Dedicated release ownership

As release frequency increases, coordination becomes just as important as engineering quality.

Having clear release ownership turned out to be one of the biggest stabilizers for us.

The role of a release manager isn’t about pressing the publish button. It’s about maintaining the overall picture:

• Aligning engineering, QA, product, design, and operations
• Tracking risks early in the cycle
• Protecting the release from late instability
• Acting as a single source of truth during rollout

Without ownership, releases often become shared responsibility, which in practice means fragmented communication.

With ownership, decision-making becomes faster and expectations stay aligned.

At the same time, it’s important that this knowledge doesn’t live with just one person. Strong documentation and backups ensure the process remains resilient as teams evolve.

Measuring success beyond “did it crash?”

Not all releases fail loudly. Some quietly degrade performance. Some increase crash rates slightly. While, some introduce issues that appear only days later. This is why defining release KPIs is important.

Metrics like:

• Crash‑free sessions
• ANR rate
• Hotfix frequency
• Rollout rollback count
• Time from code-freeze to app submissions

These signals help teams understand whether the release process is actually healthy. Used correctly, metrics highlight trends, not individuals. They help us improve continuously instead of reacting emotionally after incidents.

Release trains: shipping on schedule, not on pressure

One practice that significantly improved stability was adopting release trains.

The idea is simple:

The train leaves every week. If a feature isn’t ready, it waits for the next one.

This approach removes pressure to squeeze half‑tested features into a release just because the date is near. Teams gain confidence that another release is always around the corner. The result is better quality, fewer hotfixes, and much calmer release days.

Automation: reducing human error

Publishing a mobile app is rarely a single-click activity.

Between generating builds, signing artifacts, uploading binaries, configuring staged rollouts, and monitoring health, there are multiple points where small mistakes can have large impact.

Over time, we invested heavily in mobile CI/CD and release automation.

Large parts of the workflow, from build generation to rollout percentage increases, are now driven through pipelines instead of manual console actions. Including alerts and comms on Release Channels.

This brought us some instant benefits:

• Faster and repeatable rollout steps
• Reduced dependency on console-level permissions
• Improved security and auditability
• Significantly fewer human errors

Automation doesn’t remove humans completely, and it shouldn’t. Decision points still matter. But by letting systems handle repetition, teams are able to focus on other areas of improvement. Treating pipelines as first-class products — monitored, owned, and continuously improved — made weekly releases sustainable.

Feature flags: releasing without fear

One of the strongest safety nets in mobile releases is feature configuration.

Using feature flags allows teams to:

• Merge code safely
• Ship features turned off
• Enable functionality gradually
• Instantly disable problematic flows

This decouples deployment from exposure. In real scenarios, this has helped us handle high‑priority issues without interrupting user journeys, simply by toggling a feature off while fixes were prepared for the next release.

The trade‑off is discipline.

Unused flags and unclear ownership can create long‑term debt. Regular cleanup and documentation are essential. When managed well, feature flags dramatically reduce release anxiety. And you guessed it right, we have an internal tool developed to handle it too: CONfig-MANager

Releasing more often to move faster — Weekly Cadence

One of the hardest lessons for release managers is learning to say ‘no’ to last‑minute changes. Even well‑intended changes introduce instability. Frequent releases make that conversation easier. With weekly releases:

• Features can safely move to the next cycle
• Releases become smaller and easier to validate
• Risk is distributed instead of concentrated

Smaller releases are easier on engineers — and easier on users too. Interestingly, frequent updates also improve user perception. Regular app updates signal active development and allow feedback to flow more continuously.

Monitoring after the gate opens

A release doesn’t end when the app goes live. That’s when real usage begins. Automated monitoring helps teams understand what’s happening in production:

• Crash trends
• Performance degradation
• API failure spikes
• Startup latency

Without monitoring, teams rely on store reviews or support tickets which usually arrive too late. The challenge is balance. Too many alerts create noise, while too few create blind spots. Finding that balance turns releases from reactive to proactive. Having a clear SOP with crash/ANR budgets becomes a game changer for Release Managers to confidently call out when to halt/stop release, in case of spike numbers.

Mobile releases will never be completely risk-free. But they don’t have to be stressful.

Across Driver, Consumer, and Merchant platforms, the biggest lesson has been this: stability doesn’t come from releasing less, it comes from releasing better. Strong ownership, clear planning, predictable schedules, automation, safe rollout mechanisms, and honest metrics together create confidence.

Weekly releases didn’t make things harder.

They simply exposed where the structure was missing and gave us the opportunity to build it. The goal isn’t perfect releases. The goal is predictable, calm, confidence-driven releases — the kind that quietly works in the background while teams focus on building meaningful product improvements.

We’re still learning. We’re still refining. And with every release train that leaves the station, the system gets a little stronger.

If you’re working on mobile platforms at scale, these learnings may look familiar. And chances are, the best improvements not come just from moving faster — but from making releases easier to trust.

Mobile Release Best Practices: What We Learned While Scaling Across Apps was originally published in Gojek Product + Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.