Engineering a Multi-Region RideShare Platform on AWS
Evolving a single-region EKS application into a production-oriented platform using managed data services, delivery automation, observability, and platform engineering principles
There is a point in many Kubernetes projects where “it runs” stops being a useful definition of success. The RideShare application had reached that point.
The platform already looked cloud-native on the surface. It consisted of independently deployable services running on Amazon EKS. Helm managed the Kubernetes resources. NGINX Ingress handled north-south traffic, cert-manager automated TLS, External Secrets Operator synchronized credentials from AWS Secrets Manager, and Horizontal Pod Autoscalers and Pod Disruption Budgets supported availability.
PostgreSQL ran inside the cluster under CloudNativePG, while Redis ran under the Redis Operator. The application worked, and the architecture was technically credible.
But it was still a single-region system with a large operational surface area concentrated inside one Kubernetes cluster. The same team operating the application was also responsible for the database pods, cache pods, persistent volumes, operators, failover behaviour, upgrades, backups, and recovery procedures. A regional or cluster-level failure could take down the entire platform.
This project builds on the earlier implementation, which I documented in Building a Production-Grade RideShare Platform on Kubernetes: Lessons from Running Microservices.
This article covers the original single-region EKS platform and examines its evolution into a multi-region architecture built around managed AWS services, delivery automation, observability, and operational controls.
The goal of the project therefore became larger than deploying more infrastructure. It became:
Engineering the evolution of a cloud-native RideShare application from a single-region Kubernetes deployment i.e a single-region EKS with CloudNativePG and the Redis Operator to a production-oriented, multi-region platform using Amazon RDS, Amazon ElastiCache, automation, observability, operational controls, and Platform Engineering principles.
This article is the story of this evolution, the trade-offs made, the problems encountered, the evidence demanded before calling a phase complete, and the moments when the safest engineering decision was to stop rather than keep typing commands.
The original platform was working begging the question: why migrate to managed services?
The original system ran in a single EKS cluster in `us-east-2`. Its services included rider, driver, trip, matching, email, and frontend workloads. PostgreSQL was managed by CloudNativePG with data persisted on Amazon EBS. Redis was managed by a Kubernetes operator. The application used WebSockets for real-time matching behaviour and exposed APIs through NGINX Ingress.
Nothing was fundamentally wrong with this design. In-cluster data services can be a valid choice when a team deliberately accepts the operational responsibility and has the skills, automation, and recovery discipline to support them.
That was the first important trade-off. I was not replacing CloudNativePG or the Redis Operator because Kubernetes operators are inherently bad. I was changing the allocation of responsibility.
With CloudNativePG, Kubernetes reconciled the PostgreSQL topology, but I still owned the consequences of storage failures, database upgrades, backup reliability, failover behaviour, node disruption, and recovery testing. With the Redis Operator, I gained declarative lifecycle management, but also retained responsibility for Redis clustering, persistence, upgrades, pod placement, and operator health.
Amazon RDS and Amazon ElastiCache reduced that operational burden. They provided managed capabilities around maintenance, backups, monitoring, encryption, Multi-AZ resilience, and service lifecycle management. In return, the platform accepted greater AWS coupling, additional cost, network dependencies, and less control over the underlying database and cache hosts.
The decision was not “Kubernetes versus AWS.” It was a decision about which layer should own stateful reliability.
I treated the transformation as a sequence of controlled migrations
I divided the work into phases instead of attempting a large replacement:
0. Architecture and strategy
1. Platform governance and standardisation
2. Platform evolution and managed-service migration
3. Delivery automation and rollback validation
4. Observability and Site Reliability Engineering
Each phase had explicit exit criteria. A phase was not complete because its YAML had been merged. It was complete only when the intended behaviour had been demonstrated and recorded.
That distinction changed the way I approached the whole project. It made rollback, evidence, and negative testing part of implementation rather than follow-up work.
Rebuild rather than rename
One early decision was whether to rename and modify the existing EKS environment or rebuild it from approved definitions.
Renaming appeared cheaper and faster. Rebuilding was operationally more expensive because it temporarily duplicated capacity and required another controlled migration. I chose to rebuild.
The target clusters became:
- `rideshare-prod-use1` in `us-east-1` (use1)
- `rideshare-prod-use2` in `us-east-2` (use2)
Rebuilding exposed hidden assumptions that an in-place rename would have preserved. It tested whether the cluster definitions, Helm values, IAM relationships, add-ons, node labels, taints, and operational procedures were genuinely reproducible.
Both clusters used dedicated platform and application node groups. Platform nodes were labelled and tainted with `workload-type=platform`, while application workloads ran on application nodes. Controllers such as cert-manager, External Secrets, ingress components, and later the monitoring stack received explicit selectors and tolerations.
This separation was intentionally modest. It did not pretend to provide complete tenant isolation, but it protected critical platform services from competing directly with application workloads and made scheduling intent visible in code.
Moving PostgreSQL to RDS( reducing toil without hiding the failure domain)
The target data architecture used a shared Amazon RDS for PostgreSQL deployment as the authoritative database for both EKS clusters.
The migration sequence mattered:
· Provision encrypted, private RDS infrastructure with Multi-AZ capability
· Configure database users and connectivity
· Migrate schema and data
· Compare record counts and run critical queries
· Store credentials in AWS Secrets Manager
· Synchronize Kubernetes Secrets through External Secrets Operator
· Update application configuration to use the RDS endpoint
· Validate both clusters
· Retain the original CloudNativePG environment until rollback confidence was established.
The most tempting mistake would have been to remove the old database as soon as the first application pod connected to RDS. I resisted that. Connectivity was only one test. I also needed confidence in schema completeness, data consistency, application behaviour, and the ability to reverse the cutover.
The transition was not frictionless. Application startup failures exposed incorrect endpoints in configuration. At another point, an AWS CLI command attempted to reach `https://rds..amazonaws.com` because the region was missing. That tiny error was instructive; in a multi-region system, region is not background context. It is part of command correctness.
External Secrets introduced another chain of dependencies to worry about. Secrets Manager, IAM policy, IAM role, OIDC provider, Kubernetes service account, SecretStore, ExternalSecret, and finally the generated Kubernetes Secret. When synchronization failed, checking only the final Secret was not enough. I learned to walk the chain in order and identify the exact trust or permission boundary that had failed.
The trade-offs behind the move to RDS
Moving PostgreSQL out of EKS reduced database operational ownership and improved the managed backup and Multi-AZ story. It did not automatically create a multi-region database.
Both application regions still depended on a shared authoritative service. The remote EKS region could experience additional database latency and inter-region transfer cost. RDS could become a shared dependency whose failure affected both otherwise healthy clusters. A serious production design would therefore need explicit RPO and RTO targets, tested restoration, and a decision about cross-region database replication or promotion.
The migration improved the system, but it did not abolish trade-offs. It moved them to a different boundary.
Moving Redis to ElastiCache(less operator ownership, more network concerns)
Redis followed a similar path. The in-cluster Redis Operator was replaced by a shared Amazon ElastiCache deployment.
The migration required more than changing `REDIS_URL`. I needed to validate authentication, TLS expectations, security-group paths, endpoint selection, application timeouts, and behaviour from both regions. Redis-dependent workflows including matching and WebSocket-driven interactions had to continue working before operator-managed Redis could be retired.
I encountered intermittent connection problems caused by endpoint and network-rule mismatches. The useful lesson was not simply “fix the security group.” It was learning to separate layers:
1. Does DNS resolve the expected endpoint?
2. Is the destination port reachable?
3. Does TLS negotiation succeed if enabled?
4. Are the credentials accepted?
5. Does the application use the same URL format tested manually?
6. Does the end-to-end workflow behave correctly under real traffic?
ElastiCache reduced the burden of running Redis pods and their operator. In exchange, the platform accepted another shared dependency, cross-region latency, data-transfer cost, and greater sensitivity to VPC routing and security configuration.
Again, the decision was about operational ownership not about pretending the managed service had no failure modes.
Multi-region traffic required application-aware health, not just a healthy load balancer
The two EKS regions were placed behind Route 53 latency-based routing for `rideshare.lukmonadeokun.com`.
Latency routing supported active-active regional entry points, but it introduced an important trap which is a Network Load Balancer and NGINX Ingress can both be healthy while the application behind them is unavailable.
I saw this in practice during failure testing. Removing application capacity in one region could still produce a `503` because traffic continued to reach infrastructure that was technically alive. The health signal was measuring the wrong layer.
I introduced application-level regional health checks rather than relying solely on transport-level reachability. Route 53 needed a signal that represented whether the RideShare application could serve users, not whether an NLB could accept a TCP connection.
Even then, DNS failover was not instantaneous. Resolver caching and TTL behaviour affected convergence, and existing WebSocket sessions could not magically migrate between regions. The final design therefore documented the limitations instead of presenting DNS routing as seamless session mobility.
That distinction matters to me: resilience claims should describe what the system actually guarantees.
CI/CD became an identity and rollback problem, not just a deployment problem
The delivery pipeline used GitHub Actions to deploy the Helm release independently to both regions. AWS access was provided through GitHub OIDC rather than long-lived AWS keys.
Get Lukmon Adeokun’s stories in your inbox
Join Medium for free to get updates from this writer.
OIDC improved credential hygiene, but it also made the trust relationship precise. The GitHub repository, branch or environment claims had to match the IAM role’s trust policy. The `use1` deployment was additionally protected by a GitHub Environment approval gate.
One of the more frustrating failures happened after the workflow had successfully assumed its AWS role and updated kubeconfig. It reached “Verify target cluster,” step then kubectl returned:
“You must be logged in to the server”
That message could easily lead to random RBAC changes. Instead, I separated the path:
· Did GitHub issue the expected OIDC identity?
· Did AWS STS allow the role assumption?
· Did the workflow target the correct account and region?
· Did `aws eks update-kubeconfig` reference the correct cluster?
· Was the assumed IAM role authorised by EKS?
The eventual deployment succeeded, including the protected approval flow and post-deployment checks. The important part was not the final green tick; it was avoiding a broad permissions change simply because the first error mentioned login.
A successful deployment was not enough.I deliberately broke one
To close the delivery phase, I added Helm’s atomic failure protection:
Then I supplied a deliberately invalid frontend image tag in `use2`.
The upgrade timed out as expected. Helm recorded revision 10 as failed and automatically created revision 11 as a rollback to the previously healthy revision 9. I then verified the restored frontend image, running pods, healthy deployments, HPA state, regional health endpoint, and production health endpoint. Both health checks returned `OK`, and no pod referenced the invalid image.
This test changed the meaning of “rollback supported.” It was no longer a command in a runbook. It was observed behaviour with evidence.
Protected `main` also influenced my workflow. Changes were made on focused branches, validated, pushed, reviewed through pull requests, and merged only after checks passed. That sometimes felt slower than editing `main` directly, but it produced an auditable path from intent to production state.
Storage taught me to prefer additive changes
Observability required persistent storage for Prometheus, Grafana, and Alertmanager. The clusters already had a legacy `gp2` StorageClass using the in-tree EBS provisioner. I was concerned that replacing it with `gp3` might break existing workloads.
So I did not replace it.
I created a separate `gp3` StorageClass backed by `ebs.csi.aws.com`, enabled encryption and expansion, and used `WaitForFirstConsumer`. The existing `gp2` class remained untouched.
I smoke-tested `gp3` independently in both clusters with a temporary PVC and pod. `WaitForFirstConsumer` ensured the EBS volume was provisioned in the Availability Zone of the selected node. I wrote data to the mounted volume, read it back, inspected the resulting PV and EBS volume, then deleted the temporary namespace and confirmed that the test volume was removed.
There were small mistakes along the way such as an AWS volume query used the wrong region, a malformed JMESPath expression failed, and I mistyped a topology label. None of those mistakes required risky remediation. I corrected the observation commands and continued.
That experience reinforced a pattern I now use deliberately:
When replacing shared infrastructure, prefer an additive path that can be tested independently before changing existing consumers.
Observability: From “pods are running” to measurable platform behaviour
I deployed `kube-prometheus-stack` version `88.6.3` independently in both production clusters. The stack included Prometheus, Alertmanager, Grafana, Prometheus Operator, kube-state-metrics, Node Exporter, and Kubernetes recording and alerting rules.
Prometheus, Grafana, Alertmanager, the operator, and kube-state-metrics were scheduled onto the tainted platform nodes. Node Exporter ran across all four nodes in each cluster. Persistent data used the new encrypted `gp3` StorageClass:
Prometheus retention was bounded by both time and size (seven days and 16 GB). Grafana and Prometheus remained private `ClusterIP` services and were accessed through port forwarding during validation.
In each region, Prometheus reported 27 healthy targets and zero unhealthy targets.
use1 cluster, confirming that the regional monitoring stack was collecting metrics successfully.Four Kubernetes nodes and four Node Exporter targets were visible. RideShare pods and available replicas were observable through kube-state-metrics. Grafana provisioned 25 dashboards, and the only firing alert was `Watchdog` the intentionally active signal used to prove that an alerting pipeline is alive.
Watchdog alert, which confirms that alert evaluation is active and provides a continuous signal for validating the alerting pipeline.Not every problem was in Kubernetes. In `use1`, the password stored in the Grafana Secret returned HTTP 401 because Grafana’s internal database password no longer matched the bootstrap Secret. I verified the failure through the Grafana API instead of repeatedly guessing in the browser, then realigned the internal password with the Secret.
At the time of writing, the core observability foundation was operational in both regions, with healthy Prometheus targets, provisioned Grafana dashboards, persistent monitoring storage, and independent regional monitoring stacks. Further platform-level instrumentation, including dedicated ServiceMonitors for ingress-nginx and cert-manager, remains part of the next iteration.
use1 EKS namespaces, confirming that application, platform and monitoring workloads were successfully observable.I chose to conclude this phase at a stable and validated checkpoint. The platform’s multi-region architecture, managed data services, delivery automation, rollback controls, and observability foundation had all been demonstrated with recorded evidence. The remaining work represents continued operational maturity rather than a gap in the architecture delivered by this project.
How my troubleshooting approach evolved
The project exposed a weakness common in complex infrastructure work which is an error near the end of a chain often originates much earlier.
A pod failing to start might be caused by an incorrect RDS endpoint. A Secret not appearing might originate in an OIDC trust condition. A `503` might mean the load balancer is healthy but the application is not. A Grafana login failure might mean the Secret is correct but its persistent database has a different password. A failed AWS CLI call might be an expired token rather than an EKS problem.
I became more disciplined about troubleshooting in layers:
1. Confirm context: Which AWS account, region, Kubernetes context, namespace, release, and revision am I touching?
2. Establish a baseline: What was healthy before the change?
3. Test one boundary at a time: DNS, TCP, TLS, authentication, authorisation, configuration, and application behaviour are different tests.
4. Prefer read-only evidence first: `describe`, `get`, `status`, `history`, `diff`, and API health queries reduce guesswork.
5. Make reversible changes: Retain the old data service, use additive storage, deploy to the lower-risk region first, and use atomic Helm upgrades.
6. Validate failure, not just success: A rollback mechanism that has never restored a service is still a hypothesis.
7. Stop at ambiguous authority boundaries: Invalid credentials and uncertain account state are reasons to pause.
I also learned not to confuse activity with progress. Re-running a failing command without changing the hypothesis is not troubleshooting. The better question is: *What observation would distinguish the two most likely causes?
What is complete and what is not complete
The platform now has:
· Two standardised production EKS clusters
· Managed PostgreSQL and Redis dependencies through Amazon RDS and ElastiCache
· Private secret delivery through Secrets Manager and External Secrets Operator
· NGINX Ingress, automated TLS, health probes, HPAs, and PDBs
· Route 53 latency-based multi-region routing with application-aware health validation
· GitHub Actions deployments using AWS OIDC
· A protected production approval path
· Post-deployment application validation
· Tested automatic Helm rollback
· Regional Prometheus, Grafana, Alertmanager, kube-state-metrics, and Node Exporter foundations with encrypted persistent storage.
The project is not finished. Application-level metrics still need to be instrumented. Alertmanager needs real notification routes. Centralised logging and tracing remain. Backups and restoration require deeper operational exercises. SLIs, SLOs, error budgets, runbooks, and broader regional failure scenarios still need to be completed.
I prefer stating that plainly. “Production-oriented” is not the same as claiming the system is production-perfect.
What this project ultimately demonstrated
The most valuable outcome was not the number of AWS services deployed or Kubernetes manifests written. It was the way the platform was evolved.
I made decisions based on operational ownership rather than tool popularity. I preserved rollback paths while migrating state and adopted managed services without overlooking the shared dependencies they introduced. I treated IAM and OIDC as trust systems rather than copy-and-paste configuration. I tested DNS failover at the application layer, deliberately triggered a failed deployment to observe recovery, and kept the existing storage configuration untouched while validating its replacement. I also measured the monitoring stack instead of assuming that running pods meant working observability.
Ultimately, this project demonstrated more than the ability to build a platform. It demonstrated the engineering discipline required to evolve one while maintaining visibility, recoverability, and control of risk.
The architecture changed substantially from one cluster carrying application and stateful operational responsibility to a standardised, multi-region platform built around managed AWS services and regional observability.
My thinking has changed too. I started by asking, “How do I deploy this?” I now ask a harder set of questions:
· What failure domain am I creating?
· Who owns recovery at this layer?
· What evidence proves the change worked?
· How do I reverse it safely?
· What claim am I making about reliability, and have I actually tested it?
Those questions are the real foundation of Platform Engineering and SRE. The tools are how I expressed them.
Technology used
Amazon EKS, Amazon RDS for PostgreSQL, Amazon ElastiCache for Redis, Amazon ECR, Route 53, AWS Secrets Manager, IAM, OIDC, EBS CSI, Kubernetes, Helm, GitHub Actions, NGINX Ingress, cert-manager, External Secrets Operator, Prometheus, Alertmanager, Grafana, kube-state-metrics, and Node Exporter.
Repository
The complete implementation including the regional EKS configurations, Kubernetes manifests, Helm charts, GitHub Actions workflows, observability configuration, Architecture Decision Records, validation evidence, and operational documentation is available in the RideShare Platform GitHub repository
