Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

Sarath Krishna Prasad

Sarath Krishna Prasad

Rotate Every Secret, Railway Said. Then It Paused Deploys.

The Railway env var breach exposed a dependency most rotation runbooks never write down: the deploy pipeline of the platform that just got breached

Sarath Krishna Prasad
7 min read3 days ago

--

Press enter or click to view image in full size
On October 21, Railway told every customer to roll every environment variable they had stored on the platform. API keys, database passwords, private registry credentials, all of it. An unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents.

Reasonable advice. Standard advice. The same notice also said this:

All deployments are paused while we investigate. Until builds are restored, you will not be able to roll out new versions.

So the instruction was “rotate now”, and the mechanism you would normally use to get a rotated secret into a running service was switched off. Someone on X summed it up better than any postmortem will: “Railway: Rotate your env variables they’ve been compromised. Me: Let me do that real quick! Also Railway: All deployments have been paused btw.”

I don’t think Railway did anything unusual here. Pausing builds while you investigate a breach of the system that feeds secrets into builds is defensible. What interests me is that plenty of setups, on Railway or anywhere else, have the exact same dependency, and almost nobody tests what happens when it breaks.

Why a new value in the dashboard did nothing

Railway’s own guide spells out the mechanics in one sentence: “containers read variables only at startup.”

That is how almost every PaaS and most Kubernetes setups work. The secret gets injected as an environment variable when the process starts. Change the value in the dashboard and the running container keeps using the old one until it is redeployed or restarted. Rotation and deployment are the same operation wearing different hats.

Now look at the timeline from Railway’s status page. Builds were paused across all regions by the morning of October 21 (UTC). They came back in stages:

  • Enterprise: about 04:00 UTC on October 22
  • Pro: 07:11 UTC on October 22
  • Hobby: 21:59 UTC on October 22
  • Free and Trial: 23:22 UTC on October 22

For a Hobby workspace, that is roughly a day and a half where you knew your keys might be in someone else’s hands and could not ship a new one through the normal path.

And it got slightly worse in a way I didn’t expect. Railway reset the GitHub tokens it held, which is the right move, but the status page later noted that redeploying code from before the incident would fail until you reconnected the GitHub integration. So for a while the rollback path also depended on the platform you were trying to get away from.

The ugly choice nobody plans for

If you had an OpenAI key, a Stripe key, or an AWS access key sitting in a Railway variable that night, you had two options.

Revoke immediately. The attacker loses access. So does your app. You have chosen your own outage, and it lasts until builds come back.

Wait for builds. Your app stays up. The exposed key stays valid. If the goal of these attacks really is inference providers, as Railway believes, a valid LLM key is the thing being monetised by the hour. OpenCode users found that out a few days earlier, with some reporting hundreds of dollars and up to $1,700 in unauthorized charges overnight after their stored provider keys were exposed.

Most rotation runbooks describe only the happy path: generate new key, update the secret, redeploy, revoke old key. Step three is the assumption.

There is a middle ground for some providers, and it is worth knowing which ones support it before you need it.

AWS lets an IAM user hold two access keys, and you can deactivate a key instead of deleting it. Railway’s AWS guide recommends exactly that, so you can briefly reactivate if something you forgot about breaks:

aws iam list-access-keys --user-name railway-app
aws iam update-access-key --user-name railway-app \
  --access-key-id AKIAEXAMPLEOLDKEY --status Inactive

That is a reversible revoke. You can flip it off, watch what breaks, and decide.

Stripe is the opposite lesson. Its “Rotate key” flow can keep the old and new keys both working for up to 7 days so you can migrate without downtime. Great for routine rotation. In a compromise, that is up to seven days of the attacker’s key still working. You can pick “Now” as the expiration, which deletes the old key immediately. The point is that zero-downtime rotation was designed for a different night than the one you are having.

So the first thing I’d want to check in any setup is simple: for each secret, is there a way to cut off the old value that doesn’t depend on my deploy pipeline being healthy?

“Just use a vault” is not the whole answer

The obvious response is to stop storing secrets as platform environment variables and fetch them at runtime from something like AWS Secrets Manager or Vault. Rotate the value in the vault, the app picks it up on its next refresh, no redeploy needed.

Get Sarath Krishna Prasad’s stories in your inbox

Join Medium for free to get updates from this writer.

That helps with the specific failure above. But I’m skeptical of it as a cure, because the app still needs credentials to talk to the vault. On a PaaS, where do those live? In a platform environment variable. You’ve reduced a hundred secrets to one bootstrap secret, and if that one leaks, the attacker can read everything it is allowed to read.

That is still a genuine improvement, just a different one than people sell:

  • One credential to kill instead of a hundred, and killing it is a single action at a provider you control.
  • The vault has its own access log, so you can see which secrets were actually read and from where, instead of assuming the worst about all of them.
  • Downstream keys can rotate without touching the deploy pipeline at all.

What it does not do is make a platform breach a non-event. And for a small team on Railway, adding a separate secrets service and a refresh mechanism in every app is real cost and real complexity. I’d be honest about whether that is worth it versus just having a tested, boring manual rotation plan.

The better long-term answer, where it exists, is to have no static secret at all. Short-lived credentials via workload identity or OIDC federation mean there is nothing useful to steal from an env var store. That is comparatively easy on AWS, GCP and Azure with their own compute. It is much harder from a third-party PaaS, and plenty of API providers still hand you a long-lived key and call it a day.

What I’d actually test this week

None of this needs a big platform project. It needs an afternoon and a willingness to break staging.

1. Inventory where each secret lives. Not just the platform. CI variables, local .env files, a teammate's laptop, the cron job on some old VM. Railway's own guide reminds you to update "anything outside Railway that used the old credentials." On Railway you can at least list what a service has:

railway variable list --service api --environment production --json

That prints values, so run it somewhere you trust, and keep only the names in your inventory.

2. Prefer references over pasted values. Railway’s bucket guide makes this point: if a service has a secret pasted as a plain value instead of a reference variable, the next rotation needs a manual edit in every place it was pasted. Shared or referenced variables mean you change it once.

3. Time a rotation with deploys disabled. Pick one secret in staging and rotate it while pretending the build system is down. Can you restart the existing container to pick up a new value? Can you reach the provider’s console to cut off the old key? How long does it take? If the answer is “we can’t”, that is the most useful thing you will learn all quarter.

4. Shrink what a stolen key can do. Stripe restricted keys and access policies that limit a key to specific IP ranges or ASNs. Per-project or per-key spend limits on LLM providers where they exist. A leaked key that only works from your egress IPs, or caps out at a small budget, turns a crisis into an annoyance.

5. Know your provider’s revoke semantics. Is it delete or deactivate? Is there a grace period, and what is the default? Railway itself noted that the GitHub registry tokens it removed from its side “still work on GitHub until you revoke them.” Removing a secret from one system is not revoking it.

The uncomfortable part

Railway has since published a preliminary report, with a final one still to come. It says the attacker, apparently hunting AI provider keys for distillation, escalated privileges in Railway’s build infrastructure, then reached its CI environment and used compromised credentials to decrypt a subset of stored environment variables. Railway puts the potentially exposed share at up to 1.85% of workspaces, about 100,000, and still tells everyone who signed up before October 21 to rotate.

The lesson doesn’t depend on those details, and if anything the report sharpens it: the build infrastructure was part of the attack path, which is exactly why it had to be switched off. Most of us built secret rotation as a deploy. That works right up until the day the deploy system is the thing under investigation. And you only find out on the night when every provider dashboard is busy at the same time.

Encrypted at rest is a checkbox. Being able to cut off a secret in five minutes when your platform is down is a capability. Only one of those helped anyone on October 21.

Sources

Cybersecurity
DevOps
Secrets Management
Cloud Computing
Railway

--

--

Written by Sarath Krishna Prasad

5 followers
22 following

Help

Status

About

Careers

Press

Blog

Store

Privacy

Rules

Terms

Text to speech