Incident Report: October 21st Security Attack

On October 21st, Railway suffered a security incident from a large-scale malicious attack. Following up on the preliminary report, we outline the nature of the attack and the response to it in greater detail. We hope this helps our customers and the ecosystem better understand the attack.

What happened

At 13:58 UTC on October 20th, Railway engineers became aware of a threat actor who had unauthorized access within Railway’s systems. During the period of unauthorized access, the actor was able to chain various exploits, move laterally across multiple systems, and was ultimately able to obtain and decrypt a subset of customer variables, for at most, 1.85% (~100,000) of workspaces. After this, the actor was expelled from Railway’s systems with no further access.

Starting from the initial exploit at 07:28 UTC on October 20th through 09:24 UTC on October 21st when the attacker was fully evicted from our systems, Railway engineering worked around the clock in shifts. This was not a single breach followed by a single fix. The attacker probed many potential routes, in parallel, with Railway similarly hardening the routes in parallel. The escalating situation became a contest of agent speed, with Railway defending against abliterated models and a harness similar to ARTEX, according to Wiz.

Based on our forensic investigation, we believe the attacker was specifically targeting AI provider credentials, that they could sell to distributed transfer stations for use in distillation.

To protect our customers, we halted builds twice, at 17:03 UTC on October 20th and at 07:22 UTC on October 21st, rotated internal credentials, invalidated every user session and token at 20:19 UTC on October 21st, and restored builds for all plans by 22:49 UTC on October 22nd after significantly hardening the infrastructure. We published a preliminary report while recovery was still underway.

Timeline

Below is not an exhaustive list of actions the threat actor took, or proactive and reactive remediation actions Railway took, but rather a condensed summary of what we believe is pertinent to understand how the attack occurred. Gaps in the timestamps do not indicate that the attacker, or Railway, was inactive during this time.

First entry point

At 06:50 UTC on October 20th, the attacker began probing our build infrastructure. Railway’s build infrastructure at the time operated using a multi-tenant architecture where multiple builds could occur, in full isolation from each other, on the same virtual machine.

At 07:28 UTC on October 20th the attacker exploited a previously unknown vulnerability in our build infrastructure to partially escape filesystem isolation, granting them the ability to read some files stored on the host virtual machine. This allowed them to obtain a credential our build servers use to authenticate to a subset of internal systems, including to push images after a build completes. After they acquired this credential, the attacker attempted but initially failed to use it in a variety of systems.

An alert on multiple anomalous API calls to a registry service that stores artifacts from completed builds paged the engineer on-call at 13:58 UTC on October 20th. The on-call engineer quickly determined that these calls were not being made as part of the normal operation of the platform, and continued to investigate the root cause. The engineer traced these calls back to a compromised build credential and, through that, the build where the attacker partially escaped isolation. An incident response team was established and undertook actions in parallel: merging an underlying fix for the partial isolation escape, halting builds until the fix was rolled out, and hardening infrastructure including isolating the attacker.

From the moment that the root cause was identified to be an isolation escape, the response team started 24/7 observation of the attacker through collected signals, using audit logs, traces, and other instrumentation available to us.

Railway uses Temporal to schedule and run asynchronous tasks. The build system in use at the time used a self-hosted Temporal cluster to drive parts of the build process. These Temporal tasks contained information related to the build artifact(s), such as a read-only GitHub app token scoped to a single repository to clone said repository, the environment variables and secrets passed to the build, and metadata about how the build should be executed.

The credential the attacker gained access to was intended to authorize only work needed to complete builds, such as registry pushes, and nothing else. However, our internal Temporal instance used for builds accepted it as a trusted client. The system’s architecture allowed the attacker to call APIs intended for a Temporal client, elsewhere. Through one of the APIs, the attacker read a limited subset of stored histories of other builds, including some of the aforementioned build variables.

Many of Railway’s services deploy out of an internal monorepo, including onto Railway itself. For example, the Railway support platform, station.railway.com, deployed out of the monorepo onto Railway itself. A recent build that the attacker was able to retrieve out of Temporal was specifically for this service. Because the build credential had access to fetch snapshots of the repository state of builds, which we use for caching, the attacker was able to retrieve a copy of the monorepo. This gave the attacker a privileged view into how Railway works, enabling them to run agent fanouts, find any exploit, and move laterally with significant speed, as seen later in the attack.

Up until this point, we have no indication that the attacker targeted anything but Railway itself. We have no indication that the attacker downloaded snapshots or repositories of Railway customers. From build logs, we were able to reconstruct the requests the attacker made from the build system to internal systems using the compromised credential.

That evening, at 17:28 UTC on October 20th, we revoked and replaced the build credential and restricted what services build servers can reach.

After validating and believing we had contained the escalation pathway, we re-enabled builds for customers. After this, the attacker attempted again to escalate out of build isolation, without success.

Secondary entry point

At this time, we did not know that the attacker had a copy of the codebase. They seemingly used this access to attempt re-entries from many new angles, based on indicators they found in code. In the moment, we became aware that they had codebase access only when we saw them probing at endpoints that had not been in use for multiple weeks, months, or even years and had no live references in the system anymore.

Railway's internal systems are built and deployed automatically using a CI/CD system. This system can get short-lived cloud credentials to production infrastructure, to deploy changes to said infrastructure. After scanning our codebase for additional escalation pathways, the attacker found their foothold: an erroneously defined OIDC binding configuration in one of our cloud environments. At 03:26 UTC on October 21st the attacker managed to gain access to one of these short-lived cloud credentials through this erroneous OIDC configuration.

Our cloud infrastructure contains internal credentials used during the provisioning and deployment of some of our internal services. Relevant here are two credentials: a longer-lived credential that when combined with access to an internal VPC network, enabled reading encrypted environment variables out of a production database, and an SSH private key that our CD system uses to push binaries to a specific set of machines. These machines were in part connected to the relevant internal VPC network.

The attacker used the short-lived credential from the OIDC exchange to retrieve these internal credentials from our cloud environment, and connect via SSH to a machine that runs orchestration tasks for our fleet. This machine had a file on disk that contained a longer-lived credential that allowed decryption of encrypted variables, which this machine needed to provision customer workloads onto other nodes in the fleet.

The compromised machine also had access to the internal VPC network that the database that stores envelope-encrypted variables is connected to. Environment variables use envelope encryption, where each variable is encrypted using a per-environment data encryption key, which itself is encrypted using a root key held in an upstream cloud provider’s key management service. Between 04:07 UTC and 05:21 UTC on October 21st, the attacker used a combination of these credentials to connect to this database and retrieve encrypted envelopes out of it. Database logs show that the attacker was specifically prioritizing AI provider keys in their queries by selecting rows where the variable name matched attacker-provided regular expressions. Logs also show that the attacker was specifically targeting customers with large numbers of variables in their environment.

Between 04:42 and 05:14 UTC, and 06:36 and 07:15 UTC (all October 21st), the attacker used the longer-lived credential found on the orchestration machine to authenticate to our key management service and decrypt some of the retrieved encrypted variables using it.

At 07:14 UTC we disabled all active cloud credentials, and revoked all static credentials. We also disabled access to the key decryption service entirely, and shut down various parts of the Railway platform. At 07:22 UTC, we disabled builds for all customers again.

Our investigation shows that the attacker had no access to any customer data beyond 09:25 UTC on October 21st. However, we only established this fact later that day after we performed full validation that the attacker no longer had any access to Railway systems.

Customer impact

Since releasing the preliminary report, the forensic investigation has established two facts the timeline could not show at the time:

• The attacker's first signups began on October 18th, two days before the 13:58 UTC page that opens the timeline. During this period, the attacker spun up a variety of dummy accounts; all attempts in this initial probing were unsuccessful.
• The attacker's decryption of per-environment keys began at 04:42 UTC on October 21st, not after 05:27 UTC, and ran until we disabled the credential at 07:14 UTC.

Railway's variable storage is designed so that one compromised key cannot expose the whole platform. Each environment separately encrypts its environment variables using a per-environment encryption key that is itself encrypted by a root key in our key management service. This root key is managed by an upstream cloud platform, and cannot leave the key management service.

From logging available to us, we can validate that at most 1.85% of workspaces, about 100,000, could have had any of their environment variables decrypted. The logs identify a subset of environments, not individual variables, so that figure is an upper bound. As part of our analysis, we emailed customers who were at high risk of having their credentials stolen based on the forensics available to us. Any further data the attacker might have been able to exfiltrate cannot be decrypted because we disabled the credential the attacker was using.

The attacker also read the stored histories of 353 builds between 13:50 and 15:30 UTC on October 20th, sealed variables included. During the incident, starting at 09:25 UTC on October 21st we asked GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Google and Stripe to revoke potentially compromised keys. During this time, we could not yet determine which were exposed, so we provided hashes of every key we held to isolate blast radius as quickly as possible. Some vendors opted to revoke further keys at their discretion.

The data the attacker was able to decrypt also included linked-account tokens (BYOK agent keys and GitHub OAuth tokens) for 1,852 users. We reset every one of these tokens, notified each user, and identified no logins to Railway accounts using credentials taken in this incident. Unauthorized spend on the stolen keys was estimated to reach ~$740,000. We are working with the providers to refund it, as well as backstopping it from our own balances for those unwilling to do so. Details for remediation are under Next Steps.

Attack intent

Upon performing forensic analysis, we determined the attacker had begun creating a swarm of accounts as early as October 18th. All earlier attacks were thwarted via our automated systems. Across a variety of accounts, throughout the entire pre, during, and post incident timeline, we were able to identify the attacker and what they were after.

Our internal forensic investigation suggests AI provider keys were the primary target of the attack. We determined this by parsing the attacker’s queries, which were specifically for AI credentials. We believe the credentials were passed on to distributed transfer stations for use for distillation. Starting at 09:25 UTC on October 21st, we notified the affected providers with lists of potentially exposed keys.

We notified providers first so they could revoke potentially exposed keys while we secured our infrastructure. At that point, we could not assure customers that replacement credentials stored in Railway would remain protected. We continued our engagement with labs and providers through October 23rd as they revoked further keys out of an abundance of caution.

The attacker queried variable names for AI provider keys and decrypted the keys for a subset of the environments that matched. An additional query also matched cloud, database and payment service credentials. That said, forensics did not indicate customer databases were accessed.

The unauthorized usage we have observed since ranges from coding agents to AI training pipelines. OpenCode, Warp, Fireworks AI, and several others disclosed similar thefts between October 6th-23rd. We assess this to be one of a series of attacks on platforms that hold AI keys.

In the days following the incident, customers shared indicators of compromise, source IP addresses, and examples of unauthorized requests made with their credentials. These reports helped us investigate how the compromised keys were being used outside Railway.

The examples included coding-agent requests, reviews of AI-agent behavior for use as training data, and checks of training tasks and their scoring systems. This range of workloads is consistent with compromised credentials being distributed through services that relay or resell access to AI providers. We have not established whether the original attacker operated those services or supplied credentials to other parties.

Wiz’s “Black October” investigation documents a similar broader pattern: stolen AI credentials being checked and used by resale operations. That research provides context for our findings; it does not independently attribute the Railway incident or establish that all subsequent use came from the same operator.

We thank the customers who shared logs and request examples. Their evidence helped us distinguish the initial intrusion from the subsequent misuse of compromised credentials.

Containment and recovery

System sanitization and credential rotation

Railway engineering rotated credentials used within tooling and infrastructure. As part of that process, we bootstrapped new PKIs and instantiated a new root CA for trust. Every relevant key/secret was rotated as part of this recovery. Cloud properties were additionally restricted, which slowed down recovery efforts as we manually provisioned all new machines.

Internal tooling systems were rebuilt from validated source, to ensure everything running could be trusted. Any machines, services, and accounts where we had suspicion of threat actor access, were reprovisioned from scratch, with the old machines shut down and snapshotted for forensic analysis.

Because the CI environment was one of the routes of access, this work was carried out with CI disabled. Each rotation was applied manually, and each dependent service was restarted manually.

During this period, customer workloads remained accessible.

Railway published remediation guides at 01:22 UTC on October 22, before builds were restored, so that steps not requiring a deploy could begin immediately. Links are available in the preliminary report. Given we had customers with critical workloads, we advised them to turn off their deployments if they felt that they were at high risk.

Customers may have had to perform re-authentication actions due to the response that we undertook, listed in the table below:

Builds

During the weeks leading up to the incident, we had built and tested new build infrastructure based on a new isolation mechanism relying on individual virtual machines, as well as the original multi-tenant isolation. With this design, attackers would have to escape two separate layers of industry standard isolation.

This included a new custom control plane that enabled more isolation between orchestration and compute, an additional layer of isolation around the builds themselves, and a new storage backend for caching. We had aimed to roll this build system out to an initial set of customers in the week of October 26th.

After the initial partial isolation escape, we disabled builds until we had rolled out a fix for this issue. Additionally, before re-enabling builds, we switched the attacker’s workspaces, and all newly signed up Railway accounts at large, to use the new build infrastructure. This was done in an attempt to prevent the attacker from breaking out of the build infrastructure again while we worked to move everyone to the new infrastructure.

Within an hour of the patch to the original issue, a team internally tasked with “hardening” the original builder derived an additional exploit by modifying the original vector. Due to this, and because we could not be certain to have contained 100% of the attacker's accounts, we made the decision to accelerate the rollout of the new infrastructure to all existing users.

The new generation of build infrastructure does not share compute machines with the previous generation, and uses an entirely separate storage and cache mechanism. Hence, doing an immediate switch instead of the planned gradual rollout required provisioning and scaling up a large number of machines very quickly.

We also saw an influx in builds as a whole, as customers were rolling out new credentials for their services following our broad disclosure. This compounded capacity issues to nearly two orders of magnitude above baseline.

During the night, and the following day, our engineering teams worked around the clock to acquire the machine resourcing required to spin up the new storage infrastructure and re-provision our fleet of build machines.

We began to restore builds on October 22nd at 03:32 UTC, starting with Enterprise workspaces. Pro workspaces followed at 06:29 UTC. Once we had enough build and storage capacity available for the new system, Hobby builds were restored at 21:08 UTC, while Free and Trial builds were restored at 22:49 UTC.

During the days that followed, the Railway team continued triaging issues with the build experience. This was caused by us increasing bandwidth and load on our upstream storage provider by at least 10x above baseline, due to the switch to the new storage architecture. As a result, some customers experienced slow or failed builds.

We, and our upstream vendors, have since significantly scaled capacity. This has resolved the capacity issues, both at the time and moving forward.

Existing deployments

Running services were not affected by the pause to builds. Existing user applications did not encounter downtime, as existing deployments continued to run. That said, we understand for many customers, the inability to trigger new deployments was a substantial source of frustration during this time, particularly because revoked credentials caused running services to stop operating normally.

If a user revoked a credential for the build, at the time, they had no way to redeploy the service with the new credential until builds were back. Unfortunately, we were unaware how long the revalidation to restore builds would take, as we made the decision to move to a more-secure new generation of builder infrastructure. We acknowledge that some customers encountered downtime because of this fact.

When possible, we suggested that customers attempt credential rotation manually via SSH.

Prioritizing existing users

Because the Railway team was actively restoring capacity to all customers, we wanted to protect the experience for existing customers as much as possible. During this period, to scale capacity for existing users, we did not accept new signups. We acknowledge the disruption to some organizations who desired to invite their team members during this period, and we deployed a workaround that allowed admins to request an invite to their team.

New signups to the platform were paused on October 22nd at 19:06 UTC and reopened on November 2nd. Support tooling was fully restored on October 26th. This decision also gave time for Railway engineering to implement further system hardening and attack detection improvements, some of which are outlined below.

Next steps

In the time since containment, we’ve worked with upstream AI providers and Cloud vendors in an effort to make customers whole while diminishing the possibility of future attacks.

Unauthorized token spend

We ask for all customers to report charges that they didn’t make. To do this, please fill out this form and we will do our best to help you get compensated.

So far we’ve been coordinating with all AI token providers with recorded impact to refund unauthorized token spend. Multiple customers have reported remediation. Each vendor has different policies. For the most common vendors, here are their policies at the time of writing.

• AWS Bedrock: AWS’s existing refund policy covers fraud and their account teams have initiated refunds for customers.
• Google Gemini: Google’s existing refund policy covers fraud and their account teams have initiated refunds for customers.
• OpenAI: The vendor advises affected customers to open a support request and pass the Case ID to Railway support. We have a shared bridge where they review refunds on a case-by-case basis.
• SpaceXSI: The vendor advises affected customers to contact Railway support with their claim. We have a shared bridge where they review refunds on a case-by-case basis.
• Anthropic: As of writing, we are in active discussions with their customer team to get next steps for our joint customers. In the interim, we will be covering Anthropic breached key costs ourselves.
• OpenRouter: While OpenRouter has decided to enforce their no-refund policy as per their Terms of Service, we will be covering all costs associated with this breach in relation to OpenRouter.

As noted, for the providers who will not be covering token-related losses, we will be covering all token losses in lieu of them. We want to let people know we take these things very seriously, and additionally thank the above providers who have worked with us. They did not need to do this, and for this, we appreciate our existing and continued partnership with them.

Railway security updates

Each of the attack vectors used during the incident has been remediated, with every weakness the attacker exploited now closed. In cases where we found previously unidentified vulnerabilities in third-party software we are working with the upstream to provide responsible disclosure. The following are selected examples of improvements to our security posture and hardening but are by no means an exhaustive list of Railway engineering’s actions.

At the time of writing, Railway has obtained access to Project Glasswing with approval from the U.S. Department of the Treasury. We had sought this approval prior but were awaiting approval at the time of the incident. We’ve also identified and live-patched against multiple threat vectors across the platform. During the course of the attack, evidence indicates that the malicious actor was using abliterated models with no safeguards, with the ability to perform escalated actions in rapid succession.

Build

Each build now gets a dedicated ephemeral virtual machine, on top of the existing original isolation, with a credential that only has permission to push a specific OCI image to the image registry managed by Railway. The shared systems exploited by the attacker are no longer in use. We have both significantly increased the layers of isolation, and significantly reduced the surface boundary of this system, compared to the previous generation systems.

CI Pipeline

We’ve done a significant amount of hardening to our internal CI infrastructure and the way we handle internal secrets. Our CI systems have been completely rebuilt in order to separate sensitive components with fine-grained access control. They now require 2FA approval and biometrics in critical areas, while isolating the network path to and from our CI environments as much as possible. Lastly, the CI pipeline has extensive logging through its lifecycle and alerting through our SIEM.

Internal access

We reviewed every service account to ensure that it was the correct level of privilege necessary only to the workload that the account should be privileged to. Wherever possible, we are using short-lived credentials that are narrowly scoped and issued just in time. We’ve audited our infrastructure to ensure that any deprecated systems are fully decommissioned in order to reduce the surface area for future attacks.

Secret storage and decryption hardening

We have revamped our secret storage and decryption systems in multiple ways to make any exfiltration attempts significantly harder going forward. All encrypt and decrypt operations are now centrally audit logged, and no service other than the secrets service can get access to keys that allow decryption of data encryption keys. Each calling service is restricted to decrypting only the set of credentials it needs to operate.

Roadmap and Prioritization

As we believe that cybersecurity is a continued and mounting threat, we are both expanding our organization as well as expanding our roadmap towards more security-focused endeavors (both internal, for us, and external, for our users).

Outstanding

During the response period we sent a formal complaint to the FBI’s IC3 to allow law enforcement to begin their investigation. In addition, Railway is engaging with a specialist forensics firm, whose investigation is ongoing. Railway has shared what it knows with authorities and providers whose customers were affected.

For any compliance reporting requirements with customers, Railway is sharing IOCs and data for customers who serve clients with increased reporting requirements in accordance with all relevant laws and regulations. For all customers who have reached out with remediation requests, we have responded with a form for listing their impact. If you haven’t already, you can reach out at https://station.railway.com.

Conclusion

This was a novel and sophisticated attack against our infrastructure, one of many against companies in the developer tooling space. For our customers and the greater ecosystem, we are committed to transparency about the circumstances detailed here, and are deeply sorry for the impact it has caused for our customers.

We will continue helping affected customers, working with providers on unauthorized charges, and sharing material findings as the investigation progresses. We have taken many learnings from this attack to heart.

Protecting our customers and rebuilding your trust remains our goal.