Incident Report: January 28-29, 2026

Railway experienced intermittent GitHub authentication failures over a multi-day period caused by hitting GitHub's OAuth token rate limit. This resulted in users being unable to log in with GitHub and encountering "GitHub repo not found" errors during peak US business hours.

When a Major Outage occurs, it is Railway's policy to share the public details of what happened.

This incident caused intermittent failures for users attempting to:

• Log in to Railway using GitHub authentication
• Connect new GitHub repositories to their projects
• Deploy from GitHub repositories

The impact was most severe during peak US business hours, following an hourly pattern where rate limits would be exhausted approximately 30 minutes into each hour.

January 26th, 2026:

• 18:57 UTC — Initial reports received of "GitHub Repo Not Found" errors
• 19:06 UTC — On-call engineers began investigating GitHub API hot paths
• 20:23 UTC — Root cause identified: installationTokenById dataloader was creating new tokens on every request batch without caching across requests
• 20:44 UTC — PR submitted to cache installation tokens
• 21:14 UTC — Fix merged, reducing API calls to ~10% of previous levels
• 21:41 UTC — Rate limiting continued despite optimizations

January 27th, 2026:

• 17:54 UTC — Degradation worsened; additional optimizations began
• 18:13 UTC — PR to reduce API concurrency merged
• 18:31 UTC — Commit status updates reduced by 50%
• 20:50 UTC — Investigation continued; GitHub Support contacted

January 28th, 2026:

• 17:13 UTC — Background environment refresh identified as additional contributor
• 17:40 UTC — Multiple engineers joined investigation
• 21:16 UTC — GitHub OAuth logins temporarily disabled to reduce pressure
• 23:26 UTC — GitHub Support confirmed root cause: OAuth token requests exceeding 2,000/hour limit
• 23:32 UTC — GitHub logins re-enabled after deploying additional mitigations

January 29th, 2026:

• 14:01 UTC — Minor rate limiting resumed during peak hours
• 15:06 UTC — New incident declared; GitHub OAuth logins disabled again
• 15:28 UTC — GitHub App permissions updated to receive additional webhook events
• Ongoing — Engineering team continued deploying fixes and working with GitHub Support

For further reference, please see the incident's live updates on the Railway status page.

Railway is experiencing record user sign-up growth, and unfortunately our scale events did pass through to our customers via the form of a rate limit exhaustion on GitHub’s OAUTH token flow, and triggering unknown secondary rate limits as a result of a negative feedback loop related to an implementation of our login and repo deployment flow.

Railway relies heavily on GitHub's API for authentication, repository access verification, and deployment triggers. Over time, our growth led to increased API usage that eventually exceeded GitHub's OAuth token rate limit of 2,000 tokens per hour.

The dataloader responsible for creating GitHub installation access tokens was not caching tokens across requests, resulting in ~82 new tokens per second during peak usage. Recently, we scaled the number of running replicas of our backend worsening the issue. Doubly so when we would utilize a fresh user token when we would check for repository permissions.

We had this pattern because we take repo and code access deployment security seriously; example, if a company removed a former colleague from a repo, we would always want to check if they have up-to-date access to the repo to ensure that they have the proper permissions. However, during triage, we found that multiple code paths were making GitHub API calls that could be consolidated or cached, including commit status updates and repository permission checks. This was then compounded by scheduled workflows for caching user GitHub repositories were running at high frequency, compounding the token usage.

When the rate limit was hit, GitHub returned errors that prevented users from logging in or accessing their repositories. The hourly reset pattern created a cycle where service would briefly recover at the top of each hour before degrading again. It took long to debug the degradation due to additional secondary rate limits being hit after the initial error, returning a generic error. After collaboration with the GitHub team, we were able to correlate which call paths were exhausting the rate limits and then worked to remove the use of the user token with an app installation token. Once the engineering team updated the permissions of the railway-app GitHub application to receive the necessary events that would ensure the same level of functionality that our customers expected, we rolled out the changes to our customer base after extensive testing.

The issue was deemed solved on January 29th, after we have confirmed that no rate limits were triggered.

We have taken the following actions to prevent recurrence:

• Cached installation tokens - GitHub App installation tokens are now cached and reused for their full validity period, reducing token creation by over 90%
• Reduced commit status updates - Consolidated deployment status updates
• Switched to installation tokens - Migrated multiple code paths from user OAuth tokens to installation tokens where appropriate
• Optimized repository access checks - Implemented cached permission checks using our internal database rather than real-time GitHub API calls

At the product level, we have also moved the automatic repo sync to a manual one where customers can confirm manually if they would like to sync their GitHub account repos to Railway. We also implemented a number of improvements to general monitoring of login, repo, and rate limiting alerting to make sure that future occurrences are triaged significantly quicker.

Railway is committed to providing the best-in-class cloud experience. While GitHub's rate limits are outside of our direct control, the root cause was our own API usage patterns that had grown unsustainable. We apologize for any inconvenience caused and are working to ensure our GitHub integration remains reliable as we continue to scale.