Why We Ship the Most Code on Friday

Isn’t Shipping on Friday Bad? 🐍

The theory goes that if you ship code on Friday you’ll inevitably be working Saturday to fix what you broke. This makes logical sense; software is messy and mistakes are unavoidable.

But no, shipping code on Friday isn’t bad. At least, it doesn’t have to be.

We’re able to ship the most code on Friday and maintain the lowest incident rate on Saturday. Here’s a tally of all commits to master (which auto-deploy to production) showing that Friday is our most active day.

Number of commits for each day of the week. Master only contains squashed pull-requests commits

And here’s a tally of our 102 internal incident reports, which shows that Saturday has the fewest incidents.

Total incidence counts across all time, grouped by day

So why do we ship so much on Friday and how is it not bad? After all, we could simply avoid the risk altogether by waiting until Monday.

Our Friday Changelog [1] is something both the team and community look forward to each week. We’re fully remote and spread across the globe, so it gives us a reason to come together and share what we’ve been working on.

It’s also just a nice way to cap off the week. We finish what we’re doing on Friday, enjoy the weekend, then come back energized to start something new.

Our Changelog is shared on Discord, Twitter, email, and our website

Reality is messy, though. Software never ships when expected so Friday often begins with our team searching for one or two more things to talk about, like when everyone is heads-down on larger projects.

This subtle pressure acts as great motivation to either (1) finish something nearing completion, or (2) spend a few hours implementing a fun idea from scratch. Both do wonders for team productivity and morale, but the true benefit is being forced to build a deployment pipeline that supports confidently shipping any hour of any day.

Avoiding Friday releases can potentially reduce weekend work. But following that reasoning, why not also avoid shipping after 4PM? Why not require an extra person for every code review? Why not hire a QA team and reduce releases to one per month?

You see where I’m going with this…

The classic “This is fine” meme by artist K.C. Green showing what happens when you add too many preconditions for shipping

A more productive strategy is to reduce mistakes through prevention, not avoidance, and there’s no greater motivator than getting paged at 3AM because your past self forgot to migrate prod. 🚨

Learning from mistakes has been part of our culture since the beginning. We publish every single incident to our Status Page and have produced over 100 internal documents as a result, each outlining the what and why of the incident while also including action items to prevent repeat occurrences.

Notion table showing our 102 incident documents

This investment compounds exponentially. As a simple example, if a bug caused a memory leak to take down production, we might:

• Add automated code analysis to detect memory leaks
• Add monitoring and alerting for memory usage
• Run multiple instances of our web server

These improvements prevent the original problem from reoccurring but also help with unrelated issues like a traffic-induced memory spike. This is how the investment compounds. So how do we decide the improvements to make?

Ideally, a proposed improvement should be automated and foolproof. Something like “manually test changes after every deployment” isn’t ideal because it can’t be enforced and is prone to human error. It should either prevent the problem from occurring altogether or make the problem easier to fix or identify later.

A lot of the improvements we’ve made are fairly specific to us, like running GraphQL Inspector on pull requests to avoid breaking our public API.

Here are some of the more general examples of things we’ve done:

• Use type-safe languages and linters to catch basic blunders
• Generate database and API libraries instead of writing from scratch
• Automatically run tests and build scripts on every commit
• Automatically run database migrations on deployment
• Run multiple instances of critical services with proper restart policies
• Create monitors for everything imaginable and ensure all alerts have actionable runbooks

The great news is that most of these are quick to implement and thus have a huge return on investment.

So, is shipping code on Friday a bad idea? Well, we still wouldn’t run a scary database migration right before hopping on a long flight, but ordinary releases are no longer a big ordeal.

Having a Friday Changelog has forced us to learn from our mistakes and automate the necessary checks to minimize human error. That way we can ship on Friday and enjoy the weekend.

[1] We shifted our Changelog to Mondays in December 2022 as an experiment, but it actually decreased views so we reverted back to a Friday schedule this past week.