Upgrading to Ruby 3 in small steps

I’m on the Ruby infrastructure team and last year we upgraded our app from paperclip to ActiveStorage, then from Rails 6.1 to 7.0, and finally from Ruby 2.7 to 3.1. So many upgrades!

This is the process we used to upgrade to Ruby 3.1.

One important thing to note is that we absolutely do not want a giant PR. At Fullscript we have a culture around small PRs and there’s strong reasoning why. It’s also completely practical. Keeping a large, long-running, PR up-to-date as 100+ engineers are constantly merging and deploying code is next to impossible. You would spend more time rebasing than you would get any work done. Instead, we want our branches to be short lived by making changes that are compatible with both versions of Ruby (2.7 and 3.1) and merge these changes into production as we go. Here’s our process for doing this in small incremental steps.

First step — Run the tests

The first step is to get a sense of the damage by running all of the tests with Ruby 3.1. There’s too many tests to run locally so instead we have a fleet of CI pipelines that run our tests, build images, and validate our code. We configured one of these pipelines to run in Ruby 3.1 and we can trigger it whenever we open up a PR.

click ruby-next to run all the things with Ruby 3.1

As expected, the tests don’t even run at first…

Most upgrades are like this. You start off by creating an error, doing some investigative legwork to figure out what the problem is, and then fixing it so you can trigger the next error. The first step is really about fixing enough stuff so that the app boots and the tests run.

This will be different for everyone doing this. But for us we largely got a bunch of syntax errors because Ruby 3+ started to bundle many of its previously-included libraries into separate gems. To solve these errors it was mostly just adding the (now) missing libraries to our Gemfile and verifying everything still worked in Ruby 2.7.

Second step — Fix all the deprecation (kwarg) errors

At this point the tests boot up and run (🙌) but… they are all red. For us, the major contributor of this was the breaking change in Ruby 3.0 where positional and keyword arguments were separated. We decided to tackle this next.

By default deprecation warnings are turned off in Ruby. So we enabled them in our development environments to start.

# Enable Ruby deprecation warnings somewhere early in the stack (config/application.rb)
Warning[:deprecated] = true

There were so many kwarg warnings at first that it was overwhelming.

warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

Where do you start? Plus, how do you prevent developers from making the problem worse as you’re fixing existing errors? The answer we came up with was a test ratchet. Will Larson describes this idea in his blog post on migrations as stopping the bleeding.

Start by stopping the bleeding, which is ensuring that all newly written code uses the new approach. That can be installing a ratchet in your linters, or updating your documentation and self-service tooling. This is always the first step, because it turns time into your friend. Instead of falling behind by default, you’re now making progress by default.

Ratchet the warnings

We use a tool called deprecation toolkit to manage warning messages. The basic idea is if a new warning is noticed while a test runs, the test fails. The library has ways to record those warnings so you can ignore them for now and get back to them later (similar to a .robocop_todo.yml file). In this case we just used it as a ratchet. This is how it works:

1. Turn the ratchet on for one small piece

We went folder by folder in the /spec/ directory. For example, take "./spec/controllers"and turn on ruby kwarg warnings for it. This would break any controller specs that weren’t compatible with Ruby 3. Tests fail and print out warning messages for the location of the problem.

2. Fix everything in that section

We then took this list of failed tests and updated any code so that the deprecation warnings disappeared and the tests passed.

We opened a PR and merged that in. Once we’ve fixed all of the controller specs, from this point on, it’s no longer possible for anyone to write a controller spec or make a change to a controller that wasn’t Ruby 3 compatible. If you did, a test would fail. This ensures that everyone is writing Ruby 3 compatible code as we’re fixing incompatible code.

3. Tighten the ratchet

From here you just keep tightening the ratchet. Add another folder (or subfolder), fix it, and keep tightening. It didn’t take too long before we fixed all of the kwarg warnings and made it impossible for anyone to add new ones.

This is what our ratchet looked like:

# In our .spec/rails_helper.rb file

# --START RUBY 3 RATCHET--

FAIL_IF_NOT_RUBY_3_COMPATIBLE = [
"./spec/controllers",
]

config.before(:each) do |example|
  location = example.metadata[:location].split("/")
  root = location.first(3).join("/") # "./spec/folder_name"
  subfolder = location.first(4).join("/") # "./spec/folder_name/subfolder"
  
  if (FAIL_IF_NOT_RUBY_3_COMPATIBLE & [root, subfolder]).any?
    # Enable ruby deprecations for keywords (it's suppressed by default in Ruby)
    Warning[:deprecated] = true
    
    # Taken from https://github.com/jeremyevans/ruby-warning/blob/master/lib/warning.rb#L18
    kwargs_warnings = [
      %r{warning: (?:Using the last argument (?:for `.+' )?as keyword parameters is deprecated; maybe \*\* should be added to the call|Passing the keyword argument (?:for `.+' )?as the last hash parameter is deprecated|Splitting the last argument (?:for `.+' )?into positional and keyword parameters is deprecated|The called method (?:`.+' )?is defined here)\n\z}
    ]

    DeprecationToolkit::Configuration.warnings_treated_as_deprecation = kwargs_warnings
  end
end

config.after do
  Warning[:deprecated] = false
  DeprecationToolkit::Configuration.warnings_treated_as_deprecation = []
end
# --END RUBY 3 RATCHET--

4. Production kwarg monitoring

By now we’re in pretty good shape. The tests were (mostly) green and needed just a little bit of love to get them all green. We’ve fixed and prevented all the kwarg issues that are tested. But what about untested code? Our codebase has been around for a while and, while we have a great testing culture, I’m not foolish enough to claim that there are no Balrogs in the Mines of Moria.

We wanted to enable kwarg warnings in production and see if we missed some by pushing any errors up to our error monitoring stack (we use Sentry).

This is very simple to do. Ruby has a Warning module that you can hook into. When Warning calls .warn(message)we wanted to send that message up to Sentry.

# frozen_string_literal: true

# This module's purpose is to push any ruby warnings up to our ErrorMonitoring stack
# so that we can have visibility on all Ruby and gem warnings happening in production.
# This needs to be loaded early in the boot process so that we can catch any warnings
# from any bundled gems.
class RubyWarning < StandardError; end

module ErrorMonitoring
  module WarningHandlers
    module Ruby
      def warn(message)
        exception = ::RubyWarning.new(message)
        ErrorMonitoring::Notify.new(
          exception,
          level: :warn,
          message: message
        ).call
        super
      end
    end
  end
end

# Enable Ruby deprecation warnings
Warning[:deprecated] = true

unless Rails.env.test?
  Warning.singleton_class.prepend(ErrorMonitoring::WarningHandlers::Ruby)
end

We added this early in the Rails boot process by requiring this file in a before_configuration block in our config/application.rb file.

config.before_configuration do
  require_relative("../lib/error_monitoring/warning_handlers/ruby")
end

This worked great!

Soon after we had deployed this code we started to see thousands of warnings appear in Sentry. A small handful of them were kwarg warnings that we could take action on. But unrelated (and noisy) warnings just keep flooding in. Hundreds of thousands turned into millions overnight and little did we know that Sentry charges by the error. The next morning we got a message from our ops team saying that we had blown through our entire month’s Sentry budget in less than 24 hours. Our experiment worked a little too well. 😬

We quickly turned this off.

As an alternative we pushed these warnings up to Redis as a sorted set along with a super simple admin UI. Nothing fancy. Just enough to capture any production warnings and see how frequent they were.

Step 3. QA and release

At this point all the tests are green and we aren’t seeing any kwarg warnings in production. We’re good to hand this off to the rest of the teams.

This is our final step. A bit of manual QA by domain experts to make sure that we haven’t missed anything. As teams are looking, we prepare a final merge request to cut over the docker containers to Ruby 3.1. And that’s it! Once we’ve done our due diligence then we’re ready to release the upgrade to production.

We coordinate with ops to make sure we can roll the deploy back if something goes wrong. But I’m happy to report that this went super smooth. We had zero errors and Fullscript is now happily running in Ruby 3.1.