In a small team or a small codebase, upgrading from Ruby 2 to 3 is often a short and decisive process. However, at Wrapbook we have a large codebase split across six teams with over 50 people contributing daily. To put some meat to that, running rails stats on our majestic monolith returns over 180k lines of code.

When we broke ground on the Ruby 3 project, we needed a plan for not just the migration, but also for keeping everyone moving fast and building delightful features during an exciting growth phase of the company. In order for that to happen, our code needed to be Ruby 3 compliant before we switched over. Not only that, but we had to ensure no new code was introduced that broke compatibility. This is the story of that journey.

The Plan

The first task at hand was to put a plan to paper. This is how the project was tackled from a planning and technology perspective. Below is a representation of the first draft of the plan:

  • Dual boot so we can run Ruby 2.7 and 3 side-by-side
  • Upgrade and patch any gems required
  • Run tests against Ruby 3 and fix all the breaking tests
  • Rewrite APIs to use keyword arguments or introduce the compatibility fix of def some_method(**kword_args, *args)
  • Enable QA environments to run on Ruby 3
  • QA and bugfix until we are confident we are good to go
  • Release the final PR to perform the Ruby 3 upgrade on staging and production

While this plan was decent, the team improved on it further with these additions:

  • Introduce Matrix builds on CI so Ruby 3 runs all of our tests as part of the normal CI workflow
  • Make Ruby 3 the default on QA environments for about a month or until we were satisfied
  • Try to split the work across isolated and small PRs (easier to review and approve)
  • Make sure to engage the primarily impacted teams!

These amazing additions may seem obvious, but they are easy to overlook when you tunnel on the overall goal. Throughout the project, each of these additions paid immense dividends.

You Don’t Simply Climb Mount Everest

Could we have deployed one MEGA pull request with all the changes? If you have read GitHub’s article on the migration path and the multi-year journey of moving from Rails 3 to 5.2, then you might have an idea why we didn’t. In short, those PRs are almost impossible to review, are incredibly risky, and doesn’t help encourage folks to right Ruby 3 compatible code.

That’s why we split up the effort across many pull requests. Once dual booting (via bootboot) and CI were in place, we went folder by folder through both the app code and tests with a PR per folder in most cases. This allowed the upgrade team to move meticulously, intentionally, and with controlled impact through our codebase to bring it up to snuff. At Wrapbook, some teams own folders or most of them, so it was also very important to get their reviews; get their advice on how to update APIs; and generally let them know the change was coming and how they can help us stay Ruby 3 compatible.

If you are curious what this looked like in practice… it was somewhere around 12-20 PRs over the course of roughly 3 months. Maintaining and upgrading a large codebase with a distributed team is not an overnight effort, but the entire engineering org was incredibly happy to see us investing in our monolith and it’s been a fantastic improvement to the app overall.

Open Source Collaboration

If you have embarked on a large Rails or Ruby upgrade then you might also have experienced some difficulties with gems your app relies on. At Wrapbook we took (and still take) a collaborative approach to these upgrades. First, we forked the gems that didn’t support Ruby 3 and proceeded to add that support. Once everything was good on our end, we PR’ed those changes to the source repositories and worked with the authors through GitHub Issues to get them merged. Not only were the authors open and receptive to our changes, but in some cases it helped improve or establish relationships with those authors that still reap benefits today.

Working in Rails we stand on the shoulders of giants, through our Gemfile. At Wrapbook, we know that Open Source maintainership can be difficult and partnering on improvements, bugfixes, and changes is the best path for everyone. It’s important to contribute back to those libraries and help them continue to grow and stay relevant. In some cases we have decided to invest our time on an ongoing basis to improve those tools we rely upon to delight customers and maintain a smooth engineering experience for our engineers.

Bugs Will Bite You

I wish I could tell you that we fought the good fight and nothing bad or painful happened. I wish I could tell you it was smooth sailing and after each PR we sipped our coffee in quiet triumph and moved onto the next piece. The reality is that at times we accidentally broke critical features. Even with the strictest intentionality, manual QA, and robust test suites… stuff slips through the cracks. However, as you know by now, this isn’t a horror story.

The truth is people rallied and supplied fixes quickly and the customer impact at the end of the day was very minimal. Each time we encountered a bug we learned and leveraged that knowledge to improve the next PR. In fact, at the very end the final PR was 100% bug free. The journey wasn’t perfect and there were struggles, but as a team we overcame whatever adversity was thrown at us and kept pushing forward. We made each other and the code better and as a result the final product was everything we hoped it would be.

Patience Is a Virtue

An important acknowledgement here is the time involved to take on a Ruby or Rails upgrade on a large app. It’s so tempting to try to crank through over a few days or even a week and release a big bang PR that upgrades the app lightning fast. To be honest the initial spike on this effort was something close to that. However, in that spike we learned the importance of small PRs, communicating with teams, and in general what was required to keep confidence high and the app running smooth. It’s not a sprint, it is a marathon. If it’s worth doing, it is worth the extra time to do it well. There will be debate, there will be altering of approach; but don’t let those things deter you because at the end of the day what’s important is that there is buy-in and everyone is on board each step of the way.

Victory!

This was probably the largest scale and longest lived upgrade I have personally been involved in. There were highs and lows and it was one hell of a roller coaster ride. Reflecting on it now, I am so deeply grateful for Wrapbook allowing - nay, supporting - a brand new team member’s lofty ambitions. We came together as a team and absolutely crushed it. Today (as of March 14, 2021) we are on Ruby 3.0 and loving every minute of it.

Privilege and Support

We absolutely have to recognize the immense privilege of being empowered to do this work. You may have tried to put an objective like this on your team or company’s roadmap, only for it to be deprioritized over and over again. Here at Wrapbook, we deeply value engineering craftsmanship, code quality, and in general empowering engineers to do their best work. This means investing the time and resources intentionally so that a large technical initiative like this can succeed. From the CTO to the VP of Engineering (Zaid), this project had the full support of the company.

An advantage we had when we embarked upon this effort was an up-to-date codebase that has been meticulously maintained through the years. Dependencies were close to or at the most recent versions, we were already on Ruby 2.7, and we were already trying to avoid Ruby deprecation warnings. We also had (and still have today) a robust test suite that was instrumental in making sure we covered our bases along the way.

The last important callout is that this effort spanned all teams and many engineers. Without their support, this effort wouldn’t have ended in triumphant victory. There are so many examples of critical collaboration but a few stick out to me. First, our VP of Engineering introduced Matrix builds into our CI pipeline so that we could run our tests against Ruby 3 and get compliant. Second, one of our wonderful Staff Engineers, Erich Machado, dove in and helped fix bugs, provided additional wonderful alternative approaches to clear a path, and at the end of the effort helped carry it with me across the finish line. The last example on the top of my mind is the final QA efforts. A huge team of folks, formally 5-6 and informally another 5-10, helped make sure everything was solid in review environments and staging before we deployed the final code to production. On a large app like Wrapbook, this kind of wide-spread collaboration and support is absolutely crucial to making a large effort like Ruby 3 possible.

Teaser

Maybe you are still skeptical. I could see how! Maybe this was some large one-off and it was likely a rare occurrence to see a large upgrade like this hit a big codebase. Well… I can safely say that this was and is not an anomaly. There are two large projects the team is working on including Ruby 3.1 and Rails 7 upgrades! Personally, I can’t wait to roll up my sleeves and dig in with the team to help deliver these projects.

Closing

If you are still here reading this, I deeply appreciate you. I hope you learned some of what it takes to upgrade to Ruby 3, but perhaps more importantly, all the moving pieces and how to work with a large team to help push something like this through. The technical changes of this effort were certainly dwarfed by the communication and collaboration necessary. Anyway, thanks so much for stopping by and joining me on this reflection of a messy, gigantic, wonderful project.

PS: If the picture painted above about empowering engineers, teamwork, and support from leadership is appealing to you… Wrapbook is hiring. Come join me and all the other Wrapbookers helping to shake up the world of entertainment payroll!