LEDGER LIVE MONOREPO PROJECT: PART 4 – THE FEEDBACK (+2 years)

This is the 4th and final entry of the blog posts series “Ledger Live Monorepo Project”, where a Ledger developer tells us the story of the Ledger Live codebase huge migration project, from a multi-repo to a monorepo architecture. If you missed the previous parts, here is a quick resume (but don’t hesitate to read them if you’re interested, they are pretty quick and very interesting): 

• Part 1 – Problematics (Make it Pain): The history of this project, how it all started. How Ledger Live, your companion app for your Ledger Hardware Wallets, was a multi-repository project at first, and the problems we encountered which led to the drastic decision of this huge migration. 
• Part 2 – The Tools (Make it Shine): How we established the list of tools to bring our plan to life, by identifying our needs and doing multiple comparisons between different candidates. 
• Part 3 – The Setup (Make it Work): We dive into the migration itself, how we did it, the challenges we encountered along the way, and what was easy or hard.

Welcome to Part 4 – The Feedback (+2 years). In this one you’ll get insights on what it feels like working with a monorepo on the long run, the challenges, the good parts, and everything in between when overtaking this kind of project.

The short term

Let’s be honest: the first few weeks / months after going live are going to be the hardest.

Slow adoption

The monorepo completely changes the way you interact with tools and affects the overall developer experience. Even if it brings a slew of quality-of-life enhancements, developers are set in their own ways, and adopting new concepts and workflows can be a slow process. There is an important educational phase to support the developers and ensure they have the knowledge needed to work comfortably. This step should not be underestimated, as it will be hard to ramp up everyone. You will face confusion, misunderstandings, and sometimes even resistance. This is why it is crucial to be pedagogic during this time.

Inconsistencies

One of the scourge of developers: “it works on my computer”. Unfortunately, this is not good enough for a project of this scale, as we learned the hard way. In the weeks following delivery, we had to stabilize any inconsistencies between the different Operating Systems (Windows, Linux, MacOS) and sometimes even between different versions of the same OS.

One of the major issues here? The toolchain:

Differences between the different tools versions across each computer made it difficult to maintain a consistent project setup. It would be resolved further down the line using the toolchain manager proto.

But let’s not be grim, a lot of good things happened as well:

The flows

While it took a bit of time to ramp up everyone, working on any of the previous projects became much easier. We unified script names (as much as possible), so learning the commands once was enough for almost all projects.

Dependencies

This was the primary reason we actually undertook this migration. All dependencies were now under the same folder, and managing everything with pnpm became easier than our previous workflows. No more publishing the libraries every time we made a typo. Everything is always up to date and can be tested in isolation or in integration.

It is now possible to create unified pull requests that work on several packages, reducing context switching between different repositories.

All these improvements enhanced the developer experience and made onboarding new developers faster. As a consequence, contributing to the project as an open source developer has also become much easier.

The Continuous Integration

This was one hell of a beast to tame. In the short term, we would say that the results were mixed. All the tests that previously used to run across separate repositories are now handled by the monorepo. However, at that time the CI was not yet optimized. Every part of it was there, but, as explained a bit later on, the CI is a long term and ever evolving piece of the puzzle.

The medium / long term

Once you’re confident about the adoption of the new developer experience, the next step is to focus on optimization.

The Continuous Integration

Since the monorepo’s inception, the CI has gone through 3 different iterations. It is, imho, the most challenging part to fine tune.

1. First iteration (that we demonstrated in the previous article): Not much here, we ported and grouped the previous CI setups into a “logical” structure: apps together, libraries together etc…

2. Second iteration: About a year later, we implemented a Github App to handle the limitation imposed by Github when working with monorepos (notably, the required checks to be able to merge a PR; it is still impossible today to contextualize which checks should be mandatory or not).

While this was a good improvement, it also required significant ramp-up time for the developers, and made maintaining and improving the CI and workflows a bit more complicated.

3. Current Setup (as of December 2024): We’ve reviewed our setup once again this year. We simplified maintenance by consolidating workflows into a single master workflow, using what we learned to split workflows and define required checks more effectively.

This, as of October 2024, is the current setup we are using on Ledger Live repository. The Github app is still there, though it does not manage the PRs CI workflows anymore. It is a helper we use to enhance our experience working on the Ledger Live repository, like triggering actions when commenting on PR…

One last thing on the CI which has an extreme importance but that has not been tackled too much in this series of articles, is the usage of custom runners. Defaulting to Github runners is definitely a nice way to start, especially since the project is open source and thus gets access to the free tier runners. However, whenever performance and speed come into play, the solution will be self hosted runners. This brings another batch of issues, be it stability, availability, configuration…

Be sure to have a strong and reliable infrastructure team, they will be the heroes for this part of the story 💪

Releasing

Releasing in a monorepo is probably the trickiest part after the CI (but very linked as all of our release process is automated), and we’ve gone through several iterations as well. 

Since we are using changeset, we had to adapt to its release philosophy. Another challenge here, ramping up the release managers to understand changeset. When everything goes smoothly, the automation of the release process is kind of easy. However, if anything goes wrong, a developer who understands the git flow and concepts must step in.

A lot has been done to prevent those quirks from happening but every so often we do have to handle it manually, albeit much easier than before.

Same as the CI, the release workflow is a complex and ever evolving component of the project.

Developer Experience

Same as previous subjects, the developer experience is in constant evolution. As stated before, we moved to a toolchain manager, proto, to handle the versioning of our tools.

We’ve also updated our turbo pipelines, redefined npm scripts, updated our Github templates…

Proto Sidequest:

Proto is a nifty little tool that wraps your command’s (node, npm, pnpm…) execution and checks for a .prototools file. If there is a mismatch between the version installed and the version in the configuration file, it prevents execution. You can install the correct versions of the tools using proto use. Once the tools are synchronized with the configuration file, you can go back to executing the commands. It’s very straightforward, robust, and perfect when working with a lot of different computer setups.

Important considerations

Here is some advice that hopefully will help you undertake this kind of task.

Be pedagogic

Educating the developers that work on your repo is important. Take time, listen, be pedagogic and available to your team. It goes a long way, especially at the beginning.

Write documentation

Make sure you document what you do. Be it the documentation for the repository, the development process, notes for future articles… It is paramount.

Accept feedback

Even though you believe you made the right choices for your team, some of them might be counterintuitive or can be done in a different way that’s more tailored to your team. Listen to your teammates, ask for feedback… Every friction point is worth a look, as fine tuning a mono repo of the size is a daunting task, and it can only be done with team work.

Enjoy the ride

It’s been said enough time: a project of this size with already-in-production applications and libraries is complex, hard and it will take time.

This is exactly what makes a mission like this incredible:

• You will learn a lot and on a lot of different topics
• You will learn not everything can be as planned
• You will learn how to PoC and discard ideas
• You will learn to accept constructive criticism and feedback
• You will learn to communicate with your whole team
• You will learn to quickly react when something is broken
• You will learn how to debug
• And so much more…

The rewards will far outweigh the challenges that arise, and as developers, we are always ready to embrace a challenge 😏 !

Last words

Personally, it has been one of the most enriching missions I have ever undertaken. I’ve grown a lot as an engineer, as a person, as a communicator and as a teammate. I can communicate my ideas in a clearer way, conceptualize projects of this size with more ease, be more independent in my work and overall be better at what I do.

Writing this series was a nice trip down memory lane of these past two and a half years. Discovering again the premises of the project, the hard times, the good times, the exciting times…

Experience point acquired: you’ve gained a level!

I hope this was as enriching to you readers as it was for me to write.

Thank you again for following me throughout this series.

Valentin DE ALMEIDA
Staff Engineer – Ledger Live