LEDGER LIVE MONOREPO PROJECT: PART 3 - THE SETUP (Make it work)

This is the 3rd entry of the blog posts series “Ledger Live Monorepo Project”, where a Ledger developer tells us the story of the Ledger Live codebase huge migration project, from a multi-repo to a monorepo architecture. If you missed the previous parts, here is a quick resume (but don’t hesitate to read them if you’re interested, they are pretty quick and very interesting):

Part 1 – Problematics (Make it Pain): The history of this project, how it all started. How Ledger Live, your companion app for your Ledger Hardware Wallets, was a multi-repository project at first, and the problems we encountered which led to the drastic decision of this huge migration.
Part 2 – The Tools (Make it Shine): How we established the list of tools to bring our plan to life, by identifying our needs and doing multiple comparisons between different candidates.

Welcome to Part 3 – The Setup (Make it Work). This one is probably the most dense of this series, since we dive into the migration itself: how we did it, the challenges we encountered along the way, and what was easy or hard. We made our decisions, chose our tools, and gathered our strength. Now, it’s time to bring it all to life.

New repository

A monorepo, as the name suggests, is an architecture based on a single repository that consolidates multiple projects under a single umbrella. We talked about the advantages of a monorepo versus a multi-repo architecture in Part 1 of this series. In our case, we decided to name our new Git “umbrella” repository Ledger Live. But starting from scratch with a brand new repository means we need to answer a very important question first: what about history?

Bring it all back

A crucial aspect of our migration was the ability to access the complete history of each project—not just from the inception of the monorepo, but from the entire lifespan of each individual project. Thankfully, Git is an incredibly powerful tool. We skillfully routed each project’s history to its own package within the monorepo. To facilitate this, we set up a temporary cron job that synchronised all repositories daily. By rebasing our work on the latest commits from the integrated repositories, we ensured a seamless transition.

💡 If you’re curious about the scripts we used for this process, you can find them here https://github.com/LedgerHQ/ledger-live/blob/develop/tools/scripts/monorepo.sh and here https://github.com/LedgerHQ/ledger-live/blob/develop/tools/scripts/sync_remotes.sh

With everything in place, we were well-prepared to begin integrating the tools.

PNPM

The switch from yarn to pnpm as our dependency manager turned out to be more tricky than anticipated. However, we tackled it step by step. For the record, we explained in detail this decision in Part 2 of this series, but the main reasons were disk efficiency, speed and the built-in support for workspaces/monorepo architecture.

The easy parts

Workspace Setup: During the initial repository setup, we defined the file architecture. Now, it was time to enable PNPM’s workspace feature. This involves creating a configuration file called pnpm-workspace.yml. In this file, we specify the paths to target the various packages within the monorepo so that PNPM can recognize them.

Root package.json: In a monorepo setup, we always work from the root folder. Although the root package.json doesn’t represent a standalone project, it acts as an orchestrator, allowing access to all the different projects it houses. To achieve this orchestration, PNPM leverages a powerful feature called filters. We’ve set up convenient shortcuts (aliases) for each project, making it easy to access them from the root:

root packages.json
{
	"name": "ledger-live"
  "private": true,
  // ...
  "scripts": {
    // ...
		// aliases
    "desktop": "pnpm --filter ledger-live-desktop",
    "mobile": "pnpm --filter ledger-live-mobile",
    // other aliases...
  }
}

These filters operate based on the name property in each project’s package.json, as defined in the pnpm-workspace.yml file. With this setup, we can conveniently access any project’s package.json using our aliases.

pnpm desktop dev
# this would run the `dev` script from the ledger-live-desktop's package.json

The hard parts: dependencies

I don’t think we’ve touched on this yet, but pnpm is by default much more strict than yarn or npm . So what do we mean by pnpm being strict ? Pnpm expects you (the developer) to carefully declare the dependencies you are using in the correct dependencies group (dependencies / devDependencies / peerDepencencies). If you don’t, compilation/builds will break. All in all, this is a much better approach as it makes sure that any dependency required in your production code is correctly declared where it should. This is fine when you don’t rely on dependencies and have full control over your code. Yet, this is not the most common setup, as most projects usually work with external dependencies. It is our case as well, and this is when we realised that open source libraries are not always as thorough as we would have hoped, or at least the tools they used were not.

The TLDR:

with pnpm, you cannot in your production code import or require a dependency that is not in your dependencies in the package.json
with yarn or npm it was okay to build / compile even if the source was not in dependencies but was available in devDependencies (it actually goes deeper than this as it does the same with transitive dependencies not even declared in your package.json)

This conundrum had us creating a few helpers to be able to add the missing dependencies to the external libraries we were using. You can find the helpers and how they are used here:

https://github.com/LedgerHQ/ledger-live/blob/develop/tools/pnpm-utils/index.js
https://github.com/LedgerHQ/ledger-live/blob/develop/.pnpmfile.cjs (see the number of libs we have to patch 😱)

After taking care of all the pnpm unsupported libraries, our next step was to unify the version of the libraries and tools we were using as much as we could.

eg: React, RxJS, Jest, Eslint, Prettier, Typescript…..

Then, one missing part was to replace in each project’s package.json all the dependencies that were brought into the monorepo by "workspace":

desktop package.json
{
   "dependencies": {
    "live-common": "workspace:*"
    // ....
  }
}

This would ensure that when we install our dependencies, it would fetch the ones from the monorepo instead of looking for the ones on npmjs.org (thus achieving the wanted behaviour of always having up to date dependencies).

The really hard parts: symlink

As PNPM works with symlink to save disk space, it was a whole rodeo to make it work with the (now supporting symlink) metro-bundler for React-Native.

This was the price to use bleeding edge tools, not everything was supported out-the-box, and React Native was a big one. To compensate for this, we sometimes had to create our own custom scripts / tools to make it work.

For react-native, we made a custom metro resolver which you can find here: https://github.com/LedgerHQ/ledger-live/blob/db77ae6811c0094be139d461bc69eb3c6cc7ffe1/tools/metro-extra-config/index.js. This would handle the symlink for the metro bundler (officially deprecated since metro now supports symlinks).

Turbo

Alright, now that we have successfully setup pnpm, it’s time to work on automation and create local pipelines using turborepo . Turbo analyses the package.json file to create a tree of dependencies, and from there can order dependencies of scripts.

Example of a turbo.json

{
	//...
	"pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", "lib/**", "lib-es/**", "build/**"]
    },
		// other pipelines
}

Here we define a name for our script: build

Then we make it depend on all its dependencies build script using the ^. This means it will work in cascade and start building the lowest dependency in the tree, all the way back to the target.

Example of some scripts from the root package.json

{
	//...
	"scripts": {
		"build:lld": "pnpm turbo run build --filter=ledger-live-desktop",
		// ...
	}
}

When we run pnpm build:lld , the script will use turbo run to build all the dependencies, all the way up to ledger-live-desktop (the filter key is used on the name key of the package.json files).

The outputs key will cache the outputs specified by the globs in the array. Hence, the subsequent build:lld commands will only replay the logs and use the cached outputs (for all unchanged projects), allowing for much faster build/compile time on subsequent runs.

We can now order our builds and use cache to have a faster build time.

Builds

This is where we spent most of our time once the configuration part was done. We ran the pipelines and deep dived into each error that would pop out, until we had fine tuned pipelines that would work the way we wanted.

One of the notable upgrades on the builds was ledger-live-desktop. It’s a desktop application targeted at the three major desktop OSes (macOS, Linux, Windows).

We were able to improve the build system to only include the required libraries instead of the whole node_modules . Again, we made these improvements to enhance the electron-builder default behaviour. This brought the size of the bundled application from ~1gb down to 200mb. One last quirk was to define all node internals we did not want to bundle in the application.

💡 All the tools to make this happen can be found here https://github.com/LedgerHQ/ledger-live/blob/develop/tools/native-modules-tools/index.js

Automation

We finally made it: it works locally 🚀 !

It was time to move all our pipelines back into our Continuous Integration. First of all, we wanted to have the same tests that all the previous repositories were running, which was actually a good thing because we had a nice setup already. The most complicated parts were:

How will we split the jobs so they don’t get clogged up too long?
What made sense to test together?

The most interesting part was that we could now test in integration the code as a whole. This means that, as long as our branches were up to date (with develop branch), we would always test the latest version of each part of the code/libraries.

Continuous Integration

This is how we decided to split up our tests:

At the time of writing, this has been reworked and improved with a custom Github application, as we felt the Github limitations around status checks were too restrictive.

We have several workflows running in parallel on each push on a PR, and here is what happens:

Codecheck: We run a type checker and a linter on the code base.
Desktop Build: We create a build of Ledger Live Desktop for each platform (Linux, Windows, macOS), majorly used for QAs and Devs.
Mobile Build: We create an Android build (some caveats to build an iOS executable without signing it), and we do all the step up until the actual build on iOS (this helps us finding any issue regarding gems/pods installed with React Native and libraries)
E2E Mobile Tests: We run a suite of e2e tests on mobile using Detox, which uses an emulator to run the applications and automate user interactions (as a headless browser would). We do screenshot testing during this phase and report back whenever we have unmatched screenshots.
E2E Desktop Testing: We run a suite of e2e tests on desktop using Playwright, which acts like a headless browser but can also spawn electron instances. We do screenshot testing during this phase and report back whenever we have unmatched screenshots.
Test Libraries: We run a suite of unit tests (jest) on all our libraries (libs folder).

We use a combination of parallel workflows that allows us to run a type checker and linter on the code base, creating builds for both desktop and mobile platforms, and running e2e tests on desktop and mobile with screenshot testing. Additionally, we run unit tests on all of our libraries. This ensures that we have up-to-date tests for our project, and that we can always test the latest version of each part of our code.

Documentation

The last part before we could ship it to all the Live developers was the documentation.
There were two major steps in this:

Bring all the existing documentation from the previous repositories back to the Github Wiki of the new repository. We had to organise those documents into something that made sense for us and for the other developers, as well as update them to reflect the changes brought by the migration.
Document how to effectively use this new repository in our day to day’s work. We followed the best practice and made an exhaustive README and CONTRIBUTING to help ease the transition. We had anticipated it would not be a seamless transition and also provided a few talks to the team to give as much information as we could before making the switch.

The Switch

We made it: it works locally, it works in the Continuous Integration, it is documented, and we are ready to help any developer who would have rising issues.

We pushed the big 🟢 button.

The monorepo was live!

Conclusion ?

After months of work undercover, it felt like a proud moment to finally deliver this new developer experience to the Ledger Live team. The following weeks would prove challenging as issues kept arising for different developers as well as the CI. The adoption took time: bugs were squashed, the CI was fixed a million times, we tackled all the quirks each OS could have… but we made it. The new developer experience has significantly improved the team’s day-to-day workflow, and even though the road was rocky at some point, we’re glad we made the switch.

In the next article, we’ll look back on over two years of using this monorepo at Ledger Live. We’ll explore the good parts, the challenges, and the team’s overall experience after this journey.

Valentin DE ALMEIDA
Staff Engineer – Ledger Live