The Ledger Way – Chapter 3: Tech Practices

Ledger VP of Engineering Carl Anderson focuses on our tech practices in the third and final blog post in this series exploring what it’s like to be an engineer at Ledger. If you missed the previous posts, find Chapter 1 here and Chapter 2 here. Plus, learn more about Carl on Linkedin here, and check out our Careers page for more information on how to join the team.

When it comes to Tech Practices at Ledger, our North star is Continuous Delivery and the importance of a DevOps culture, where, as software engineers, we fully own the application layer: “we build it, we own it”.

Continuous delivery is a set of capabilities that enable us to get changes of all kinds — features, configuration changes, bug fixes, experiments — into production or the hands of users safely, quickly, and sustainably.

But Ledger didn’t invent the concept of Continuous delivery so if you want to know more about it in general, I strongly recommend you to read Accelerate. Nonetheless, here’s how we’re going about it.

Key Principles

There are five key principles dear to us:

Build quality in

At Ledger, we build quality in, supported by the tools and talent allowing us to detect and resolve issues quickly. Issues can be fixed straight away when quality is a priority.  

Work in small batches

Organizations tend to plan work in big chunks — whether building new products or services or investing in organizational change. By splitting work into much smaller chunks focused on measurable business outcomes, we get essential feedback allowing us to correct its course when necessary. 

Even though working in small chunks adds some overhead, it reaps enormous rewards by allowing us to avoid work that delivers zero or negative value for Ledger.

A key goal of continuous delivery is changing the economics of the software delivery process so the cost of pushing out individual changes is very low.

Computers perform repetitive tasks; people solve problems

One important strategy to reduce the cost of pushing out changes is to invest in simplifying and automating repetitive work that takes a long time, such as regression testing and software deployment. We free up people for higher-value problem-solving work instead, such as improving the design of our systems and processes in response to feedback.

Relentlessly pursue continuous improvement

The most important characteristic of high-performing teams is that they are never satisfied: they always strive to get better. We make improvement part of everybody’s daily work.

Full ownership

As we have learned at Ledger, having segmented teams with devs focusing on throughput, QA on quality, and operations on stability only gets us so far.

With each team focused on a system-level outcome rather than our customers, we attain “ok” performance. We believe excellence can only be attained by fully owning the whole as a single DevOps team.

In Practice: How We Do It

For Continuous Delivery to be implemented in our teams, the following foundations are necessary:

Comprehensive Configuration Management

It should be possible to provision our environments, build, test, and deploy our software in a fully automated fashion purely from information stored in version control. Any change to environments or the software that runs on them should be applied using an automated process from version control. This still leaves room for manual approvals — but once approved, all changes should be applied automatically.

Continuous Integration

Many software development teams are used to developing features on branches for days, or even weeks. Integrating all these branches requires significant time and rework.

Following our principle of working in small batches and building quality in, we’ve learned, thanks to our code design and techniques like feature flags, to keep branches short-lived (less than one day’s work) and integrate them into trunk/master frequently. Each change triggers a build process that includes running unit tests. If any part of this process fails, developers fix it immediately. The main branch should be releasable at all times.

Continuous Testing

Testing is not something that we initiate once a feature or a release is “dev complete.” Because it is so essential, we treat testing as an integral part of the development process.

We run automated unit and acceptance tests against every commit to give developers fast feedback on their changes. Furthermore, developers are able to run tests on their machines in order to triage and fix defects faster.

Since testers are freed from basic acceptance testing, they can perform exploratory testing continuously against the latest builds to come out of our CI. Finally, we’re not “done” with any work until all relevant automated tests have been written and passed.

Security baked-in

Our mission is to make it secure and easy for users to access digital ownership.

This means our software is used to manage digital value and therefore security is paramount and cannot be compromised. Just as testing, this cannot be neglected.

At Ledger, the development team works closely with the Product Security team every step of the software delivery lifecycle, from design through demos to helping with test automation.

Architecture

As per our “Aligned Autonomy” principle (see previous blog post), there’s no pre-defined, one-size-fits-all architecture. Since teams are fully responsible for their code — after all, they’ll be operating it, maintaining it, and evolving it — they have an inherent responsibility in choosing their system’s architecture. Of course, each team also has the help of the architecture team and their broader view of all systems and projects at Ledger.

The guideline here is to achieve a loosely coupled, well-encapsulated architecture with an organizational structure to match, which unlocks two important points:

1. First, we can achieve better delivery performance, increasing both tempo and stability (i.e. avoid regressions) while reducing the burnout and the pain of deployment.
2. Second, we can substantially grow the size of our engineering organization and increase productivity linearly—or better than linearly—as we do so.

Check out Ledger’s careers page to join the engineering team and find the previous blogs on Ledger’s working culture and team practices here!