Enter the arena

In this blog, I’m going to talk to you about my journey towards Functional Programming (FP). It all started 6 months ago when I joined Ledger as a Back-End Engineer.

The goal is not to lay down how smart I am as a programmer, but rather to share the difficulties and joys of this ongoing journey. This adventure is quite a challenge, because it’s more than just learning a new language, it’s a whole new skill.

The beginnings

In my previous job as a Principal Software Engineer, the main challenge has always been performance. For instance, how to build a system able to respond quickly and absorb a high load to have the best possible user experience. At first, I was working on the client side i.e. the device (first on mobile phones then on video decoders). Throughout the years, thanks to the evolutions of the internet, the client has become lighter and lighter to become a simple web browser and most of the domain specific logic shifted on the server side. This also created a much better user experience to make it more interactive, more collaborative (e.g. recommendations and live feeds). The fact is technologies were changing, and I knew I had to evolve as well.

The shift

But how to get there? The gateway to the Back-End for me was Java. With 10+ years of experience in Java, I decided to take the path of Scala and FP (Functional Programming). In Java 8, there is support of Lambda functions and streams. But Scala takes FP to a whole new level. It is not just calling map and flatMap on collections but a new programming paradigm.

Functional Programming is based on pure functions and immutable values. The gain on the performance side is obtained by using parallelism to get the full advantage of multiple cores on the server and as the values are immutable, we cannot have 2 threads modifying the same value and concurrency issues are much easier to deal with.

Scala offers a very rich ecosystem of FP libraries and an active community of developers maintaining them. Take for example Cats, Cats Effect, FS2, http4s, ZIO and more.

Every developer who made the path from Imperative to Functional will say that it requires a non-trivial effort. This is exactly what attracts me and stimulates my curiosity. How can one pretend to be a Senior Developer with 20 years of experience without having FP in their programming toolbox?

But is it going to work? Isn’t it better to stay in my comfort zone?

Leaving the cave

To illustrate this, I will use Plato and the allegory of the cave.

The subject and his fellows are in a cave and see shadows projected on a wall. They believe the cave is the whole world, and that the shadows on the wall are real. But what is projecting these shadows? Where are they coming from?

The prisoners are frightened to leave the comfort and warmth of the cave. Some of them are curious but most of them prefer the comfort zone and do not dare to go outside.

One day, a prisoner decided to leave the cave to find out the truth.

When he left, he was so dazzled by the light. He couldn’t stand it at the beginning. There was too much for him to bear. At the same time, he was so marveled by what he saw that he resisted his urge to go back to the cave. He couldn’t stand what he saw and all that light coming at once. But instead of giving up, he allowed himself some breaks while carrying on and transitioning progressively.

The idea is that exploring the unknown can be scary, and learning a new skill can be discouraging at first. But you must not give up, just take it one step at a time. It is worth it.

Challenge and learning curve

This is what happens when you start a new job on a new subject. You feel so excited and enthusiastic about the unknown that you say to yourself: “Yes I want to do that!”. At the same time you think about your previous position, and remember how comfortable it was to be an expert.. Everybody came to you when they had questions, and today you are the one asking questions all the time. There is also that imposter syndrome you feel from time to time, which is really part of the process. There are times when you feel ignorant and stupid.

There is so much work to do on the Back-End to make it fast and scalable. When you make a transaction online on your favorite website, you don’t care if there are hundreds or thousands of customers before you, you do not queue, you just get served.

And not only performance, there are also other problematics : code maintainability, concurrency issues introduced by parallelism. FP promises to make developers’ lives easier by using “The Category Theory” and function composition. FP says “Don’t worry, trust the mathematicians, they have the solution for you”.

FP in Scala offers the right abstraction to make the code composable, evolutive and less prone to bugs.

At Ledger we are more than 300 engineers, including 30 passionate Scala devs. Most of them did not program in Scala before and brought their own understanding of the subject. There is a safe space to learn and share your learnings. From day one and during the onboarding, I received all the help and encouragement I needed from my colleagues. My managers trusted my capacity and potential and believed that in the end, it would be good for the company.

Of course, this is not an easy task and the path isn’t just a straight line. It is just like learning any new skill.

Learning curve

The first time you read the definition of a monad, you ask yourself why on earth did they invent such a thing? And then little by little you start understanding its utility and the problem it solves. I can humbly say now that it allows to chain operations sequentially, even asynchronous ones while taking into account error cases.

Practical example

To give you a taste, please have a look at this simplified example inspired from our code base, it is an extract from our Counter Value Service. This is the service that shows the price of Bitcoin in the user fiat currency, on the Ledger Live main page.

Ledger Live market page

The service sequences asynchronous operations, like read from the cache, see if we found a valid entry otherwise perform an http request to an external provider and upon reception insert that entry in the cache.

The IO monad does the magic of this sequencing in a pure functional way and with the syntactic sugar from Scala, the code ends up looking like imperative one.

case class Price(from: String, to: String, price: BigDecimal, expirationDate: Instant)

trait LatestRepository {
 def insertPrice(from: String, to: String, now: Instant): IO[Unit]  = ???
 def selectLatestPrice(from: String, to: String): IO[Option[Price]] = ???
}

val repo: LatestRepository = ???

def fetchAndInsert(from: String, to: String, now: Instant) =
 for {
   // sequence 2 asynchronous operations: a http call and a database insertion
   priceFromProvider <- fetchFromProvider(from, to)
   _                 <- repo.insertPrice(from, to, now)
   // eviction of expired values from cache is handled by a background task
 } yield priceFromProvider

def getCounterValue(from: String, to: String): IO[BigDecimal] =
 for {
   now            <- Clock[IO].realTimeInstant
   priceFromCache <- repo.selectLatestPrice(from, to)
   res <- priceFromCache match {
     case Some(p) =>
       if (p.expirationDate.isAfter(now))
         // found a valid value in cache, return it
         IO.pure(p.price)
       else
         // value in cache has expired
         fetchAndInsert(from, to, now)
     case None =>
       // no value in cache
       fetchAndInsert(from, to, now)
   }
 } yield res

Without monadic composition, the code would have had much more boiler-plate and headache to handle asynchronous operations and errors. FP in Scala makes programming much easier to handle than working with traditional techniques, such as threads, locks, and callbacks.

This is just a small example but there is so much more to say when effectfulness is explicitly modelized, like for instance:

• Parallelism is easy to handle,
• Retry mechanisms and cancellation are also very easy to put in place,
• Resource allocation with no worries about release, especially interdependent resources.

Mathematics and Category Theory

At the time I write these lines, there are still aspects of the IO monad I do not fully understand, like its mathematical foundations. In this website, we see its definition in Category Theory:

A monad is a structure that is a lot like a monoid, but that lives in a bicategory rather than a monoidal category. In other words, the concept of a monad is a vertical categorification of that of a monoid.

This is intimidating right ? Actually, we do not really need to fully understand these foundations to get the benefits of monadic composition in our programs. There are courses on Category Theory available and I already started digging into the subject but the gap is still huge between the theory and the usage in programming.

For the time being, I am being pragmatic and focusing on the monadic effects we can use in our day to day programming. Like any developer, we have roadmaps and deliveries and hence need to find the balance between investing time to learn but also delivering and helping the team.

Conclusion

Please see this story as an invitation to enter the arena, put yourself a little bit at risk. Try new ventures. Be up for the challenge. When faced with two paths, choose the arena even if it is easier to throw rocks from the sidelines. It is scary and lonely in the arena but this is where the growth happens.