Tech | 03/01/2023
The Vault, the DApp, and the Message
Unlocking DApps, DeFi, and NFTs for institutions is Ledger Enterprise’s raison d’être. Every damn day, more than a hundred engineers, products, and designers are building together Ledger Vault, the most secure web3 platform on the market, cracking the adoption challenge. Besides this lovely sales pitch, what does it all mean from a technical perspective?
I was lucky enough to lead the development of Message Signing, the latest Vault feature that empowers its platform vision. Message Signing brings the missing piece of the platform puzzle, enabling Vault accounts to authenticate to any Ethereum DApp like OpenSea or dYdX and proceed with any actions, such as listing an NFT for sale or executing a DeFi trade order, with no concession on the security nor the governance that made the Vault what it is today. If you’re about to rage-quit reading because we’re only considering Ethereum, chill out; Message Signing is here to scale to other ecosystems.
The feature has all the ingredients for a riveting story:
- Customer-centric: with complex B2B sales cycles, promoting and aligning a roadmap with strategic partners is a good practice, and Message Signing is a key feature for non-financial brands like Farfetch, TIME, and Salesforce.
- Innovative: pushing the industry state of the art with governance and clear signing over any DApp interaction, be it a smart contract call or a message to sign.
- Challenging: firmware C implementation of EIP-712 and EIP-191 standards, JSON to Protobuf parser in Scala, orchestration of messages in Python, WalletConnect, and socket programming in TypeScript …
Let me walk you through the past twelve weeks of intense building, which made the Vault a better product, and myself probably a better engineer, through adversity to the stars.
Feature scoping and Architecture
When writing a project’s first lines of code, it’s usually hard to come up with the right level of abstraction that can absorb all the long-term future evolutions. They are unpredictable. Developers model the world with business-as-usual objects to solve their problems, and ours began with Accounts and Transactions. Integrating new currencies is a substantial part of our day-to-day job to keep pace with the industry. Unsurprisingly, engineers tailored the codebase to this process.
Stick a fork in it; Messages are joining Transactions at the governable level.
Before jumping into the feature design, It’s worth reflecting on what differentiates a message from a transaction.
Well … “from an information standpoint, [there’s] not a huge difference between, say, just sending a direct message and sending a payment” says Elon Musk while planning to turn Twitter into a bank. If it’s hard to predict whether his latest whim will ever find an application in the real world, still he got a fair point: messages, like transactions, are just blobs of data to sign and send from one point to another. Depending on the system, messages may raise different security concerns. In the Twitter system, a message is a relatively harmless piece of information: it will never drain a user account nor grant any authority to the receiver. The latest doesn’t apply to messages in the web3 system, where an attacker could list your bored APE for five dollars on OpenSea. In the web3 jungle, messages, and transactions require the same amount of security — clear signing and trusted hardware — but diverge on the following:
- Crafting: A signature scheme consists of a hashing and a signing algorithm. Crafting is the logic to apply before the hashing. When working with Ethereum, crafting messages and transactions involves a different approach to prevent collisions between the two.
- Destination: Messages are returned to the DApp after signature. Transactions are broadcasted on-chain.
Now that we see things clearly, it’s time to propose an architecture and get a validation stamp from the architects’ OGs. Hello Yacine Badiss. It’s no small feat, especially when significant changes are involved. As a feature tech owner, your first challenge is to pass the rigorous review of the architects while ensuring you can deliver on time. Seek feedback, defend your ideas, and strive for agreement. They may understand the ideal architecture better, but you are closer to the code constraints.
Let’s look at the Message Signing architecture and dive into every component. We will see them with a focus on their role in the message signature flow.
Message Signing Architecture
Don’t hesitate to return to the above schema later in the reading. Better to remember where the components live before capturing their essence.
DApp — (e.g., OpenSea)
A DApp is a decentralized application that provides a web3 service. DApps can send messages to users to authenticate their accounts and validate their actions. OpenSea sends an EIP-191 when a user logs in with a new account and an EIP-712 to confirm any NFT listing. If you’re curious about the OpenSea internals, you should read more on Seaport, their open-source marketplace protocol.
Here is the welcome message to be signed with the user account’s private key:
Welcome to OpenSea!
Click to sign in and accept the OpenSea Terms of Service: https://opensea.io/tos
This request will not trigger a blockchain transaction or cost any gas fees.
Your authentication status will reset after 24 hours.
Wallet address:
0xd8fe3244fff6d333f41347afcba66a1b1e7852f7
Nonce:
47c6302f-cd86-4766-95c7-2ba77e94f0c0
Welcome to OpenSea!
Click to sign in and accept the OpenSea Terms of Service: https://opensea.io/tos
This request will not trigger a blockchain transaction or cost any gas fees.
Your authentication status will reset after 24 hours.
Wallet address:
0xd8fe3244fff6d333f41347afcba66a1b1e7852f7
Nonce:
47c6302f-cd86-4766-95c7-2ba77e94f0c0And here is the almost EIP-712-compliant OpenSea NFT listing confirmation message:
{
"types":{
"EIP712Domain":[
{
"name":"name",
"type":"string"
},
{
"name":"version",
"type":"string"
},
{
"name":"chainId",
"type":"uint256"
},
{
"name":"verifyingContract",
"type":"address"
}
],
"OrderComponents":[
{
"name":"offerer",
"type":"address"
},
{
"name":"zone",
"type":"address"
},
{
"name":"offer",
"type":"OfferItem[]"
},
{
"name":"consideration",
"type":"ConsiderationItem[]"
},
{
"name":"orderType",
"type":"uint8"
},
{
"name":"startTime",
"type":"uint256"
},
{
"name":"endTime",
"type":"uint256"
},
{
"name":"zoneHash",
"type":"bytes32"
},
{
"name":"salt",
"type":"uint256"
},
{
"name":"conduitKey",
"type":"bytes32"
},
{
"name":"counter",
"type":"uint256"
}
],
"OfferItem":[
{
"name":"itemType",
"type":"uint8"
},
{
"name":"token",
"type":"address"
},
{
"name":"identifierOrCriteria",
"type":"uint256"
},
{
"name":"startAmount",
"type":"uint256"
},
{
"name":"endAmount",
"type":"uint256"
}
],
"ConsiderationItem":[
{
"name":"itemType",
"type":"uint8"
},
{
"name":"token",
"type":"address"
},
{
"name":"identifierOrCriteria",
"type":"uint256"
},
{
"name":"startAmount",
"type":"uint256"
},
{
"name":"endAmount",
"type":"uint256"
},
{
"name":"recipient",
"type":"address"
}
]
},
"primaryType":"OrderComponents",
"domain":{
"name":"Seaport",
"version":"1.1",
"chainId":"1",
"verifyingContract":"0x00000000006c3852cbEf3e08E8dF289169EdE581"
},
"message":{
"offerer":"0xD8FE3244ffF6d333F41347afcba66a1B1e7852f7",
"offer":[
{
"itemType":"2",
"token":"0xa83452Ef1A19BB74731Acc49635E158074Ec9b3D",
"identifierOrCriteria":"527",
"startAmount":"1",
"endAmount":"1"
}
],
"consideration":[
{
"itemType":"0",
"token":"0x0000000000000000000000000000000000000000",
"identifierOrCriteria":"0",
"startAmount":"87500000000000000",
"endAmount":"87500000000000000",
"recipient":"0xD8FE3244ffF6d333F41347afcba66a1B1e7852f7"
},
{
"itemType":"0",
"token":"0x0000000000000000000000000000000000000000",
"identifierOrCriteria":"0",
"startAmount":"2500000000000000",
"endAmount":"2500000000000000",
"recipient":"0x0000a26b00c1F0DF003000390027140000fAa719"
},
{
"itemType":"0",
"token":"0x0000000000000000000000000000000000000000",
"identifierOrCriteria":"0",
"startAmount":"10000000000000000",
"endAmount":"10000000000000000",
"recipient":"0x118aB57514481103D54fBa63a08EdDa2eBE55309"
}
],
"startTime":"1674577944",
"endTime":"1677256344",
"orderType":"2",
"zone":"0x004C00500000aD104D7DBd00e3ae0A5C00560C00",
"zoneHash":"0x0000000000000000000000000000000000000000000000000000000000000000",
"salt":"24446860302761739304752683030156737591518664810215442929806945348033033735137",
"conduitKey":"0x0000007b02230091a7ed01230072f7006a004d60a8d4e71d599b8104250f0000",
"totalOriginalConsiderationItems":"3",
"counter":"0"
}
}An attentive eye — if not abnormal — has spotted the line that doesn’t make this message an EIP-712: totalOriginalConsiderationItems": "3", . Indeed, the types definitions don’t include totalOriginalConsiderationItems.
Directly jump to the implementation section to see how we overcome inconsistencies.
Wallet Connect Bridge Server
WalletConnect is an open protocol to communicate securely between Wallets and DApps. The user can connect with WalletConnect by copying a deep link from the DApp into Ledger Vault.
Technically speaking, WalletConnect v1 — WCv1 — is an E2E-encrypted RPC coupled with a message queue. WCv1 is a protocol and an infrastructure component since it hosts the bridge server.
Here is the connection flow:
Wallet Connect Live App
The Wallet Connect Live App is a user-friendly, open-source web application that runs on Next.js technology. Its purpose is to seamlessly integrate with Ledger products and enable users to access DApps that support the WalletConnect wallet adapter. The app connects to the WalletConnect bridge server and implements the necessary RPCs (Remote Procedure Calls). These include the EIP-191 message RPCs, such as personal_sign and eth_sign (come here for a discussion on their differences), as well as the EIP-712 message RPC, eth_signTypedData.
The app transforms payloads into hexadecimal format before calling the message.sign RPC, which developers must implement to join the party.
In short, the Wallet Connect Live App streamlines the connection process, reduces payload size, and organizes related information in one place. This abstraction layer proves particularly useful with the upcoming WCv2 update, which is not backward compatible. All consumers services can continue to rely on the message.sign RPC, not worrying about protocol update breaking changes.
Front end
Ledger Vault front-end implements the message.sign RPC to trigger the message signature flow. The front end is not a trivial React application but has a fair share of complexity. To complete the message signature, it has to fit an asynchronous process — the message approvals flow — into a synchronous process — the DApp’s waiting for the signed message — while handling the wallet connect session open. To that extent, the front end receives socket events from the orchestration layer through a rabbitmq and catches NEW_MESSAGE_HAS_BEEN_SIGNED to know when the signature is ready. Ledger Vault is the first solution in the industry to bring multi-approvals over DApps messages. As said earlier, they can be as dangerous as transactions. We do not concede on security.
Personal Security Device
The PSDs are the devices we ship to our customers for interacting with the Vault. They have an embedded Secure Element (SE) running a proprietary Operating System. They communicate with the HSM through an E2E-encrypted tunnel. The users will review the message on their PSD before clear-signing in peace. Each PSD has its private key for a user to sign data. However, at the end of the approvals process, the message is signed with the Ethereum account’s private key secured by the HSM, not with any of the PSD’s private keys. The DApp will then verify the signature with the account public key using standard ECDSA maths. Let’s not sound cocky; by saying standard, I don’t mean easy but overused in the industry.
Orchestration
Welcome to the back end. The orchestration layer, or the “gate” to the friends, is a big requests state machine. It onboards new customers, passes encrypted data from the front end to the HSM, and crafts/broadcasts transactions with our internal blockchain services. It stores the history of all vault objects — User, Accounts, Transactions … — to provide our users with reporting.
The mission was to add a new Message object, reusing as much code as possible. It paved the way for abstraction over Transactions, Messages, and future governable objects with an Approvable interface. Is governance-as-a-service the future of the gate? Probably.
HSM Driver
The HSM Driver is an old Scala 2.12 component on one primary mission: transforming JSON from the gate into bytes to communicate with the HSM. Serialization is achieved with Protobuf. As a backend team who mainly use Python, It has been challenging to adapt to a harsh legacy Scala component. Despite the somehow hostile environment, I secretly love Scala. Without respect to personal consideration, the historical Scala pick for the HSM Driver saved me for message signing. Its powerful type system and the functional paradigm are the best allies to write a parser in a few lines of code that feel elegant. FlatMap that s***.
object EIP712Parser {
// String.toIntOption methods comes with scala 13 but we're stuck in scala 12 world
def toIntOption(s: String): Option[Int] = Try(s.toInt).toOption
def toIntWithDefault(s: String, default: Int): Int = toIntOption(s).getOrElse(default)
sealed abstract trait EIP712Type {
def validate: Either[EIP712ParsingError, EIP712Type]
def toProto: FieldType
}
case class INT(size: Int) extends EIP712Type {
def validate = if (size >= 0 && size <= 256 && size % 8 == 0) Right(this)
else Left(EIP712ParsingError("Fail to parse int"))
def toProto = FieldType().withType(DataType.INT).withSize(size)
}
case class UINT(size: Int) extends EIP712Type {
def validate = if (size >= 0 && size <= 256 && size % 8 == 0) Right(this)
else Left(EIP712ParsingError("Fail to parse uint"))
def toProto = FieldType().withType(DataType.UINT).withSize(size)
}
case class BYTES(size: Int) extends EIP712Type {
def validate = if (size >= 0 && size <= 32) Right(this)
else Left(EIP712ParsingError("Fail to parse bytes"))
def toProto = FieldType().withType(DataType.BYTES).withSize(size)
}
case class STRING() extends EIP712Type {
def validate = Right(this)
def toProto = FieldType().withType(DataType.STRING)
}
case class STRUCT(name: String) extends EIP712Type {
def validate = Right(this)
def toProto = FieldType().withType(DataType.STRUCT).withStructName(name)
}
case class ADDRESS() extends EIP712Type {
def validate = Right(this)
def toProto = FieldType().withType(DataType.ADDRESS)
}
case class ARRAY[T <: EIP712Type](eip712Type: T, size: Int) extends EIP712Type {
def validate =
if (size >= -1) Right(this) else Left(EIP712ParsingError("Array size can't be negative"))
def toProto = FieldType().withType(DataType.ARRAY).withArrayType(eip712Type.toProto).withSize(size)
}
case class EIP712Field(name: String, `type`: String) {
def parsePrimitiveType(typeDesc: String)(implicit customTypes: Set[String]): Option[EIP712Type] = {
val UintPattern = "uint(.*)".r
val IntPattern = "int(.*)".r
val BytesPattern = "bytes(.*)".r
typeDesc match {
case UintPattern(size) => Some(UINT(toIntWithDefault(size, 0)))
case IntPattern(size) => Some(INT(toIntWithDefault(size, 0)))
case BytesPattern(size) => Some(BYTES(toIntWithDefault(size, 0)))
case "string" => Some(STRING())
case "address" => Some(ADDRESS())
case _ if (customTypes contains typeDesc) => Some(STRUCT(typeDesc))
case _ => None
}
}
private def parseType(typeDesc: String)(implicit customTypes: Set[String]): Either[EIP712ParsingError, EIP712Type] =
if (typeDesc(typeDesc.length - 1) == ']') {
val arrayPattern = "(.*)\\[(.*)\\]".r
typeDesc match {
case arrayPattern(subtypeStr, size) => parseType(subtypeStr).flatMap(ARRAY(_, toIntWithDefault(size, -1)).validate)
case _ => Left(EIP712ParsingError("Fail to parse array"))
}
} else
parsePrimitiveType(typeDesc) match {
case Some(t) => t.validate
case None => Left(EIP712ParsingError(s"$typeDesc is not a valid EIP712 type"))
}
def toProto(implicit customTypes: Set[String]): Field =
parseType(`type`) match {
case Right(validType: EIP712Type) => Field(name, Some(validType.toProto))
case Left(error: EIP712ParsingError) => throw error
}
}
case class EIP712Message(
types: Map[String, List[EIP712Field]],
domain: Map[String, Any],
@JsonProperty("primaryType") primaryType: String,
message: Map[String, Any]
) {
def validate: Either[EIP712ParsingError, EIP712Message] = {
if (!(types contains primaryType))
return Left((EIP712ParsingError(s"primaryType $primaryType is not described in types")))
val expectedMessageShape = types(primaryType).map(_.name).toSet
if (expectedMessageShape == message.keySet)
Right(this)
else
Left(EIP712ParsingError("message doesn't match its description"))
}
def toProtoStructs: Seq[Struct] = {
implicit val customTypes = types.keys.filterNot(_ == "EIP712Domain").toSet
types.map { case (structName, fields) => Struct(structName, fields.map(_.toProto).toSeq) }.toSeq
}
def toProto: EncodedMessageDescription =
validate match {
case Right(_) =>
val description = Eip712Description(toProtoStructs, primaryType)
EncodedMessageDescription().withEMsgDesc(MessageDescription().withEip712Desc(description).toByteString)
case Left(malformedError) => throw malformedError
}
}
def fromRawMsg(rawMsg: String): EIP712Message = {
val mapper = JSONMapper()
val content = Utf8.unapply(Buf.ByteArray(HexUtils.valueOf(rawMsg): _*)).get
mapper.readValue(content, classOf[EIP712Message])
}
}Ledger’s teams are working on a new Python 🐍 component to replace the HSM Driver. It will probably get simplified, and I will probably regret it.
If Scala is your thing, we’re recruiting at different levels of experience in the backend services team. Don’t be afraid. This team works on projects using Scala 3 with FP industry best practices and they have serious reasons. We have a really strong Scala culture.
Hardware Security Module
The HSM is a Secure Element with our business-specific operating system. It is responsible for generating private/public keys, deriving account addresses, and signing transactions, messages, and other vault objects. Coding in this environment requires extreme attention to detail. Resources are limited in terms of memory and computation, and C programs running in the HSM are the last rampart to potential attacks. If one vulnerability is introduced here, it could have disastrous consequences. To manage the risk, the Donjon reviews the firmware code before it can be deployed in production.
HSM message signing C script produces the blob of data to sign (i.e., the hash) from the raw message, applies checks conformally with the EIP-191 & EIP-712 specs, and proceeds with the signature at the end of the approval flow.
The implementation, a collaborative experience
This post has already gone on for too long before introducing Nicolas Chataing. The man, the myth, the legend. He is the firmware engineer behind the message-signing C code. His job was to read the EIP specs a million times and craft the one-and-only valid hash for any raw message. As explained, the produced hash would be signed and returned to the DApp. A single invalid byte in the hash would lead to a completely different signature for the DApp to reject. It’s painstaking work. To help him, I found the tests of the go-ethereum’s message signing implementation. We used them to compare the output hash from our C program vs. the official implementation.
That’s not it. Nicolas wrote the protobuf file to define how to communicate with the HSM from the HSM Driver. Compiling this file with scalapb would generate Scala classes that can serialize bytes sequence for the HSM.
So, what does the HSM expect?
For EIP-191, nothing but the raw message in hexadecimal. For EIP-712, it’s another story. I had to parse the “message” payload to RLP- encode the values only, reorder, and drop the undefined fields, if any. A field is undefined if not specified in the types dictionary. (Cf., the OpenSea TotalOriginalConsiderationItems). Dropping them from the message before hashing was not intuitive nor specified anywhere. We just tried it in our despair after dozens of rejected signatures. That’s how web3 development is: sometimes you try, sometimes you read the contract source code, and sometimes both.
A second parsing phase ensured the message values respected their type definition. Being strict with the type checking but generic with the message form enables us to minimize the attack surface while coping with EIP-712 inconsistencies.
As you can see, maximum work is done before the message reaches the HSM for several reasons:
- Reducing the workload in the HSM
- Fail-fasting with dedicated error messages
- Splitting the responsibilities across teams
And that’s it! All these efforts to finally list an NFT for sale on OpenSea directly from Ledger Vault 🎊
What’s next?
In a previous post, I discussed the importance of EVM transaction simulation and how it works internally. If you’re interested in learning more, you can get your hands dirty with this workshop I’ve prepared for ETH Bogota. Ledger has since launched its transactions predictive impact platform. But does it handle off-chain data, like messages? Recently, Uniswap’s update to their protocol created a vulnerability exploiting an EIP-712 message.
The answer is YES, and it’s already on the roadmap for the Web3 checks team. They have you covered 🛡
