When I took over as the release manager of a self-custodial Bitcoin wallet, I felt uncomfortable. A gun to my head and I could and would have released whatever change to our product, netting the guy with the gun tens if not hundreds of millions of dollars in Bitcoin from our customers.

Knowing a solution to this problem, I told my boss that I needed him to prioritize working on what would protect our customers from this horror scenario: all users losing all funds all at once.

Back then, we explored how other providers dealt with the issue and we were surprised to learn that most simply did not deal with it at all.

Reproducibility explained

The solution referred to is "reproducibility" and works as follows:

Let's say Alice is the release manager of a Bitcoin Wallet. Whenever the team agrees that it's time to release all the new features and bug fixes they had been working on this last week, she presses some buttons on her computer to generate the application from the agreed upon source code. "Source code" is a folder with sub-folders full of human readable text files and media files such as app logo or payment-received-sound. She then uploads the result - imagine a zip file - to the team's server.

Her colleague Bob now can do the same process only that instead of uploading his result, he downloads Alice's result and compares the two. If they are bit for bit identical, Bob knows that Alice did a good job and did not plant some backdoor. At least not after the team agreed on the code looking good. If they are not identical, then there are several possibilities:

• Alice and Bob did compile different code. This might be for harmless reasons like Bob not having updated to the exact same version of the source code as Alice. If Alice was on the wrong version: Great! The team caught a wrong release before delivering it to the clients. But it could also be because Alice was under duress and planted a backdoor in their product. Or hackers planted a backdoor via a virus in Alice's computer.
• The app is not reproducible. Making and keeping an app reproducible is an ongoing challenge. Tools used to generate an app can behave differently on different computers. For example they might bake the username into the result. If the two results upon closer inspection differ only by one containing "alice" and the other "bob" in the same 2053 spots, the team might decide this is good enough and clearly not malicious. Or the images from the source code get compressed slightly differently on the two computers. Again, they might deem the deviation benign. The often not technical boss might provide pressure to release certain features or bug fixes and without users caring about this very technical aspect, almost all teams rely on gut feelings when it comes to some non-reproducibilities to rush out long awaited new versions.

Co-worker Bob signaling that release manager Alice's upload is "clean" should be part of every wallet's release process.

A side project was born

While still working for the wallet app, I founded WalletScrutiny in November 2019 to educate and to monitor where we stand with regards to popular wallets. Over the years, the situation improved although only very slightly.

WalletScrutiny only considers publicly available data, so closed source projects might internally validate builds and thus protect their users from a release manager under duress but we cannot attest to this and less can we make promises about the provider not executing an exit scam often referred to as "retirement attack" themselves.

The problem with rankings

As you read this post thus far, you are probably quite tech savvy. Most wallet users are not. Most want a "verified" wallet - whatever that means - and the top category of a security focused list of wallets feels the closest to being "verified". And that is why I'm writing this today:

A product I'm highly uncomfortable with, made it into the top category - Samourai Wallet. For years they lied about us, bombarded us with personal attacks, came up with creative ways of deceiving their followers into believing their product actually was reproducible even though nobody neutral confirmed this and now they finally came around and provided a "reproducible" build of their product version 0.99.98g with the fingerprint 7826e46e46133598f8498f4d192d2327a46bc2f2abad7436cbc5b94c06175f61. We attest that using their source code and build instructions it is possible to reproduce the app found on Google Play (except for the signature part, which is expected in all reproducible Android apps).

Being transparent doesn't mean to be good

At the same time we urge users to extreme caution with this product as its main feature - coinjoin using the Whirlpool protocol - is severely flawed. Samourai Wallet sends the list of all Bitcoin addresses to the Whirlpool server, allowing them to correlate transactions to accounts. While they claim they do not save this data, they do admit that the data is sent to the Whirlpool server.

Protocols exist that avoid this problem, leaving the coordinator - the role of the Whirlpool server - completely in the dark of who's coins are whose.

Samourai is not a provably privacy preserving product as advertised and given past behavior points to there being more than meets the eye.

Also please consult independent reviewers as we did not review the code but that disclaimer we have with every reproducible product.