November 18, 2016

How to (Actually) Process Bitcoin Transactions

Filed under: bitcoin — Benjamin Vulpes @ 8:29 a.m.

James Stanley developed opinions about how to handle bitcoins while building a nifty, supposedly anonymous, Bitcoin/telephony bridge. Unfortunately, he labors outside the walls of the Most Serene Republic of Bitcoin, and so he's missing the typical 30-point bump in IQ that can be had for simply reading the logs1. As they say, "build one to throw away"--and it's never truer than if you don't know what you're building in the first place2.

My first treatment apprently didn't expound on the subject in adequate depth, and so here I am again: correcting the internet. This guide is less "do these three simple things" and more "do these two impossible things". What, you wanted to learn something, didn't you?

- you (can) run a full TRB node
- you can extract transaction data from raw blocks

Running a full TRB node is table stakes for this game. It means getting up to speed with "V", The Republic's software distribution tool, and running the daemon you'll compile with mod6's offline build system on a semi-serious piece of metal. The most-public node that I run consumes over 5.7 gigs of RAM, and its current uptime is ~48 days3. Use a process supervisor (I'll leave writing up the details to trinque, but start with runit, and its accompanying log service svlogd. A TRB node will shit several gigs of log lines in a month, so do consider accounting for the disk consumption.

Once your node runs reliably, you need to sync it. If you don't already have a synced node, there's no way around simply downloading the whole blockchain and verifying it sequentially (pity the poor Ethereum bagholders, whose blockchain is over 75 (!) gigs at this point4. ). If you're a lucky sod, and are running a previously-synced 0.5.X node, you can simply copy the bitcoin data directory to your new box and start the new bitcoind process over there. Do remember to stop the synced node first, lest you bring down the data-integrity gods' wrath upon your head. Given the times involved, if you have a 0.6.X, and possibly a 0.7.X series node running and synced, consider pulling the same trick. I have no idea if it'll work, but I am eager for reports on that kind of science.

With node in hand, you may now proceed to dissect blocks (if you've pressed a standard TRB, it has a 'dumpblock' command. Go forth and figure out how to use it). I've already written up how to extract block headers, and while I failed to beat the "binary-types" library into cleanly parsing the binary blocks (see "binary-types:define-binary-class bitcoin-block-header" for what a clean binary definition looks like in that DSL), I turned around and implemented all of the slicing code behind mimisbrunnr in just a few days once I decided to do it by hand and forgo the bijective transform5. This code is not publically available, but if you show up in #trilema, get in the WoT and ask nicely, I might give you a copy.

Given the ability to slice blocks into headers and transactions, you may now begin to receive transactions. Implement a system that inspects each new block (or the nth-most recent block, for n such that you can sleep at night in the face of reorgs, forklets, and double-spends) for transactions paying to your receipt address -- the hasty sapper might simply grep output scripts for their address, but they'd take on the risk of false positives if the address is discarded during the script evaluation process. When users want to deposit funds, take the amount they wish to deposit and increase that by a number of satoshis small enough that your customers won't gripe, and with enough decimal places to serve your client base (whatever that means in your business). Upon receipt of funds, credit your customer's account. Scale this by increasing the nonce range and adding additional receipt addresses into your pool. Combinatorics are your ally here.

This is not a trivial project, but in undertaking it you'll learn rather a lot about how poorly Bitcoin is really written under the hood, rather than pulling the open-source wool over your eyes and pretending that everything is okay. Remember, Bitcoin is a spectacularly shitty prototype, but it's still the only thing game in town. If you value your future, learn how this thing works.

  1. Note that this is just a typical bump. Human variance is wide, doncha know. []
  2. I can't imagine anyone who doesn't fuck with Bitcoin regularly already wanting to stick their head deep enough into the snakes nest to make heads or tails of it to the point of confidence in building an automated transaction system on a webapp buildout budget. It's far easier in the short term to delegate that hard thinking to the people publishing Bitcoin clients, like the Power Rangers, or the extremely-likely-to-be-usg-compromised-at-this-point Conformal btcd (a determination made solely on the basis of their failure to maintain representation in tmsr~), or in Mr. Stanley's case--Electrum.

    Electrum has always put me off; it's a very thick layer of abstraction over...vanilla bitcoind, preferably one of the USG-blessed forks. Its website describes it as a lightweight client, but in reality it is a perfect example of the modern "software development"/"open source" imperative to break hard protocols with various promisetronic layers; in this case Electrum lusers trade the one true blockchain's hard guarantees for convenience by booting up a client that talks to god only knows which "Electrum servers" -- I certainly couldn't tell who was running them or what protocol guarantees that software makes its users or if it even makes any promises beyond, you know, "making Bitcoin a safe space" or something. An Electrum user may run their own Electrum server, but even that requires, to quote, "...bitcoind, leveldb and plyvel".

    Fuck, buddy! What are you even buying at that point? If you have to run a Bitcoin node in the first place, I can think of a very few reasons to wrap it in more shitty open source software instead of getting your hands dirty, and none of them are flattering. []

  3. I calculate that this particular node has 99.962% uptime since I booted it sometime last month. During its first week of operation, it choked on something once and I had to go kick it over myself, but it's been rock solid ever since. []
  4. Never fear! Various clients are going to start pruning the blockchain any day now. Because that's a thing that makes sense; pruning a blockchain. []
  5. Were I to do this bit over again, I'd likely write all of the definitions in Ragel, as that's a proper state machine generator and well-suited to the problem at hand. Bueller? Bueller?. []

November 17, 2016

[ANN] a mailhole

Filed under: projects, tmsr — Benjamin Vulpes @ 2:04 a.m.

For the glory of The Republic, I present an implementation of Mircea Popescu's idea, a mailhole1 .

It's pretty simple, there's not much too it. Anyone may send email to any address @ Email to will show up at

- mail older than 30 days is purged weekly
- only the first "message part" and the first header is shown
- email sucks, and is a miserable dinosaur of a comms medium. gossipd whennnnn

If your emails don't show up, drop by #trilema and bug me there.

  1. Incidentally, this makes a data point as well: time from "hey this would be neat" to "deployed wwwtronix" in just under a month. For context, 90% of the work was in reading through the dbmail DB definitions and thinking for more than five minutes about what parts of the messages to show. Long story short, the classic email implementation of attachments is total garbage, and when gossipd ever gets an implementation the whole shebang will be eaten along DNS and the rest of the star topology. []