November 18, 2016

How to (Actually) Process Bitcoin Transactions

Filed under: bitcoin — Benjamin Vulpes @ 8:29 a.m.

James Stanley developed opinions about how to handle bitcoins while building a nifty, supposedly anonymous, Bitcoin/telephony bridge. Unfortunately, he labors outside the walls of the Most Serene Republic of Bitcoin, and so he's missing the typical 30-point bump in IQ that can be had for simply reading the logs1. As they say, "build one to throw away"--and it's never truer than if you don't know what you're building in the first place2.

My first treatment apprently didn't expound on the subject in adequate depth, and so here I am again: correcting the internet. This guide is less "do these three simple things" and more "do these two impossible things". What, you wanted to learn something, didn't you?

- you (can) run a full TRB node
- you can extract transaction data from raw blocks

Running a full TRB node is table stakes for this game. It means getting up to speed with "V", The Republic's software distribution tool, and running the daemon you'll compile with mod6's offline build system on a semi-serious piece of metal. The most-public node that I run consumes over 5.7 gigs of RAM, and its current uptime is ~48 days3. Use a process supervisor (I'll leave writing up the details to trinque, but start with runit, and its accompanying log service svlogd. A TRB node will shit several gigs of log lines in a month, so do consider accounting for the disk consumption.

Once your node runs reliably, you need to sync it. If you don't already have a synced node, there's no way around simply downloading the whole blockchain and verifying it sequentially (pity the poor Ethereum bagholders, whose blockchain is over 75 (!) gigs at this point4. ). If you're a lucky sod, and are running a previously-synced 0.5.X node, you can simply copy the bitcoin data directory to your new box and start the new bitcoind process over there. Do remember to stop the synced node first, lest you bring down the data-integrity gods' wrath upon your head. Given the times involved, if you have a 0.6.X, and possibly a 0.7.X series node running and synced, consider pulling the same trick. I have no idea if it'll work, but I am eager for reports on that kind of science.

With node in hand, you may now proceed to dissect blocks (if you've pressed a standard TRB, it has a 'dumpblock' command. Go forth and figure out how to use it). I've already written up how to extract block headers, and while I failed to beat the "binary-types" library into cleanly parsing the binary blocks (see "binary-types:define-binary-class bitcoin-block-header" for what a clean binary definition looks like in that DSL), I turned around and implemented all of the slicing code behind mimisbrunnr in just a few days once I decided to do it by hand and forgo the bijective transform5. This code is not publically available, but if you show up in #trilema, get in the WoT and ask nicely, I might give you a copy.

Given the ability to slice blocks into headers and transactions, you may now begin to receive transactions. Implement a system that inspects each new block (or the nth-most recent block, for n such that you can sleep at night in the face of reorgs, forklets, and double-spends) for transactions paying to your receipt address -- the hasty sapper might simply grep output scripts for their address, but they'd take on the risk of false positives if the address is discarded during the script evaluation process. When users want to deposit funds, take the amount they wish to deposit and increase that by a number of satoshis small enough that your customers won't gripe, and with enough decimal places to serve your client base (whatever that means in your business). Upon receipt of funds, credit your customer's account. Scale this by increasing the nonce range and adding additional receipt addresses into your pool. Combinatorics are your ally here.

This is not a trivial project, but in undertaking it you'll learn rather a lot about how poorly Bitcoin is really written under the hood, rather than pulling the open-source wool over your eyes and pretending that everything is okay. Remember, Bitcoin is a spectacularly shitty prototype, but it's still the only thing game in town. If you value your future, learn how this thing works.

  1. Note that this is just a typical bump. Human variance is wide, doncha know. []
  2. I can't imagine anyone who doesn't fuck with Bitcoin regularly already wanting to stick their head deep enough into the snakes nest to make heads or tails of it to the point of confidence in building an automated transaction system on a webapp buildout budget. It's far easier in the short term to delegate that hard thinking to the people publishing Bitcoin clients, like the Power Rangers, or the extremely-likely-to-be-usg-compromised-at-this-point Conformal btcd (a determination made solely on the basis of their failure to maintain representation in tmsr~), or in Mr. Stanley's case--Electrum.

    Electrum has always put me off; it's a very thick layer of abstraction over...vanilla bitcoind, preferably one of the USG-blessed forks. Its website describes it as a lightweight client, but in reality it is a perfect example of the modern "software development"/"open source" imperative to break hard protocols with various promisetronic layers; in this case Electrum lusers trade the one true blockchain's hard guarantees for convenience by booting up a client that talks to god only knows which "Electrum servers" -- I certainly couldn't tell who was running them or what protocol guarantees that software makes its users or if it even makes any promises beyond, you know, "making Bitcoin a safe space" or something. An Electrum user may run their own Electrum server, but even that requires, to quote, "...bitcoind, leveldb and plyvel".

    Fuck, buddy! What are you even buying at that point? If you have to run a Bitcoin node in the first place, I can think of a very few reasons to wrap it in more shitty open source software instead of getting your hands dirty, and none of them are flattering. []

  3. I calculate that this particular node has 99.962% uptime since I booted it sometime last month. During its first week of operation, it choked on something once and I had to go kick it over myself, but it's been rock solid ever since. []
  4. Never fear! Various clients are going to start pruning the blockchain any day now. Because that's a thing that makes sense; pruning a blockchain. []
  5. Were I to do this bit over again, I'd likely write all of the definitions in Ragel, as that's a proper state machine generator and well-suited to the problem at hand. Bueller? Bueller?. []


  1. Mircea Popescu says:

    That's ok, nobody involved with Ethereum is interested in things such as chain verification etc. They just want to revolutionize things etc.

  2. Leclerc says:

    Excellent post, especially because we know you have actually processed a lot of transactions and also because bitcoin addresses are limited 1 per person. Thanks!!

  3. Leclerc says:

    It is so nice to see the subject rebuffed with actual arguments. You must be so proud!

  4. Mircea Popescu says:

    You can not bring actual arguments until and unless you are somebody. So who are you ?

  5. Benjamin Vulpes says:


    When you misquote, misrepresent, or fail to read what I write, noblesse oblige does not extend to pretending that your shannonizer is working.


    we know you have actually processed a lot of transactions

    The methodology stands independently of its developer.

    because bitcoin addresses are limited 1 per person

    You must have missed:

    Scale this by increasing the nonce range and adding additional receipt addresses into your pool. Combinatorics are your ally here.

    Which is fine, I certainly don't expect nobodies to be able to add, much less read.

    Remind me, "Leclerc", what was the subject again? That I build functional systems and write complete specifications that you can't read, right?


RSS feed for comments on this post. TrackBack URL


« How to (Actually) Process Bitcoin Transactions --- A Pox Upon Your House! »