CH

February 1, 2015

Hacking on the Satoshi Codebase: Some Pointers

Filed under: Uncategorized — @ 12:00 a.m.
Hacking on the Satoshi Codebase: Some Pointers

Here's a script to download and compile an early-vintage Satoshi codebase. Originally written by mod6 and PinkPosixPXE with extensive refactoring by yours truly and some slick nixy improvements from the ever-useful jurov1.

#! /bin/sh

set -e
set -u

# ..::[ The Bitcoin Foundation ]::..
# Who: mod6 <modsix@gmail.com> 0x721705A8B71EADAF
# What: EXAMPLE SCRIPT DEMONSTRATING PATCHING AND COMPILING
# When: 20150130
# Version: UNVERSIONED - FOR REVIEW ONLY

# NOTE: If you don't have the dependencies already plugged in on
#       Debian 6 (squeeze), you'll want to run following commands:
#
# sudo apt-get update && sudo apt-get -y upgrade
# sudo apt-get install -y build-essential libssl-dev libdb4.8-dev \
# libdb4.8++-dev libboost-all-dev wget git curl gnupg

# Bitcoin Dev Home
# Script requires that you supply a location in which it should build

HOME=$1
pruned="bitcoin-v0_5_3_1/"
base="bitcoin-bitcoin-a8def6b/"
SHA256="/usr/bin/sha256sum"
mod6_fp="027A 8D7C 0FB8 A166 4372 0F40 7217 05A8 B71E ADAF"
v053="https://codeload.github.com/bitcoin/bitcoin/legacy.tar.gz/v0.5.3"
v053_hash="aab1f8ea8c7f131ff69dfa3b9437ba35531018be760132dd6373f41a591f6382"
chicken="http://thebitcoin.foundation/chicken.tar.gz"
chicken_hash="63164ea54f042226a6aa8768b98b0f323d66f08693c6fd271a850c00308517ad"
upnp="http://thebitcoin.foundation/rm_rf_upnp.tar.gz"
upnp_hash="f5f27b806b90b0ffd3d2d73e240eba02f13c5dec3d693b37e08686d04164861a"
https="http://thebitcoin.foundation/https-snipsnip.tar.gz"
https_hash="047844a601bc575d7660826b02f0d640f02ae84cd641f45552c825b72bc116f4"
alert="http://thebitcoin.foundation/turdmeister-alert-snip.tar.gz"
alert_hash="39b5aac2b6de016eda23a96fe6f9a565dbbe4f4aa46e8584c102ee71fa4fe7f4"
win32="http://thebitcoin.foundation/goodbye-win32.tar.gz"
win32_hash="5149a481629e0f10b659d38fee0736e0b6e631ab3bbeb6cde449aa61226cdbb3"
ppp_fp="3DAE 5385 2AF4 7432 CE3C 9318 FF23 EAA3 D815 427F"
dbconfig="http://thebitcoin.foundation/db_config.tar.gz"
dbconfig_hash="b315dc1494334d24606622b7bd3a2569b53e5ad970971ee666ae7adc28fc7d65"

verify_hash() {
    set -e
    local filename=$1
    local known_hash=$2
    local hash=$(sha256sum $filename | sed -n 's/^\(.*\) .*$/\1/p' | tr -d ' ')
    if [ "$known_hash" != "$hash" ];
    then
        echo "error! hash for download $1 failed to match known hash!"
        echo "expected: $known_hash"
        echo "got: $hash"
        exit 1
    fi
}

download_and_verify_hash() {
    local source=$1
    local filename=$2
    local known_hash=$3
    echo "downloading $source to $filename"
    curl $source -s -o $filename
    echo "verifying hash of $filename"
    verify_hash $filename $known_hash
}

verify_patch_signature() {
    local sig=$1
    gpg --logger-file "$sig.log" --homedir "$HOME/.gnupg" --verify "$sig"
    if [ "$?" != 0 ]; then
        echo "possible bad signature on $sig. see $sig.log for details."
        exit 1
    fi
}

# Change dir to $HOME
mkdir -p $HOME
chown -R $(whoami):$(whoami) $HOME
cd $HOME

# Get bitcoin-v0.5.3.tar.gz base code & verify checksum hash
download_and_verify_hash $v053 "bitcoin-v0.5.3.tar.gz" $v053_hash

# Unpack the Bitcoin v0.5.3 code:
tar -xf bitcoin-v0.5.3.tar.gz

# Download Patches and Verify Hashes
download_and_verify_hash $chicken "chicken.tar.gz" $chicken_hash
download_and_verify_hash $upnp "rm_rf_upnp.tar.gz" $upnp_hash
download_and_verify_hash $https "https-snipsnip.tar.gz" $https_hash
download_and_verify_hash $alert "turdmeister-alert-snip.tar.gz" $alert_hash
download_and_verify_hash $win32 "goodbye-win32.tar.gz" $win32_hash
download_and_verify_hash $dbconfig "db_config.tar.gz" $dbconfig_hash

# Unpack Patches:
mkdir -p chicken && tar -xf chicken.tar.gz -C chicken
tar -xf rm_rf_upnp.tar.gz
mkdir -p https-snipsnip && tar -xf https-snipsnip.tar.gz -C https-snipsnip
mkdir -p alert-snip && tar -xf turdmeister-alert-snip.tar.gz -C alert-snip
mkdir -p goodbye-win32 && tar -xf goodbye-win32.tar.gz -C goodbye-win32
tar -xf db_config.tar.gz

# Load PubKeys:
mkdir -p pubkeys
curl -s "http://pgp.mit.edu/pks/lookup?op=get&search=0x2AFA1A9FD2D031DA" \
| sed -n \
"/-----BEGIN PGP PUBLIC KEY BLOCK-----/,/-----END PGP PUBLIC KEY BLOCK-----/p" \
> pubkeys/ben_vulpes.pub.asc
curl -s "http://pgp.mit.edu/pks/lookup?op=get&search=0x721705A8B71EADAF" \
| sed -n \
"/-----BEGIN PGP PUBLIC KEY BLOCK-----/,/-----END PGP PUBLIC KEY BLOCK-----/p" \
> pubkeys/mod6.pub.asc
curl -s "http://pgp.mit.edu/pks/lookup?op=get&search=0xB98228A001ABFFC7" \
| sed -n \
"/-----BEGIN PGP PUBLIC KEY BLOCK-----/,/-----END PGP PUBLIC KEY BLOCK-----/p" \
> pubkeys/asciilifefrm.pub.asc

# Verify GPG PubKey import
gpg --homedir "$HOME/.gnupg" --version
gpg --logger-file gpgout.log --homedir "$HOME/.gnupg" --import pubkeys/*.asc
if [ "$?" != 0 ]; then
    echo "Did not import relevant keys correctly. Check gpgout.log for details."
    exit 1
fi

# Verify Patch Signatures:
verify_patch_signature chicken/bitcoin-asciilifeform.1.patch.sig
verify_patch_signature rm_rf_upnp/rm_rf_upnp.patch.sig
verify_patch_signature https-snipsnip/bitcoin-asciilifeform.2-https_snipsnip.patch.sig
verify_patch_signature alert-snip/bitcoin-asciilifeform.3-turdmeister-alert-snip.patch.sig
verify_patch_signature goodbye-win32/bitcoin-asciilifeform.4-goodbye-win32.patch.sig
verify_patch_signature db_config/bitcoin-v0_5_3-db_config.6.patch.sig

mkdir -p $HOME/$pruned
cd $base
for i in $(cut -d ' ' -f 3 $HOME/chicken/bitcoin-0.5.3-no-crud.sha256.manifest); do
    cp --parents "$i" $HOME/$pruned;
done
cd $HOME

count1=$(find $pruned -xtype f -print0 | xargs -0 sha256sum | wc -l)
count2=$(cat chicken/bitcoin-0.5.3-no-crud.sha256.manifest | wc -l)

if [ "$count1" != "$count2" ]; then
  echo "Error! The resulting file count after pruning did not match\n
        the file count from the manifest.";
  echo "post-pruning: $count1";
  echo "manifest:     $count2";
  exit 1;
fi

# Patch Bitcoin v0.5.3 Source Base & Clean Up:
cd $pruned

patch -p1 < ../chicken/bitcoin-asciilifeform.1.patch
patch -p1 < ../rm_rf_upnp/rm_rf_upnp.patch
patch -p1 < ../https-snipsnip/bitcoin-asciilifeform.2-https_snipsnip.patch
patch -p1 < ../alert-snip/bitcoin-asciilifeform.3-turdmeister-alert-snip.patch
patch -p1 < ../goodbye-win32/bitcoin-asciilifeform.4-goodbye-win32.patch
patch -p1 < ../db_config/bitcoin-v0_5_3-db_config.6.patch

# [ CLEAN-UP ]:
cd ..
rm -rf {alert-snip*,chicken*,db_config*,goodbye-*,https-snipsnip*,pubkeys,rm_rf_upnp,turdmeister-alert-snip*,https_snipsnip.sig.log,rm_rf_upnp.*,bitcoin-v0.5.3.tar.gz,prune_manifest.sh,.gnupg}

# Compile Patched Source:
cd $pruned/src
make -f makefile.unix
echo "Done!"
cd ..
### EOF ###

Patches, as always, welcome.

At this point, I'm sure that you're curious as to what the actual dev loop looks like. Allow me to illuminate:

$ scp make_five.sh btc-deb:/my_bitcoin_dir/make_five.sh && ssh btc-db
$ rm bitcoin_dir/test_build && ./make_five.sh /absolute_path_to_bitcoin/test_dir

Which kicks off the downloading of files and verifying of patches2.

Post-the build, all sorts of shit goes sidways with the 0.5.3(.1)3. 0.5.3.* attempts to track every single "orphan block"4, trips over its own feet and gets shot in the head by the kernel regularly. To address this, and run something that might have the proverbial chance in hell of catching up to the full chain, I've cooked up the following frankenstein of a script-assemblage:

In ~/TESTINGDIR/builds:

#! /bin/sh

set -u

BITCOIND_ROOT=$1
LISTEN=$2
RPCPORT=$3

$BITCOIND_ROOT/bitcoind -datadir=$BITCOIND_ROOT -debug -port=$LISTEN -rpcport=$RPCPORT stop

closed=1
until [ "$closed" -eq 0 ]; do
    tail $BITCOIND_ROOT/debug.log | grep "Bitcoin exiting"
    if [ "$?" -eq 0 ]; then
        $BITCOIND_ROOT/bitcoind -datadir=$BITCOIND_ROOT -port=$LISTEN -rpcport=$RPCPORT -daemon -debug &
        closed=0
    fi
done

This script expects 3 arguments5:

  • directory where the compiled bitcoind lives and the data directory is expected to be
  • port on which to listen for connections
  • port on which to listen for JSON/RPC calls

Then, a simple cron file:

0 */2 * * * /path/to/reboot.sh /path/to/bitcoind/dir
0 */2 * * * /path/to/reboot.sh /path/to/other/bitcoind/dir

Those bitcoind dirs will of course need a bitcoin.conf with rpcuser=foo and rpcpassword=bar entries to keep your dumb ass safe from yourself6.

Then, assuming that I haven't completely borked the above scripts and instructions, you should be able to build, boot and run a bitcoind. If you're savvy enough to read through this stuff, you should be aware that the above instructions are the equivalent of a hand-fabricated mortar apparatus - entirely likely to blow your hands off.

Footnotes:

1

Hey, wasn't someone barking about commissioning an actual website for the new #bitcoin-assets WoT? Where is that thing, anyways?

2

The pruning (process of turning whatever's being served from Github into a source tree whose provenance we can vouch for) in particular's caused mod6 and myself no end of headaches. Eventually (which is to say after the 3rd time doing it by hand) we automated all of the checks involved and all of the cleanup to boot. Not that it couldn't benefit from a second, third…fifth, set of eyes reading through it.

3

Don't tell anyone we've started cutting our own version numbers yet, okay?

4

A misnomer discussed at length in the forum of La Serenissima, as they're not properly speaking part of an orphan chain. Properly speaking, they're "bastard blocks" - blocks for which the downloading node doesn't know a proper ancestry. At a certain point, the data structure responsible for tracking these blocks exceeds what's available on the server in question and the whole "bitcoind" process is unceremoniously killed. The upstream "solution" (if a boy can be forgiven the charity of calling it a solution) is to set a threshold (750, another magical number hard-coded in the Satoshi tree) and prune the set of "bastard blocks" whenever it approaches that size.

Working on this codebase has really illuminated (both for myself and the rest of La Serenissima) the myriad of ways in which the Satoshi codebase is entirely unfit for production use: hard-coded DNS seeds that will indubitably leveraged by the enemies of hard crypto to degrade the network quality; a block download process I charitably describe as "naive"; and a wallet paradigm that's guaranteed to lose user funds in the long term.

The best that can be said about the Satoshi codebase is that it was and always will be a mere prototype. That those to whom Satoshi handed the baton went haring off in the directions they did (colored coins, hierarchical deterministic wallets, vain attempts to embed perpetual inflation in the ever critical scarcity model [yes, Gavin Andresen, I'm looking at you and your doomed inflationary block proposal]) instead of taking a good hard look at the many ways in which the RI has always been wholly inadequate to the demands placed upon it speaks volumes about their inability to reason about the world in which they found themselves.

The failure of the open source model as applied to Bitcoin deserves a bit of a callout here. Andresen only ever pulled paychecks to support the image of Vessene's foundation, which was itself only ever a route for Vessenes to embezzle more coins from lucky early adopters who wanted desperately to matter (well in excess of their desire and ability to defend their…riches). Software may be "open sourced" by various companies who depend on that software to do their business (Puppet, Chef and Go are all excellent examples of the pattern: monolithic software projects of such scale that they either approach or actually are their own languages for which the source is freely available, but the source for which nobody is really supposed to read), but in nearly all of those cases the code may as well be a closed hairball for all the interest the entities in question have in documenting and maintaining the various hairballs (I have friends at Puppet, and I'm sure they can take a joke). Training always costs an arm and a leg (take a look at the companies pushing Clojure for more examples of people creating languages around which they can build consulting practices), and certifications inevitably grow up around the system as the various companies seek to extract every dollar from the chumpatron they find themselves manning.

The "Bitcoin Foundation" poured an unforgiveable number of coins into the market to feed Gavin Andresen and his family…and for what? 3 years of development that's just about to get flushed down the drain? The appearance of relevance without any of the hard work of thinking and making things work correctly? Just because a herd of C++ "developers" can bend the Berkely Database to their will with critical sections and other corrosive mindrot does not make their work in any sense a "contribution" to the "bitcoin economy".

Consider the tragedy wrought by an internet forum accidentally blessed with untold riches and weep.

5

The last two are sort-of (as in the first can be done away with by connecting the nodes under test to a known-good-node and not letting them listen) necessary in order to prevent the bitcoind's from listening on the same ports. Defaults, yo. Always a fun problem.

6

Aren't you glad that Gavin's on the case? He's really helping those noobs not shoot themselves in the foot with insecure password policies.

---