CH

June 21, 2015

a tour of bitcoind booting to its first thread

Filed under: Uncategorized — Benjamin Vulpes @ 12:00 a.m.
a tour of bitcoind booting to its first thread

I am a very poor C programmer. I barely understand how compilers work (although I'm familiar with the philosophical implications of the technology in a security-hostile world, a la Trusting Trust), and I've only today managed to bully bitcoind into running under a debugger (GUD in Emacs, of course!). In this post I'll document what I did (trivial) and what it looks like, why I did it (inane), and what I got out of the whole process.

why

Let's be frank: I have no idea what I'm doing when it comes to computers. I'm the kind of guy who just has to piss on each and every fence nearby to find out which ones are electrified, at what frequency, how many volts they carry and how many amps they want to dissipate through my poor dong. $DAYJOB covers the standard backend stacks (Python, PHP, Ruby, Clojure, misc. Linux), and also has a huge dependency on the Apple development toolchain. As a direct result of most of the work on the table for the shop coming in on the "mobile" vector, I use an Apple computer as my "daily driver".

This has lead to no end of headaches when hacking on the bitcoind source. I've built bitcoinds in Docker containers and shipped them off to virtualized servers in The Cloud, I've built them in virtual machines local to my own development machine and run them in the selfsame VMs, and I even burned a few hours today attempting to compile bitcoin natively on my Mac (hey, someone said it was possible!).

Today I moved my entire development flow for hacking on bitcoind into an Ubuntu 14.04 virtual machine. This is just a stopgap until I get the time to build a fresh Linux tower with which I'll undertake Gentoo Quest. Again. For real this time, I swear.

While in the process of doing all of this, I wrote a new script to ease the procurement of dependencies and coordinating of state to get all of the bits into the right buckets at the right time:

#!/bin/bash
set -xeu

# requisite folders:
mkdir -p distfiles
mkdir -p ourlibs/lib
mkdir -p ourlibs/include
mkdir -p ourlibs/include/openssl

# dependencies:
which realpath
which curl
which g++

# globals:
BOOST_ADDRESS_MODEL=64
BOOST_ARCH_TYPE=x86_64
SSL_ARCH_TYPE=linux-x86_64
OPENSSL=openssl-1.0.1g
BDB=db-4.8.30
BOOST=boost_1_52_0
DIST=./distfiles
OURLIBS=$(realpath ./ourlibs)

# hashes and download locations:
SSL_LOC="http://openssl.org/source/old/1.0.1/openssl-1.0.1g.tar.gz"
SSL_HASH="53cb818c3b90e507a8348f4f5eaedb05d8bfe5358aabb508b7263cc670c3e028"
BDB_LOC="http://download.oracle.com/berkeley-db/db-4.8.30.tar.gz"
BDB_HASH="e0491a07cdb21fb9aa82773bbbedaeb7639cbd0e7f96147ab46141e0045db72a"
BOOST_LOC="http://sourceforge.net/projects/boost/files/boost/1.52.0/boost_1_52_0.tar.gz"
BOOST_HASH="eea637a4ce9f9b45a0d5e00bb9462c9f084086264d85d63133dd6d240398b28f"

# compiler flags:
CC=gcc
CXX=g++
LD=ld
CFLAGS=-I/usr/include
LDFLAGS=-L/usr/lib
PATH=$PATH:/usr/bin

# paths
export BOOST_INCLUDE_PATH=$OURLIBS/include
export BDB_INCLUDE_PATH=$OURLIBS/include
export OPENSSL_INCLUDE_PATH=$OURLIBS/include
export BOOST_LIB_PATH=$OURLIBS/lib
export BDB_LIB_PATH=$OURLIBS/lib
export OPENSSL_LIB_PATH=$OURLIBS/lib

check_hash() {
    local stored=$1
    local file=$2
    local computed=$(sha256sum $file | awk '{print $1}')
    if [[ "$stored" = "$computed" ]]; then
        return 0
    else
        return 1
    fi
}

memoized_download() {
    local remote_location=$1
    local download_target=$2
    local stored_hash=$3
    if check_hash $stored_hash $download_target; then
        return 0
    else
        curl -L $remote_location -o $download_target
        check_hash $stored_hash $download_target
    fi
}

expand() {
    local file=$1
    tar xfvz $1
}

compile_openssl() {
    pushd $OPENSSL
    make clean
    ./Configure --prefix=$OURLIBS $SSL_ARCH_TYPE \
        no-dso no-shared
    make
    make install_sw
    cp libssl.* $OURLIBS/lib/
    cp libcrypto.* $OURLIBS/lib/
    mkdir -p $OURLIBS/include/openssl
    cp include/openssl/*.h $OURLIBS/include/openssl/
    popd
}

compile_bdb() {
    pushd $BDB/build_unix
  ../dist/configure --enable-cxx --prefix=$OURLIBS
    make clean
    make
    make install
    popd
}

compile_boost() {
    pushd $BOOST
    echo "using gcc : $BOOST_ARCH_TYPE : $CXX ;" > tools/build/v2/user-config.jam
    ./bootstrap.sh
    ./bjam --clean
    ./bjam --build-type=minimal toolset=gcc address-model=$BOOST_ADDRESS_MODEL \
        link=static -sNO_BZIP2=1 -sNO_ZLIB=1 -sNO_COMPRESSION=1
    sudo ./bjam toolset=gcc address-model=$BOOST_ADDRESS_MODEL \
        link=static --prefix=$OURLIBS install
    popd
}

compile_bitcoin() {
    pushd bitcoin/src
    make -f makefile.unix clean
    make LMODE=static LMODE2=static -f makefile.unix bitcoind
}

memoized_download $SSL_LOC $DIST/$OPENSSL.tar.gz $SSL_HASH
expand $DIST/$OPENSSL.tar.gz
compile_openssl
memoized_download $BDB_LOC $DIST/$BDB.tar.gz $BDB_HASH
expand $DIST/$BDB.tar.gz
compile_bdb
memoized_download "$BOOST_LOC" "$DIST/$BOOST.tar.gz" "$BOOST_HASH"
expand "$DIST/$BOOST.tar.gz"
compile_boost
compile_bitcoin
echo "done!"

This is a Bash script, not a sh script as distributed in the Foundation's tarball. This is so that I can leverage "crash only semantics" (to borrow a Yarvinism). Unset variables cause the script to fail, and function return values other than 0 cause the script to fail. This is convenient: when the script does fail (and fail it will, for the bedrock is made of pressed human excrement), the operator must simply scroll backwards through his or her terminal output until the line marked with a "+" (telltale sign of a Bash script containing the "-x" flag printing each line before executing it) and debug from there.

Extremely useful this, as you'll not see the happy "done!" printout unless each and every single intervening step succeeded as well.

what (it looks like)

First,

M-x gdb <RET>

at the prompt (which will read "Run gdb (like this):"), enter:

#+begin
gdb -imi=mi –args $PATH_TO_BITCOIND_ELF -printtoconsole -debug
#+end_src

To kick things off, setting automatic breakpoint at the main function, type "start" into the debugger. This will bring up the source with the appropriate breakpoint next to the debugger interface:

gud_simple

You can then:

M-x gdb-many-windows

which brings up the following (perhaps overwhelming) interface:

btc_gud

Starting in the upper left corner and walking clockwise, we have:

  • the GUD CLI
  • the locals/registers buffer, where local variables and registers are available for inspection
  • a buffer displaying program input/output
  • a buffer displaying threads and breakpoints
  • a buffer displaying the call stack
  • the source buffer

All of this and more wonders can be found in the manual and "RMS's gdb tutorial".

a guided tour of bitcoind booting up

Our journey begins in init.cpp with the application's main function:

int main(int argc, char* argv[])
{
    bool fRet = false;
    fRet = AppInit(argc, argv);

    if (fRet && fDaemon)
        return 0;

    return 1;
}

main calls AppInit, which itself calls AppInit2 catching all exceptions in two seperate logic branches, shuting itself down if the second layer of AppInit doesn't fire correctly:

bool AppInit(int argc, char* argv[])
{
    bool fRet = false;
    try
    {
        fRet = AppInit2(argc, argv);
    }
    catch (std::exception& e) {
        PrintException(&e, "AppInit()");
    } catch (...) {
        PrintException(NULL, "AppInit()");
    }
    if (!fRet)
        Shutdown(NULL);
    return fRet;
}

AppInit2 is where all of the initialization magic happens. This is a beast, so let's take it one step at a time:

bool AppInit2(int argc, char* argv[])
{
    // lotsa shit happens in here
}

Due to mysterious BDB shenanigans on application shutdown, bitcoind needs to catch the termination, interrupt and hangup signals and clean its pants up nicely before going home to mummy:

// Clean shutdown on SIGTERM
struct sigaction sa;
sa.sa_handler = HandleSIGTERM;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGTERM, &sa, NULL);
sigaction(SIGINT, &sa, NULL);
sigaction(SIGHUP, &sa, NULL);

An aside here: I can't find the specific location in the logs (and believe you me: I tried for at least 5(!) minutes), but at one point someone involved in this butchery suggested replacing the JSON-RPC system by which bitcoind communicates with its operator with a set of signal responses. This was (IIRC) dubbed crazy or hacky almost immediately, and tabled anyways as an exceptionally low-priority item.

From there, bitcoind progresses to calling

ParseParameters(argc, argv);

Which looks like (util.cpp):

void ParseParameters(int argc, char* argv[])
{
    mapArgs.clear();
    mapMultiArgs.clear();
    for (int i = 1; i < argc; i++)
    {
        char psz[10000];
        strlcpy(psz, argv[i], sizeof(psz));
        char* pszValue = (char*)"";
        if (strchr(psz, '='))
        {
            pszValue = strchr(psz, '=');
            *pszValue++ = '\0';
        }
        if (psz[0] != '-')
            break;
        mapArgs[psz] = pszValue;
        mapMultiArgs[psz].push_back(pszValue);
    }
}

mapArgs is the global holding pen for the arguments with which bitcoind was booted. You may inspect its various call sites at jurov's lxr instance. It is used as a classic dictionary/map/lookup table, giving the RPC server a handle to the RPC username and password; as a truthy boolean lookup to indicate whether an option was passed in on the command line for downstream code; as the test in a ternary operator1, and to trigger alets to the user about the insecurity of their JSON-RPC setup2 to highlight the largest classes of use. It's not that bad in the grand scheme of things, but it is also this giant blob of state that just sits there in the program waiting for SoftSetArg to poke it (util.cpp):

bool SoftSetArg(const std::string& strArg, const std::string& strValue)
{
    if (mapArgs.count(strArg))
        return false;
    mapArgs[strArg] = strValue;
    return true;
}

SoftSetArg being the thing that's responsible for setting all sorts of default behavior if the user doesn't bother with it themselves.

On top of all of this, there's also the mapMultiArgs data structure that holds (to the best that I can tell) command-line arguments for which the user could provide multiple values. Why this has to have a seperate data structure I don't think I'll ever know.

Immediately after ParseParameters, we encounter some 13 lines for handling the case when the datadir does not exist (util.cpp):

if (mapArgs.count("-datadir"))
{
    if (filesystem::is_directory(filesystem::system_complete(mapArgs["-datadir"])))
    {
        filesystem::path pathDataDir = filesystem::system_complete(mapArgs["-datadir"]);
        strlcpy(pszSetDataDir, pathDataDir.string().c_str(), sizeof(pszSetDataDir));
    }
    else
    {
        fprintf(stderr, "Error: Specified directory does not exist\n");
        Shutdown(NULL);
    }
}

Even though the application's default behavior is to create the ~/.bitcoind directory itself unprompted (and yet not the bitcoin.conf file that the damn thing expects in there), if you pass it a data directory that doesn't exist, it refuses to create it.

Schizophrenic damn design, this. "We'll create the default directory for you if it doesn't exist, we'll spit a bullshit username/password combo for your RPC server to the console that you have to manually derp into a file at $datadir/bitcoin.conf, but if you pass a $datadir that doesn't exist let's just shit our pants and sit on the floor like a feral child confronted with a television showing tits." Bitcoind as written would be the best possible example of condescending software design if it were worthy of being called "designed" instead of "crufted together by a monotonically increasingly febrile group of contributors".

Then we move into ReadConfigFile (init.cpp):

void ReadConfigFile(map<string, string>& mapSettingsRet,
                    map<string, vector<string> >& mapMultiSettingsRet)
{
    namespace fs = boost::filesystem;
    namespace pod = boost::program_options::detail;

    fs::ifstream streamConfig(GetConfigFile());
    if (!streamConfig.good())
        return;

    set<string> setOptions;
    setOptions.insert("*");

    for (pod::config_file_iterator it(streamConfig, setOptions), end; it != end; ++it)
    {
        // Don't overwrite existing settings so command line settings override bitcoin.conf
        string strKey = string("-") + it->string_key;
        if (mapSettingsRet.count(strKey) == 0)
            mapSettingsRet[strKey] = it->value[0];
        mapMultiSettingsRet[strKey].push_back(it->value[0]);
    }
}

The only thing worth noting here is the complixty cost (witness the backflips required to prioritize command line arguments over config file arguments) of tracking configuration in multiple places and having a prioritization scheme for it.

if (mapArgs.count("-?") || mapArgs.count("--help"))

I want to know who does this:

./proggy -?

…to get documentation. If you find this person, please send them to me.

Omitting some things, this is where bitcoind forks itself off if you pass the "-daemon" flag (init.cpp):

fdaemon = GetBoolArg("-daemon");
// omitted
pid_t pid = fork();
if (pid < 0)
{
    fprintf(stderr, "Error: fork() returned %d errno %d\n", pid, errno);
    return false;
}
if (pid > 0)
{
    CreatePidFile(GetPidFile(), pid);
    return true;
}
pid_t sid = setsid();
if (sid < 0)
    fprintf(stderr, "Error: setsid() returned %d errno %d\n", sid, errno);

Our first print statements! This is what it looks like in GUD:

gud_io.png

Ensure that only a single process gets a hold of the bitcoind data directory (init.cpp):

string strLockFile = GetDataDir() + "/.lock";
FILE* file = fopen(strLockFile.c_str(), "a"); // empty lock file; created if it doesn't exist.
if (file) fclose(file);
static boost::interprocess::file_lock lock(strLockFile.c_str());
if (!lock.try_lock())
{
    wxMessageBox(strprintf(_("Cannot obtain a lock on data directory %s.  Bitcoin is probably already running."), GetDataDir().c_str()), "Bitcoin");
    return false;
}

Moving along, we have the function call in a test used to load addresses (init.cpp):

if (!LoadAddresses())
    strErrors += _("Error loading addr.dat      \n");

The implementation of which is:

bool LoadAddresses()
{
    return CAddrDB("cr+").LoadAddresses();
}

Calling…

class CAddrDB : public CDB
{
public:
    CAddrDB(const char* pszMode="r+") : CDB("addr.dat", pszMode) { }
private:
    CAddrDB(const CAddrDB&);
    void operator=(const CAddrDB&);
public:
    bool WriteAddress(const CAddress& addr);
    bool EraseAddress(const CAddress& addr);
    bool LoadAddresses();
};

(db.cpp)

bool CAddrDB::LoadAddresses()
{
    CRITICAL_BLOCK(cs_mapAddresses)
    {
        // Get cursor
        Dbc* pcursor = GetCursor();
        if (!pcursor)
            return false;

        loop
        {
            // Read next record
            CDataStream ssKey;
            CDataStream ssValue;
            int ret = ReadAtCursor(pcursor, ssKey, ssValue);
            if (ret == DB_NOTFOUND)
                break;
            else if (ret != 0)
                return false;

            // Unserialize
            string strType;
            ssKey >> strType;
            if (strType == "addr")
            {
                CAddress addr;
                ssValue >> addr;
                mapAddresses.insert(make_pair(addr.GetKey(), addr));
            }
        }
        pcursor->close();

        printf("Loaded %d addresses\n", mapAddresses.size());
    }

    return true;
}

Our first CRITICAL_BLOCK! This is the C++ mutex used throughout the bitcoind codebase to ensure that only one thread executes a given section of code at any point in time.

In any event, let us onwards to our next interesting stop on this marvelous train of C++: the first thread!

It begins here (init.cpp):

int nLoadWalletRet = pwalletMain->LoadWallet(fFirstRun);

Which calls pwalletMain->LoadWallet (wallet.cpp):

int CWallet::LoadWallet(bool& fFirstRunRet)
{
    if (!fFileBacked)
        return false;
    fFirstRunRet = false;
    int nLoadWalletRet = CWalletDB(strWalletFile,"cr+").LoadWallet(this);
    if (nLoadWalletRet == DB_NEED_REWRITE)
    {
        if (CDB::Rewrite(strWalletFile, "\x04pool"))
        {
            setKeyPool.clear();
            // Note: can't top-up keypool here, because wallet is locked.
            // User will be prompted to unlock wallet the next operation
            // the requires a new key.
        }
        nLoadWalletRet = DB_NEED_REWRITE;
    }

    if (nLoadWalletRet != DB_LOAD_OK)
        return nLoadWalletRet;
    fFirstRunRet = vchDefaultKey.empty();

    if (!HaveKey(Hash160(vchDefaultKey)))
    {
        // Create new keyUser and set as default key
        RandAddSeedPerfmon();

        std::vector<unsigned char> newDefaultKey;
        if (!GetKeyFromPool(newDefaultKey, false))
            return DB_LOAD_FAIL;
        SetDefaultKey(newDefaultKey);
        if (!SetAddressBookName(CBitcoinAddress(vchDefaultKey), ""))
            return DB_LOAD_FAIL;
    }

    CreateThread(ThreadFlushWalletDB, &strWalletFile);
    return DB_LOAD_OK;
}

Which looks for various error conditions (such as the wallet being locked, bad DB statues), creates a new key if none exist in the DB, sets the "address book name" (braindamage alert!), and then kicks off the ThreadFlushWalletDB function on a seperate thread (db.cpp):

void ThreadFlushWalletDB(void* parg)
{
    const string& strFile =1
        return;

    unsigned int nLastSeen = nWalletDBUpdated;
    unsigned int nLastFlushed = nWalletDBUpdated;
    int64 nLastWalletUpdate = GetTime();
    while (!fShutdown)
    {
        Sleep(500);

        if (nLastSeen != nWalletDBUpdated)
        {
            nLastSeen = nWalletDBUpdated;
            nLastWalletUpdate = GetTime();
        }

        if (nLastFlushed != nWalletDBUpdated && GetTime() - nLastWalletUpdate >= 2)
        {
            TRY_CRITICAL_BLOCK(cs_db)
            {
                // Don't do this if any databases are in use
                int nRefCount = 0;
                map<string, int>::iterator mi = mapFileUseCount.begin();
                while (mi != mapFileUseCount.end())
                {
                    nRefCount += (*mi).second;
                    mi++;
                }

                if (nRefCount == 0 && !fShutdown)
                {
                    map<string, int>::iterator mi = mapFileUseCount.find(strFile);
                    if (mi != mapFileUseCount.end())
                    {
                        printf("%s ", DateTimeStrFormat("%x %H:%M:%S", GetTime()).c_str());
                        printf("Flushing wallet.dat\n");
                        nLastFlushed = nWalletDBUpdated;
                        int64 nStart = GetTimeMillis();

                        // Flush wallet.dat so it's self contained
                        CloseDb(strFile);
                        dbenv.txn_checkpoint(0, 0, 0);
                        dbenv.lsn_reset(strFile.c_str(), 0);

                        mapFileUseCount.erase(mi++);
                        printf("Flushed wallet.dat %"PRI64d"ms\n", GetTimeMillis() - nStart);
                    }
                }
            }
        }
    }
}

In which:

  1. bitcoind checks the fOneThread global, returning if its true (which appears to be used in precisely three places in the entire codebase, all in these three lines right here. TODO for intrepid muntzers: determine if this is safe to remove)
  2. bitcoind checks the global statewad of arguments for the "-noflushwallet" option, returning if it's true
  3. Goes into a while loop, sleeping for half a second until the global "fShutDown" variable kicks over into the "true" state
  4. In which it tracks the the time when it last went through the iterator and the time at which the wallet was last updated and the time of the last flush; waiting for the wallet to be updated without flushing and for the time since the last flush to exceed two seconds…
  5. At which point it prints a datetime, updates its time-counters internally, closes the database, and wraps up all of the Log Sequence Numbers currently in use

All of which strikes me as ridiculously ham-handed and unnecessarily complex given the goals on the tin and neglecting the…odd technical decisions driving all of this bookkeeping code. The damn cult of complexity, rearing its head to bury clean code (not that this project ever had any, I'm told) under a mountain of shit mostly only serving to inflate the egos of the individuals involved3.

Anyways, that wraps tonights tour of bitcoin to its first thread. Enjoy!

Footnotes:

1

As another aside, in reading the source for this thing (again) I suspect that it opens itself up to inbound connections from any addr, although I'd like confirmation of this.

2

A bit of code indicative of the disdain with which the "dev team" regards the consumers of their work. "Oh, you forgot to set a username and password for the gaping security hole right in the middle of your military-grade cryptocurrency installation. We'll keep you from shooting yourself in the foot in that particular way, while still encouraging you to…run bitcoind nodes on servers attached to the internet." The attitude goes hand-in-hand with the "make bitcoin accessible to all and sundry" attitude the socialists currently blessed by the Eye of Sauron parade around in public to legitimize themselves. It can't work, and it won't work, because people so stupid as to need their hands held when running their bitcoind nodes at the outset are going to be so stupid that they're going to make other operational mistakes that you'll be unable to prevent. The whole charade is akin to handing an untrained child a .45, trigger-guard wrapped in tape and the safety mysteriously missing.

"Oh, but SSL will protect your credentials in transit…" No, it most certainly will not—remember "Heartbleed"?. In all likelihood, using SSL anywhere near your bitcoind instance is going to leak private keys.

3

Or to satisfy their handlers' desires that strong crypto libs be utterly unreadable and unmaintainable.

  1. const string*)parg)[0]; static bool fOneThread; if (fOneThread) return; fOneThread = true; if (mapArgs.count("-noflushwallet" []

No Comments »

No comments yet.

RSS feed for comments on this post. TrackBack URL

Reply

« Arrival --- How to (actually) "learn programming" »