CH

August 29, 2014

so you want to coordinate an EC2 instance and an RDS instance in a VPC from scratch

Filed under: Uncategorized — @ 12:00 a.m.
so you want to coordinate an EC2 instance and an RDS instance in a VPC from scratch

Well, don't. But if you have to, you've come to the right place, as I just spent the past week automating cluster provisioning:

Some Philosophical Considerations

I endeavor to write as much of my provisioning in Bash as possible, so that I can run it with a minimum of modification1 wherever else it pleases me to do so. Working with AWS, though, pretty much demands that I2 hack out the AWS-related parts of these provisioners in Python3.

All this setup for: "My provisioner doesn't shell out. I wrote it in Bash, and it sells out to Python".

You're welcome. It hurt to write too.

Geting Everything in the Right Place, at Just the Right Time

I wrapped everything involved with the AWS networking stack into a python function I call "newvpc". Clearly, this function has side effects, and these side effects include "creating a new VPC4 and all attendant networking hoober joober". Inside this beautiful diamond of a function is a mess of external network calls responsible for all of the hoober joober in question.

Step 1: VPC and Attendant Mysteries

new up a connection:

conn = boto.vpc.connect_to_region(region, aws_access_key_id, aws_secret_key)

The boto namespaced connection thing is an interesting OOP pattern. There's this generic AWS connection that gets reused across all of the various boto namepsacen, onto which all of the namespaced commands (for the VPC aspect of the api, instead of the RDS or EC2 aspects…) sit.

Determine what CIDR block your whole VPC will entail. I use 10.0.1.0/24, which is a lot of IP addresses. You could do all sorts of crazy stuff here, but don't. KISS, neh?

cidr_block = "10.0.1.0/24"

Split that block into two subnets:

subnet1 = "10.0.1.0/25"
subnet2 = "10.0.1.128/25"

Determine the availability zones to which you have access:

zones = conn.get_all_zones()

Create a VPC and enable DNS support:

vpc = conn.create_vpc(cidr_block=cidr_block)
conn.modify_vpc_attribute(vpc.id,
                          enable_dns_support=True)
conn.modify_vpc_attribute(vpc.id,
                          enable_dns_hostnames=True)

Create a gateway and attach it to the VPC:

gateway = conn.create_internet_gateway()
conn.attach_internet_gateway(gateway.id, vpc.id)

Create a route table:

route_table = conn.create_route_table(vpc.id)

Security group:

sg = conn.create_security_group(
        name=cluster_label,
        description=cluster_label,
        vpc_id=vpc.id)

Set yourself up with SSH access to the EC2 instance we'll eventually provision:

sg.authorize(ip_protocol='tcp',
             from_port=22,
             to_port=22,
             cidr_ip="0.0.0.0/0")

Create two subnets:

subnet1 = conn.create_subnet(vpc.id, subnet1,
                             availability_zone=zones[0].name)
subnet2 = conn.create_subnet(vpc.id, subnet2,
                             availability_zone=zones[1].name)

Note that I'm just picking off random AZ's out of the zone list (declared above). You may consider doing something smarter than that, but I just want 2 AZ's to satisfy the RDS gods.

Associate the route table with the subnets, and create a route to the internet:

conn.associate_route_table(route_table.id, subnet1.id)
conn.associate_route_table(route_table.id, subnet2.id)
conn.create_route(route_table.id, '0.0.0.0/0', gateway.id)

If, like me, you want to tear this cluster down after you've provisioned it5, you'll need:

  • vpc id
  • security group id
  • subnet ids
  • gateway id
  • route table id

So return those things to whatever called this function6 that creates a vpc and tower of ancillary networking…stuff.

From there, launching the EC2 instance is pretty trivial:

ec2conn = = boto.ec2.connect_to_region(
       region,
       aws_access_key_id=creds['AWSAccessKeyId'],
       aws_secret_access_key=creds['AWSSecretKey'])
reservation = ec2conn.run_instances(
       ami,
       key_name=key,
       instance_type=instance_class,
       security_group_ids=[security_group_id],
       subnet_id=subnet_id)

Allocate and associate an elastic IP with the instance in question (keep in mind, the instance may not be ready for EIP association yet):

elastic_ip = conn.allocate_address("vpc")
elastic_ip.associate(instance_id=instance_id)

Because we want an RDS in this VPC, we'll need a database subnet group with VPC subnets in two different AZ's (don't ask! I don't know! Appease the RDS gods!):

conn.create_db_subnet_group(identifier, identifier, subnet_ids)

Create the actual RDS instance:

instance = conn.create_dbinstance(
    id=identifier,
    instance_class="db." + instance_class,
    allocated_storage=allocated_storage,
    engine=engine,
    db_name=db_name,
    master_username=master_username,
    master_password=master_password,
    db_subnet_group_name=identifier,
    vpc_security_groups=security_groups,
    port=port)

Where all of the values are the values that you want this instance to have. I've made the call here to keep my RDS instance at the same class as the web instance. I reserve the right to refactor this.

And there you go! Create a VPC with security groups, subnets, gateways, route tables, associate it all together per above, create the db subnet groups when you make the RDS and blammo, you've got an RDS and an EC2 in a VPC together!

Footnotes:

1

To the extent that I'm able to reduce provision- or deploy-time dependencies on tools like Python or Ruby, that makes reusing my stack provisioning scripts just that much easier.

2

You may prefer to work in Ruby. There's an SDK for that! High-level semantics should be largely the same.

3

This is not actually a big deal. Much like Bash, Python can be written in an adequately Lisp-y fashion. Just…functions, is all. You keep your insane class hierarchies to yourself…man.

4

Virtual Private Cloud. While building my familiarity with the AWS API I slowly realized that this whole thing is targeted at corporate IT people looking to extend their networks, not the derpy web developer trying to stand up a set of resources in a simple and understandable fashion. So - one learns.

5

Because dear lord the seventh time clicking through the AWS console to rip out the S3 bucket, EC2 instance, security groups, release the elastic IP…well, that kind of pain is why we automate, isn't it friends?

6

Maybe you think to yourself "screw this guy and his functions, I'm just going to do all of this in one big imperative main() function" and you leave the vpc, sg, subnet etc. objects hanging around for use later. Whatever you do - you'll want those values later to tear down your cluster. Or maybe you're happy clicking at the web interface! Who am I to enforce a workflow on anyone…

Coinbase "insurance" - cold storage uninsured, nothing insured against the government

Filed under: Uncategorized — @ 12:00 a.m.
Coinbase "insurance" - cold storage uninsured, nothing insured against the government

Two notes about Coinbase's insurance:

1:

Coinbase is insured against employee theft and hacking in an amount that exceeds the average value of online bitcoin it holds at any given time.

That is to say that all of the coins in cold storage are entirely uninsured.

2:

Coinbase's insurance also contains standard policy exclusions (e.x. force majeure).

That is to say that should the United States Government decide to requisition all BTC in exchange for notes that promise delivery of bitcoins in the future which will never be redeemed, everyone keeping coins at Coinbase is completely fucked.

Put this announcement in the context of the USG's relentless endeavours to bring as many coins as possible under their control, and:

Consider the repeated and ongoing scam of refusing to sell coins during periods of high volatility (they can't even move money to Bitstamp fast enough to buy coins quickly enough to cover the liabilities that they incur on an ongoing basis - how much does that reassure you about this particular BTC counterparty?).

Consider the absurdity of asking people where they're sending bitcoin and why1.

Consider the case of "drugs and terrorism" - where they closed someone's account on the basis of the account's description of the intended use of the funds2.

Consider that they refuse to transmit coins to certain addresses3.

Consider Coinbase a barely-masked front for USG activities related to Bitcoin. They're advertising insurance that will not pay out for theft of funds in cold storage, a trivial feat for the government to accomplish by bank coercion. They're advertising insurance that will not pay out for "force majeure", which blankets all government action.

Given that one of the most important features of Bitcoin is how (for those capable of handling the radioactive stuff in the first place) it keeps the government from seizing your funds4 and the CB insurance barely protects funds in hot storage, consider their claims of insurance void and worthless. Sheeeeit! It doesn't even make FDIC guarantees! What lolz.

Updated with a bit of fun from contribuderpers to the lulzfarm:

Being able to be in control of your own money is great, if you can do it, but I tend to think most people don't want or can't handle that level of responsibility.

Let me give you a slightly different perspective: the only ones who will have any money in the future are those responsible enough to encrypt it and back it up redundantly themselves.

Footnotes:

1

If this question doesn't strike you as absurd, you don't understand bitcoin yet.

2

If this doesn't strike you as the utterly batty practices of a pathologically insane state, you've yet to realize that you live in a socialist dystopia.

3

If this consideration doesn't strike you as laughable, you don't understand how bitcoin works in some dangerous ways.

4

Shy, of course, a soldering iron in the pooper…

All About the MPEx DERPs

Filed under: Uncategorized — @ 12:00 a.m.
All About the MPEx DERPs

http://explo.yt/post/2014/08/12/DERPs

Jurov commissioned the above-linked piece from me. You could commission the writing of similarly boring and in-depth things!

August 28, 2014

upgrade u

Filed under: Uncategorized — @ 12:00 a.m.
upgrade u

Beyonce and Jay-Z killing it as far back as 2006:

Lil' Wayne's remix:

Nominally, Weezy's remix was a freestyle, but I don't really buy that story: the whole Drought 3 was just so damn good that it had to have been written.

For context, da drought three:

D3 was the height of Lil' Wayne's rap career. Such great raps. Followed by such godawful guitar…

Conformal are the Real Bitcoin Core Devs

Filed under: Uncategorized — @ 12:00 a.m.
Conformal are the Real Bitcoin Core Devs

Many moons ago, a company by the name of Conformal wrote a bitcoin-core replacement they call "btcd". The program is fast, and the code is gorgeously readable, tested to all hell, and documented back from.

The "core dev team", known elsewhere as "the power rangers" or "that despicable group of USG stooges who merged Heatbleed into bitcoin core and have done any other number of things over the past two years to degrade the network quality1" has recently deployed BIP0064 to the public. The nominal goal of BIP0064 is to allow network clients to request information about unspent transactions.

David Hill (of the team in question) writes about BIP0064, and the "core" development team's development process:

Another issue this brings up is the amount of testing, or the lack thereof, that goes into protocol changes. It is worrisome that protocol changes get merged without accompanying extensive test coverage. This is why the Conformal team puts such a strong emphasis on complete test coverage to help catch such issues that easily go unnoticed by developers.

Mike Hearn responds:

The reason the getutxo patch does not have any unit tests was not lack of trying.

A for effort! Instead of doing the hard and correct thing, which is ensuring that your patch to the crappy codebase works as expected, just commit the change and push it to the unsuspecting public.

Suggestions for anyone considering their own bitcoin infrastructure:

  • Don't trust a word out of the "core" team2
  • Run Conformal's btcd, and not bitcoind

"Bitcoin Core Devs": considered harmful.

Footnotes:

1

Including but not limited to the hilarious change in the wire protocol that includes node IP addresses in Bitcoin network communications.

2

There is professional software development, and there is what happens to that poor C++ harball. "Play" would be a charitable description, as it implies that that folks doing things over there are learning, growing, or developing somehow as human beings.

Older Posts »

---