Well, don't. But if you have to, you've come to the right place, as I just spent the past week automating cluster provisioning:
Some Philosophical Considerations
I endeavor to write as much of my provisioning in Bash as possible, so that I can run it with a minimum of modification1 wherever else it pleases me to do so. Working with AWS, though, pretty much demands that I2 hack out the AWS-related parts of these provisioners in Python3.
All this setup for: "My provisioner doesn't shell out. I wrote it in Bash, and it sells out to Python".
You're welcome. It hurt to write too.
Geting Everything in the Right Place, at Just the Right Time
I wrapped everything involved with the AWS networking stack into a python function I call "newvpc". Clearly, this function has side effects, and these side effects include "creating a new VPC4 and all attendant networking hoober joober". Inside this beautiful diamond of a function is a mess of external network calls responsible for all of the hoober joober in question.
Step 1: VPC and Attendant Mysteries
new up a connection:
conn = boto.vpc.connect_to_region(region, aws_access_key_id, aws_secret_key)
The boto namespaced connection thing is an interesting OOP pattern. There's this generic AWS connection that gets reused across all of the various boto namepsacen, onto which all of the namespaced commands (for the VPC aspect of the api, instead of the RDS or EC2 aspects…) sit.
Determine what CIDR block your whole VPC will entail. I use 10.0.1.0/24, which is a lot of IP addresses. You could do all sorts of crazy stuff here, but don't. KISS, neh?
cidr_block = "10.0.1.0/24"
Split that block into two subnets:
subnet1 = "10.0.1.0/25" subnet2 = "10.0.1.128/25"
Determine the availability zones to which you have access:
zones = conn.get_all_zones()
Create a VPC and enable DNS support:
vpc = conn.create_vpc(cidr_block=cidr_block) conn.modify_vpc_attribute(vpc.id, enable_dns_support=True) conn.modify_vpc_attribute(vpc.id, enable_dns_hostnames=True)
Create a gateway and attach it to the VPC:
gateway = conn.create_internet_gateway() conn.attach_internet_gateway(gateway.id, vpc.id)
Create a route table:
route_table = conn.create_route_table(vpc.id)
sg = conn.create_security_group( name=cluster_label, description=cluster_label, vpc_id=vpc.id)
Set yourself up with SSH access to the EC2 instance we'll eventually provision:
sg.authorize(ip_protocol='tcp', from_port=22, to_port=22, cidr_ip="0.0.0.0/0")
Create two subnets:
subnet1 = conn.create_subnet(vpc.id, subnet1, availability_zone=zones.name) subnet2 = conn.create_subnet(vpc.id, subnet2, availability_zone=zones.name)
Note that I'm just picking off random AZ's out of the zone list (declared above). You may consider doing something smarter than that, but I just want 2 AZ's to satisfy the RDS gods.
Associate the route table with the subnets, and create a route to the internet:
conn.associate_route_table(route_table.id, subnet1.id) conn.associate_route_table(route_table.id, subnet2.id) conn.create_route(route_table.id, '0.0.0.0/0', gateway.id)
If, like me, you want to tear this cluster down after you've provisioned it5, you'll need:
- vpc id
- security group id
- subnet ids
- gateway id
- route table id
So return those things to whatever called this function6 that creates a vpc and tower of ancillary networking…stuff.
From there, launching the EC2 instance is pretty trivial:
ec2conn = = boto.ec2.connect_to_region( region, aws_access_key_id=creds['AWSAccessKeyId'], aws_secret_access_key=creds['AWSSecretKey']) reservation = ec2conn.run_instances( ami, key_name=key, instance_type=instance_class, security_group_ids=[security_group_id], subnet_id=subnet_id)
Allocate and associate an elastic IP with the instance in question (keep in mind, the instance may not be ready for EIP association yet):
elastic_ip = conn.allocate_address("vpc") elastic_ip.associate(instance_id=instance_id)
Because we want an RDS in this VPC, we'll need a database subnet group with VPC subnets in two different AZ's (don't ask! I don't know! Appease the RDS gods!):
conn.create_db_subnet_group(identifier, identifier, subnet_ids)
Create the actual RDS instance:
instance = conn.create_dbinstance( id=identifier, instance_class="db." + instance_class, allocated_storage=allocated_storage, engine=engine, db_name=db_name, master_username=master_username, master_password=master_password, db_subnet_group_name=identifier, vpc_security_groups=security_groups, port=port)
Where all of the values are the values that you want this instance to have. I've made the call here to keep my RDS instance at the same class as the web instance. I reserve the right to refactor this.
And there you go! Create a VPC with security groups, subnets, gateways, route tables, associate it all together per above, create the db subnet groups when you make the RDS and blammo, you've got an RDS and an EC2 in a VPC together!
To the extent that I'm able to reduce provision- or deploy-time dependencies on tools like Python or Ruby, that makes reusing my stack provisioning scripts just that much easier.
You may prefer to work in Ruby. There's an SDK for that! High-level semantics should be largely the same.
This is not actually a big deal. Much like Bash, Python can be written in an adequately Lisp-y fashion. Just…functions, is all. You keep your insane class hierarchies to yourself…man.
Virtual Private Cloud. While building my familiarity with the AWS API I slowly realized that this whole thing is targeted at corporate IT people looking to extend their networks, not the derpy web developer trying to stand up a set of resources in a simple and understandable fashion. So - one learns.
Because dear lord the seventh time clicking through the AWS console to rip out the S3 bucket, EC2 instance, security groups, release the elastic IP…well, that kind of pain is why we automate, isn't it friends?
Maybe you think to yourself "screw this guy and his functions, I'm just going to do all of this in one big imperative main() function" and you leave the vpc, sg, subnet etc. objects hanging around for use later. Whatever you do - you'll want those values later to tear down your cluster. Or maybe you're happy clicking at the web interface! Who am I to enforce a workflow on anyone…