March 13, 2018

Classically inept couriering

Filed under: Uncategorized — Benjamin Vulpes @ 8:34 p.m.

Catastrophes in complex systems typically fall out of failure cascades: for want of a nail, a kingdom was (quite nearly!) lost. I delivered most of the gear Pizarro expected out of my courier run, but the number of close calls and skin-of-the-teeth saves doesn't sit well with me, I executed poorly on several fronts, and could have responded better to on-the-ground challenges under time pressure. Cleared the pillbox, sir, but lost a few men doing so. In the spirit of my lecture on how to make mistakes, here is a summary of the trip, the things I did wrong, and what I'll do differently next time.

The root causes of my failure cascade were haste, niggardliness, and seeking insufficient counsel. I sought to staunch Pizarro's hemorrhaging of cash by moving the machines down to our datacenter in Montevideo as quickly as possible, on as much of a budget as possible. To this end, I picked a travel period that provided about one week less of preparation time than the other options in order to save ~400 dollars on the plane ticket, and crammed all of the machines into a single crate to save on baggage overcharge fees on the order of another hundred and fifty dollars.

As the clock wound down for departure, the RAID card in a machine I was provisioning for a customer refused to perform any RAID-ly duties. I'd had the foresight to buy a RAID card (that , if it proves out, will likely hit the standardized hardware list). A mixup in cable specification ensued, which I resolved in the hours immediately preceeding takeoff.

In the runup to this trip, I called both Aeromexico and GOL (the airlines on which I was to fly) several times to check maximum bag weight; and on all occasions was told that there was a weight schedule, the airport would tell me the fees, and that provided I paid the extra weight fees they'd have no problem loading and transiting the bags. When I checked the crate containing the four machines in at the Delta-operated gate, the first warning signs that everything might not go according to plan cropped up. First: the gate staff called down to the baggage handling area to confirm that they could in fact accept a bag weighing 150 pounds (who confirmed that they could indeed). Second: junior gategirl needed an assist from senior gategirl to figure out how to check the bags (one for the machines, one for the rails) through to Montevideo from Sao Paolo (which the Urugayos inexplicably insist on referring to as San Pablo), transferring from Aeromexico to GOL.

Bags checked, I checked my own butt through security, and settled down for a beer before the flight to Mexico City, four fifths of the way through which Delta paged me back to the baggage check, to inform me that they'd been wrong all of the times I'd asked about weight limits and that they were going to refuse to check the bag unless I could pull its weight down under 100 pounds. Thirty minutes later, I had extracted one machine from the once-checked bag, all power supplies and disks from the machines remaining in said bag, and sent the lonely machine back to my base of operations. My girl gracefully arose from her slumbers, brought me additional bubble wrap, and ran the abandoned server home. May the Republic be forever so blessed. Another five minutes passed, and at nearly precisely the same time as I discovered that the extracted drives and power supplies would not fit in my bag and leave it of carry-on size, the gate staff offered to check another bag gratis. I dashed around the corner, procured a bag that'd reasonably hold the remaining bytes, stuffed them in, and checked it. This is the point at which the gate staff and I both made our final, near-fatal, and certainly expensive mistake: they failed to check the bags through to Montevideo, and I failed to check the baggage claim ticket.

Fast forward forty eight hours, to baggage claim in Montevideo, where, having retrieved the rails and servers, I awaited the third bag of componentry. The gears meshed with a sickening thud in my mind as I realized that the Delta agents for Aeromexico failed to check the third bag through to Montevideo, and I'm now extremely likely to have a customs nightmare to navigate. I filed the appropriate papers with the GOL babes in luggage claim, shuttled the machines through customs with barely a second glance, and rode with BingoBoingo and a datacenter representative directly to the racks for installation of what material made it off the plane per the plan.

At our cabinet in the DC, the posts were too far apart for our rails, which demanded a trip (first to the ferreteria, and then the local chinesium dealer once it became apparent that the lady at the head of the line intended to wax poetic about the qualities of the single sheet of sandpaper she was considering buying) to acquire a crescent wrench. Wrench in hand, we retired to the basement to continue the racking project. Moved the posts, bolted everything back down, installed the rails, racked the machines, ran ethernet to the switch from each of the machines to the switch, and then bailed. Hotel, cambio, comida, cowork, and back to the hotel to patch up some amount of sleep debt.

The missing bag came in on the next flight, around 23:45 that night. In retrospect, there is a low chance that my being on hand to retrieve the bag as it came off of the carousel might have avoided the customs shitshow that ultimately ensued, but I elected instead to sleep and retrieve the bag first thing in the morning. File it under mistakes, because how is one to know.

I arrived at the airport at 09:00 the next morning, and after another thirty minutes of haranguing the information booth staff, managed to get my hands on a GOL representative to walk me backwards through security into the customs office to pick up my bag. The lady on duty, who'd waved me through with barely a second glance yesterday, now insisted that the items in my bag all counted as imports and needed proper handling, and sent me away with instructions to get an "expediter" to take care of the rest of the business and that they wanted nothing further to do with me.

At this point, BingoBoingo already had a lead on an expediter: the son of the man who owns the coworking space at which BingoBoingo rents a desk. Before leaving the airport, I reached out to this new contact, one Martin, to get the ball rolling on a quote to extract the goods. Back to the cowork, print various papers, and bake a list of items and prices that will plausibly pass with the MVD customs agents. We also put together a little dictionary of photos and descriptions we can hand to our expediter so that he knows which line items on our fabricated invoice correspond to which items in the bag, and get a disappointed board to signoff on the fees that we'll have to pay to get our gear out of hock (~1800 USD).

My mistakes: haste, which nearly cost us the ability to deploy a RAID-capable machine; inattention, which (by not inspecting all three baggage claim receipts) allowed for a bag misrouting and cost ~.19 BTC to fix; inadequately consulting with asciilifeform and mod6 during the first round of chaos when checking bags, which got me (entirely reasonably) caned for letting the irreplaceable FUCKGOATS out of my sight, and a failure to be at the airport for the next flight from Sao Paolo to minimize customs interception risk.

What I've learned: Baggage claim receipts are to be scrutinized. Input from the forum/board is to be sought aggressively when under duress and more likely to make bad decisions. Crates for transport are to weigh absolutely no more than 100 pounds, and if machines and crate and power supplies and drives all together exceed the hundred pound line, everything of value must be transported in carryon, lest it get separated and subjected to additional scrutiny. Based on my experiences with customs both times, anyone taking hardware to Spaceship Montevideo must conceal as much of it in servers as possible, and unbag that which cannot be plugged in, as items that are "new in box" trigger the customs officials no end. For instance, drives in caddies were effectively ignored, while those in boxes spent an inordinate amount of time in customs agents hands. Finally:

asciilifeform: for future reference : 1) they travel in a, e.g., 'pelican', labeled 'industrial sample', and you walk in with a prepared speech where you solemnly promise to whomever wants , that you will return back to usa with them . (in the quite unlikely event , like meteorite falling in the same funnel twice, that you are held to this on return path: 'orcs stole'em' )

On the upside, 3 of the servers, and all of the hardware those machines needed arrived safe and sound, although at a rather steep cost.


  1. BingoBoingo says:

    With the hastily packed third bag, some beach towels and other generic vacation paraphernalia may have helped the cause, but haste. And lest we ever forget, Super Mercado Winn Dixie!

RSS feed for comments on this post. TrackBack URL

Reply to BingoBoingo

« Best ecommerce landing page of 2017 --- The CIA's flickr account »