Friday, January 25, 2008

Kahn! K-A-A-A-A-A-HN!

Professionally, I get to spend some time thinking about the Unthinkable. The problem is you can't really think about the Unthinkable, because it's well unthinkable. This is not a Cheap Laff; we forget how to think about the Unthinkable as we learn self-preservation as a useful life skill. So we content ourselves with thinking about the probably not-able, and pretend that's an acceptable substitute for the Unthinkable.

A few years ago I got to manage company preparations for Y2K. Who remembers Y2K, the most expensive Unthinkable non-event in human history? For those of us who can't remember such pretend-disasters, Y2K was supposed to potentially be the end of civilization as we knew it because unprescient early computer programmers economizing on coding, on account of a criminal lack of memory available in prehistoric steam-driven computing machines, left off the "19" in writing date/time codes. The programmers assumed that somebody would come around and patch the code long before the turn of the century. If it wasn't fixed, then at 00:00:00 on 1/1/2000, the computing machine would think that it was 00:00:00 on 1/1/1900 and be unable to access any date-certain file until 85 to 95 years later, which would be inconvenient when applied to things like power transmission systems, or the world's monetary system, or setting your VCR to record Walker, Texas Ranger. Problem was, that hadn't been done with bet-your-life certainty by 1998; and thanks to the well unthinkable explosion of computing machines integrated into many if not most electromechanical conveniences of modern society, no one could say with any degree of certainty that every single device that could be affected by the rollover had been identified and properly patched. And it became sadly apparent that because there was potentially a computing machine in every single electromechanical device, in order to be fully prepared one had to audit every single electromechanical device in a business/personal inventory and verify that it was patched. In this regard, IT departments busily checked the obvious computers that were part of the business' normal IT inventory; but their scope, interest, professional concern stopped at the limits of their inventories. All those other devices were Somebody Else's Problem. For most American businesses, the Somebody Else was the facilities department; and the luck of the ticketpunch put me in that particular hot seat in 1999.

So we had to check and test everything that was digitally controlled in any way. Now if the computing machine head end was fairly obvious, like say an HVAC or building system controller, you could check with the vendor for a patch, which was usually available for a hefty fee since it was not considered to be a routine upgrade. But if the computing machine head end was not obvious, like say the timer in a coffeemaker or the thermostat in a refrigerator, that was a little trickier to verify. It meant conducting a pre-forensic audit of every mechanical thing that you owned; and the audit not only had to account for the presence of the thing, it had to account for a field test that verified either that the device was unaffected and would function properly, or was affected and patched, or was affected and unpatched but a patch had been identified and was on order and would be installed well before The Day, or was affected and unpatched and no patch was available so the device had to be taken offline and either replaced or junked with a patched device. This was an incredibly tedious, time-consuming and expensive process.

At one point we were checking emergency building systems; and because the Worst Case Scenario was that Y2K would shut down the PJM Interconnection, which is the entire east coast power transmission network, we needed to verify that the emergency generator would in fact function properly in case of a full power grid failure. Once I actually spoke these words I experienced the full pushback of the entire maintenance operation, because a full grid failure was, well, Unthinkable. Had never happened in Our Nation's Capital, not since the grid was turned on back in ought-whenever. The closest thing anybody could think of was the occasional power failure caused by summer storms, which never lasted more than a day. Or a week. And happened every summer. But had never happened downtown, we being on the same grid as the White House dontcha know and the White House will never be affected by the same reality that affects you and me and Bobby McGee. It seemed to me that that being the case, it was likely that the emergency systems had never been tested since they were commissioned, which was almost twenty years previously: there were certainly no records of testing, not even of turning the generator over to make sure that it still kicked on. In which case it seemed unwise to test the generator involuntarily at 00:00:00 on 01/01/00; we probably wouldn't be able to get a repair guy in until well after the Apocalypse. So I proposed that we simulate a total grid failure, take the entire building completely off line just to see what happened. This provoked Son of Pushback; as I recall, it took the better part of two weeks and the expenditure of every erg of clout and the call-in of every chit I had out there, plus a couple of well-placed threats of termination for insubordination and the payout of a bunch of overtime, to put a fake blackout in play. Because such a thing had never happened, could never happen, and preparing for it was the dumbest thing said by a white boy since the Greaseman ran his mouth on the air. Just, well, Unthinkable.

So we pulled the plug on a mid-week night in early August. Nights in August in Our Nation's Capital are hazy, humid, hot and horrible. Pulling the plug showed us two things: the 800,000 square foot building did not hold temperature worth a poop as it quickly equalized to outside temperature in about 15 minutes; and the emergency generator spun up as soon as the last power leg was taken off line, and just as quickly shut itself off and took all the life safety equipment down with it. Huh. Unthinkable. And great was the embarassment across the land. Turns out that the generator's power sensors had been miswired from the get-go; and instead of monitoring the normal building power feeds for proper power flow, it monitored its own output. So as soon as the generator kicked on, it measured power flowing (well yeah, that's what a generator does -- it generates power, dontcha see) and it shut itself down again. Which it would have done every time it was turned on and tested. Which meant that it had never been tested or exercised since commissioning; and that it had been commissioned without testing. Which was extremely embarassing to the maintenance department, many members of which had been around since the system was commissioned.

Me, I thought it was a great test because everything failed miserably. That meant we didn't have to assume anything worked, so we might as well fix the whole thing. It also occurred to me that there was a pretty good chance that somebody knew that everything would fail miserably, hence the pushback; but I couldn't prove it, so that battle would have to be fought another day.

But what does this have to do with thinking about the Unthinkable? Even I wasn't prepared to think this one all the way to its logical conclusion, which would have been to arrange for backups and transfer systems sufficient to provide 11.7 kilovolts continuous service for up to four weeks. Basically, this would be generators the size of locomotives, transfer switches the size of bedrooms, and a supertanker full of diesel. Would have been kind of a tough sell to the Board of Directors, would probably have required the entire organization budget for the year, and would probably not have been available as a minority set-aside government procurement -- and all for a "just in case the world ends we can maintain normal business operations for a month" scenario. Because that's Unthinkable.

We had a few more adventures like this, and at 11:55:00 on 12/31/99 four of us hanging around at work with nothing better to do at the turn of the century decided that we might as well mosey on up to the security command center and peep over the operators' shoulders as the computer terminals rolled over from 11:59:59 12/31/99 to 00:00:01 01/01/00, just in case. Which they did. Kept right on going, too, Lights didn't go out or nothing. All that plywood we bought to board up the windows in case Our Nation's Capital went blip and concerned citizens decided to throw a few celebratory trashcans through said windows, for naught. Sighs of relief and toasts of Diet Coke all around (we were on duty and on the public's dime, you know) and home we went, accompanied by the happily tipsy revelers in their dumb paper party hats and horns tooting the Millenium and their fabulous lives.

1.75 years later some guys flew some planes into some buildings. Didn't see that one coming, either. Unthinkable.

Tomorrow I am going in for crowd control duty. For seven hours I will do my best to direct 30,000 people one at a time to two ticket will call windows or two credit card terminals. I don't want to think about what would happen if we had to get all those people out of the building because a fire alarm goes off, or some young person thinks another young person looked at him/her in a less than wholly respectful way and decides that a few disciplinary rounds from their personal peacemaker, or a scientifically applied whack from their personal machete, is in order; or a student of a variant sect decides that another wakeup call for the Great Satan is in order and detonates himself in a field vest filled with C-4 and nails. Because that stuff is, well, Unthinkable.

No comments: