EP2000

The mind is a wondrous tangle of connections and memories. For Proust, one bite of a cookie (a Madeleine, I think it was) tugged at recollections that, when unraveled, stretched into something like seven volumes of reminiscence that defined 20th century French literature and could kill a man with sheer weight.

My reminiscing hasn't gone to that length. Either I lack his patience, or have spent more of my life actually doing things, rather than lying in bed in the cork-lined attic of my parents' apartment.

But I had a little shot of Proustian recollection yesterday, while strolling downtown Palo Alto with my wife. We passed the local incarnation of The Cheesecake Factory, and I couldn't help but being vividly drawn back 12 years, to the first (and perhaps only?) time I ever ate at one of those restaurants.

It was the day after our third wedding anniversary, which I had spent, as the three days before, in a hotel room in Chicago. With five other sweaty, sleep-deprived software engineers. Flailing at a hopeless tangle of code, rearranging its bits like deck chairs on the Titanic while we watched the bow of our company dip inexorably below the arctic waves.

Everything had started innocently enough, about 9 months earlier. Satinder - my MIT officemate - and I were engineers with a quirky British software company called Harlequin. Harlequin had always been driven by audacious, overreaching ideas that somehow, inexplicably, managed to (almost) work. Over the prior 10 years, Jo, our visionary CEO and sole owner of the company, had built the little basement consulting firm into a 300-person outfit with six offices on two continents. What was remarkable was that he'd managed to do this entirely without external investment, by extending his own meager assets through the time-honored tradition of robbing Peter to pay Paul. Well, not exactly "robbing". He'd systematically finance projects underway by pre-selling the Next Big Thing under development.

Anyhow, the trouble started when Sang, our VP of sales, poked his head into our office one afternoon.

"Hey guys - got a minute?"

Now, Sang's nickname was "Drive by" - as in shooting. He'd show up and pause for a brief conversation before moving on, but tended to leave (figurative) blood, broken glass and general carnage in his wake. So "got a minute?" was as ominous start as any, but hey - he was the VP of Sales; the only reasonable answer had to be "Sure! What can we do for you?"

Harlequin built software for the digital pre-press world. Our clients were the high-end print shops, the places where Life, Newsweek and Vogue were printed. We sold them code that helped lay out their text, tweak their images, and drive the massive printing presses that turned electrons on a disk into uncountable miniscule droplets of four-color ink on a page.

Harlequin thrived in this market because print shops, by and large, were still dragging themselves out of the late 17th century. They used computers, but each task was performed on a separate physical machine. When you came in with a job for them to do, they'd expect you to drop it off in person as a physical pile of disks in a transit bag. They'd look up at the master task board, where other jobs had been scrawled out in grease pencil, squint a bit while trying to figure out what spare capacity they had in their shop, and give you an estimate on how long it would take them to get it done.

When they got around to working on your job, the taskmaster (yes, that's what they called him!) would assign your bag to someone on the shop floor. He'd then figure out what actually needed to be done, and start hunting around for vacant machines on which to do it. He'd load the disks onto the first available layout machine he found, do the layout work, then pull the disks out and look around to see if there was an available image proofing station. Then... you get the idea.

Well, the year was 1997, the world was abuzz with networking, and Sang had dreamed up was a beautiful idea:

"So guys - instead of the print shops manually lugging disk packs around the floor, we could build something that just let them send tasks from machine to machine electronically, right?"

"Absolutely."

"And we could do some sort of analysis to model how long each of the tasks should take on each machine, right?"

"Well, maybe..."

"And if we knew how long each task would take, we could throw some scheduling software on top, and have the system do all its resource management automatically, right?"

"It's conceivable, but..."

"Wouldn't that be awesome?"

"..."

"How long do you think it would take us to build something like that?"

We explained that any estimate we gave him then and there was going to be meaningless. This was large-scale modeling, predictive control and integrated scheduling. As I recall, we shoveled as many weasel words in as we could, and told him that three years would be a reasonable guess for starters.

Sang frowned. "Hmmm, that's not good. Do you think you could get something working in 9 months?"

"Uh, no. But why?"

"Because I just came out of a sales meeting with Heidelberg, and I told them that's when we could deliver it to them. They're really excited. Oh crap, that's my wife calling. Gotta go - let's talk tomorrow about staffing this thing up. It's going to be great!"

And he was gone.

The following months were a painful blur. EP2000, as the project came to be known, was loudly promised to our customers as the messianic system to bring digital pre-press into the 21st century. Jo announced that we'd unveil the system in all its glory at Print 97, a mammoth trade show in Chicago, just 7 months down the line. Internally, all was chaos, folded over the inevitable recurring themes: the chorus of engineers repeating "No, we just can't build it by then!" The chorus of sales managers retorting "But you must build it by then!" The rationale was that, when they'd given the engineering team impossible assignments in the past, we'd always succeeded. So why were we slacking now?

Cliff, our august Director of Technology, was a veteran of IBM and the mythical man month; he'd been blindsided by the EP2000 sales promise too. He knew what the endgame looked like, and was doing his best to prepare us for soft landing, but there's only so much you can do when your superiors designate you pilot-in-command of a plummeting brick of a project. While Jo trumpeted the triumphs of the digital revolution and Sang sold it to every one of our customers, Cliff scrambled behind the scenes trying to moderate expectations with pushed-under-the-door cautions to "recall that these are only anticipated features and projected delivery dates."

In the layers below Cliff, Harlequin's engineers had long ago pulled out the stops. In defiance of the inevitable, more developers from each of Harlequin's six offices were being thrown into the desperate fray. I wish I could remember all their names, but as I said, it's all a painful blur at this point.

In the end, it came down to that final week before Print 97. Jo had rented a massive booth to display the EP2000 hardware, and promised to unveil the system at the scheduled press conference. "Unveil" was looking more and more like running a cardboard box demo that simply took a single prespecified job in the front, recognized its components, estimated the time required for each, and fired the tasks off in sequence from a master controller. And getting it to work was still a longshot. The network code hadn't ever been tested; none of the components had ever had the chance to even *try* to run together. And we'd already lost over a quarter of the team to burnout.

Someone made the decision to bring the tech leads and key developers to Print 97, to finish the work on site. Satinder had left the company for the safety of an academic position, and by dint of my hand in the resource estimation code, I was sucked out to Chicago for the week. We couldn't use the floor show EP2000 machines while the trade show was in progress, so during the day we piled into a trade show hotel room festooned with ethernet cables and power cords connecting an improvised tangle of laptops and workstations.

Pizza was delivered twice a day, and showers were clearly optional. At night, we'd emerge like nocturnal animals, creeping the darkened tradeshow floor for an hour or two before they locked up, to try what we'd cobbled together on the "real" hardware.

It was an awful time, mixed with visceral fear, desperation, and the occasional glimmer of hope. On the last night, there was a feeling that, in spite of it all - in spite of what it meant to ourselves - we might just make it. The press conference had been delayed and rescheduled again for the closing hour of the show; the last plausible moment when Jo could "unveil" the system to the world. The evening before, amid our tangle of cables, pizza boxes and crushed cans of Mountain Dew, Angus had managed to get a test job all the way through the test system. The kludges were almost unthinkable, but they were holding together. That process that would hang after running 5 minutes in the queue? Run another background process to kill and restart it every four minutes. The estimator occasionally returning "minus infinity" as a resource requirement? Slap on a postprocessing step that "adjusted" times to be in the range between 30 seconds and, oh, just under four minutes.

We were entirely spent, but we'd gotten it to work on the test machines. There was just one last thing to do: get to the trade show floor when opened in the morning and see whether the code worked on the actual setup.

I stood back in the morning and let the systems team take the lead when the doors unlocked. Our emotions were spinning like Wheel of Fortune. Hardware booted up, files copied, job loaded. People were starting to mill around the floor, hovering tentatively at a safe distance, wondering if this was a preview of the much-anticipated EP2000 debut. We didn't look anything like the natty slick sales guys or booth babes that normally staff the display. We were an anomaly and were rapidly becoming the center of attention. It was now or never.

Someone (Angus?) uttered a mild oath in a language I didn't understand, and hit "Enter". Network lights came alive, disks on a remote workstation spun up. The task list scrolled on the master console, then...nothing. Absolutely nothing. We stood for perhaps 30 seconds holding our collective breath - maybe one of the disks was still paging? Maybe one of the machines had a loose cable? Maybe...? But we all knew it was over. Angus delivered a brief, much stronger, oath under his breath and slowly backed away from the console, much like an erstwhile bank robber stepping out of the cashier's line and strolling for the exit. There was nothing here see, just an engineer or two checking out the machines. We followed his lead and melted into the crowd, trying to be inconspicuous.

I went back to my room, took a shower, put on a clean shirt, and stepped out into the first sunshine I'd felt for days. Wandering into the business district surrounding the convention center, I found myself confronted with the brick and glass of the local Cheesecake Factory and went in. I was curious what real food tasted like after so much pizza.

That afternoon at the press conference, Jo "unveiled" the system, pointing to the massive array of whirring disks and flickering screens. Reportedly, when someone asked to see it in action, Jo responded "Oh, no, this is just a static demo. If you want to see the system in action, you'll have to come see us at the Seybold trade show - four weeks from now!"

I don't know if EP2000 ever shipped. I began work on my resume over the weekend, and I understand that less than half the EP2000 team checked in for work the following Monday.