Title: A Query Language for Understanding Component Interactions in Production Systems [pdf] [slides]

Published: International Conference on Supercomputing (ICS), 2010.

When something unexpected happens in a large production system—a program crashes, a node’s performance flags, a power supply overheats—administrators face several problems at once. First, they may be unable to describe the event any more accurately than the approximate time it occurred. Second, they must diagnose the problem using only the data that was recorded when the issue manifested (primarily log files); this data may be noisy and may not describe all components and their interactions. Third, the system may have many components (tens to thousands), and the administrators must identify which components and component interactions are likely to have been involved.

Consider the following example. Users notice that their jobs are failing more frequently. The typical process for a system administrator is to search the job logs to figure out what components were used by these jobs, scour the system logs from those components for any messages that might hint at a cause, and possibly expand the search to other related components based on their expert knowledge of the system. The key observation is that this is fundamentally a search problem—one for which the state-of-practice is primarily manual, tedious, and ad hoc—where the administrator asks, “What components and interactions are likely to be involved with these job failures?” The input to the search is the available measurements from instrumentation and a simple description of the behavior we wish to understand; the goal of the search is to identify the components and interactions that are likely to be involved.

In a paper to appear at ICS 2010, we present a method for using simple user specifications of when and where a problem manifested, together with existing instrumentation, to compute the components and interactions that are likely to be involved with the problem. Our method computes which system components statistically influence the behavior of other components and which components are statistically linked with the problem.

Our system, QI (pronounce ‘chee’), does not require modifications or perturbations to the system, access to source code, or even knowledge of all the components in the system or their dependencies on one another. Our assumptions are considerably weaker than most previous work and they reflect, in our experience, the reality faced by administrators when they must diagnose a problem. The answers QI provides are limited by these contraints: a passive, black-box technique can, at best, suggest the components and interactions that seem statistically most likely to be involved with a problem. The main advantage is that, because of the weak assumptions, such a system can leverage all of the information available. This is precisely what our method provides, and it does so in a way that is computationally efficient and applicable to a wide variety of systems.

In particular, we evaluate QI using nearly 1.22 billion lines of code from unmodified production systems: four supercomputers, two embedded systems, and a server cluster. On these data, we correctly answer a wide variety of exploratory and diagnostic questions about dynamic system behavior, usually in a couple of seconds.

Title: Using Correlated Surprise to Infer Shared Influence [pdf] [slides]

Published: International Conference on Dependable Systems and Networks (DSN), 2010.

During the DARPA Grand Challenge race in 2005, the autonomous vehicle named Stanley, Stanford’s entry, slowed down and swerved around an obstacle that was not actually there. Stanley did this several times over the course of the race, nearly causing it to be disqualified. Although Stanley went on to win the competition, the Stanford Racing Team was justifiably vexed: why had Stanley hallucinated these obstacles?

Using a hand-crafted dependency diagram—the golden ideal to which all previous work on dependency inference aspires—it took the designers of the system early two months to isolate the problem, which was originating from a buffer component shared by the laser sensors. This shared buffer was intermittently dropping measurements, causing Stanley to see stale, inconsistent data about the world around him, which sometimes meant seeing obstacles where there were none. Every other component of the system was behaving according to specification, which partially explains why the dependency diagram was so unhelpful: the source of the problem and its outward manifestation (swerving) were on opposite logical ends of the system but the dependency diagram advised looking at almost every component in between. The shared buffer was not even on the diagram.

In a paper to appear at DSN 2010, we introduce the idea of computing influence, a type of component interaction that is orthogonal to dependencies and allows us to capture implicit interactions among components and subsystems. For the Stanley swerving bug, our method not only infers an influence directly between the swerving behavior and misbehavior near the laser sensors, it also implicates an uninstrumented component shared by those lasers: the true cause of the problem. The Racing Team says the results of our analysis, which took only a few seconds to compute, would have saved them two months of debugging.

Computing the strength of shared influence between components is straightforward:

1. Represent the behavior of each component as a function of surprise over time, called an anomaly signal.
2. See how well these functions “line-up” using a standard technique called cross-correlation.
3. Summarize cross-correlations in a Structure-of-Influence Graph (SIG), where the edges indicate the strength and time-delay of the influence.

Our paper gives a mathematical foundation for influence, as described above, and evaluates it using both simulations of idealized systems and case studies with real systems, including Stanley, his successor (Junior), and the Thunderbird supercomputer.

We propose a method for identifying the sources of problems in complex production systems where, due to the prohibitive costs of instrumentation, the data available for analysis may be noisy or incomplete. In particular, we may not have complete knowledge of all components and their interactions. We define influences as a class of component interactions that includes direct communication and resource contention. Our method infers the influences among components in a system by looking for time-correlated divergence from models of individual component behavior. We summarize the strength and directionality of shared influences using a Structure-of-Influence Graph (SIG). This talk explains how to construct a SIG and use it to isolate performance bugs, and presents both simulations and an in-depth case study using data from two autonomous vehicles.

Meanwhile, I’ve been keeping busy. Here’s a recent sampling:

• The US Patent Office granted me a second patent. The first one, meanwhile, has already passed into obscurity.
• I got an alert detection paper [pdf] accepted to ICDM in Pisa, Italy, where I’ll be going in December to talk about it.
• I served on the program committee for a new workshop called WASL and am on the program committee for NAS ‘09.
• I applied to graduate… sort of. The requirements for a Master’s are a subset of those for my Ph.D., so I filled out the paperwork to pick up an extra degree. I already have an M.Eng. from MIT, so this one’s redundant and will be obsolete whenever I get my doctorate.
• I gave a few invited talks, most recently at a workshop at LACSS [slides].
• I agreed to serve on the CS Department’s faculty search committee.

I’m working on a system modeling paper for submission on Monday. Wish me luck!

[A man in a blue suit strides to the podium, the enormous ballroom is filled to overflowing with scientists and military officers. Cameras are focused on him from all sides, projecting his visage onto the screens behind him and into the many satellite viewing rooms throughout the hotel.]

Good morning. I’m going to talk to you today about the future: a vision of the future as seen by the DARPA Made-Up Technologies Office. The best of the best. DARPA’s DARPA. Rambo to STO’s Barney Fife.

Imagine a world in which soldiers cannot die. In which their armor adapts to new threats instantaneously, their weapons target flawlessly and inflict the desired damage, and their hair maintains its shine and bounce, even in the harshest of combat conditions. Imagine a world where a global information network is accessible at your fingertips, or even closer, like at your knuckles or wrists. Where you can detect enemies breathing behind concrete walls, clot and repair a bleeding femoral artery with a simple tourniquet, and where a universal replacement part can assume whatever shape or function you desire. A wrench becomes a hammer. Wings take dream.

We at the MUTO are imagining exactly that.

Soldiers must fight in extremes. In the snow dunes of the arctic, the sand drifts of the desert, deep beneath the ocean, on mountain peaks, and, someday, in outer space and in the center of our sun.

[Slide show displays the Sun. Speaker gestures meaningfully.]

Our opponents are smart, capable, well-trained, and fighting on their home turf. Some of them can yodel. Most of our soldiers can barely manage a passable Star Spangled Banner. Our Army Rangers train in the mountains of Georgia, while Afghani fighters are acclimated to altitudes tens of thousands, no, millions of feet higher. Geese can handle these altitudes, why can’t our warriors?

Our enemies have rockets launchers. Some of them have elephants. They may even have figured out how to put rocket launchers on elephants. You can’t prove they haven’t. And when they do, will you be able to say you did everything possible to prepare?

The work we do at MUTO represents not merely fundamentally unique technological achievements, but entirely new fields of research. A calculus of awesomeness, if you will. It revolutionizes not only urban combat, but warfare in its entirety. And also poetry.

Allow me to give you a moment for your brains to stop smoking.

[Stares wistfully into the distance.]

Now that you have some idea of the preponderance of cutting edge research that is discussed at length in our office, let me introduce the next speaker, who will frighten you with outrageously melodramatic nightmare scenarios, entice you with nonexistent but sexy technology, and ease you into a peaceful and meditative state with utopian vistas of the future. Your future.

But only if the money keeps flowing to DARPA. Thank you.

The talks were obviously not the main attraction of the conference, for me. Rather, I enjoyed walking around the exhibit hall and learning about the amazing projects already underway. I especially liked some of the simpler ones, like the sniper rifle equipped with a cross-wind detector, which would indicate where one should aim in order to compensate.

The project I was there to help present is called Vernier, which aims to leverage application communities to detect and control exploits. We had a live demo that showed Vernier successfully detecting, controlling, and recovering from a self-propagating worm as it spread through a community of twenty nodes.

It was strange seeing military officers, including a three-star general complete with military entourage, checking out the latest geeky wares. Then again… there, but for the funding from DARPA, go I.

My paper talk goes well, and I post the manuscript and slides on my Research page. I make a surprise announcement at the end that we are able to release our data; there is much rejoicing.

I decide to skip a portion of the afternoon sessions to be a tourist. I hop into a cab from the Hilton and ask for the Scotch Whiskey Heritage Center. There is a pause. He mumbles something and starts driving. I say again, half-question, half-repetition, “Scotch Whiskey Heritage Center?”

“Scaaatch,” the cabbie retorts, mocking my American pronunciation.

“Scotch whiskey,” I try again in my best Scottish imitation.

“I understood ya’, I joos had ta think about it a wee bit.”

At the booze museum (for what else is it, really?), I meet a Canadian named Dean with whom I have lunch after the tour. We do a flight of scotch drams from the four regions of Scotland: Lowlands, Highlands, Speyside, and the Islands. According to an extremely scientific blind experiment, I can identify two of the four by smell, and all four after tasting. I win a 1 pound bet with him about whether our waitress was Scottish or Irish. Sláinte mhath!

The conference excursion takes us to Stirling Castle, where we have a guided tour followed by champagne in the garden and a banquet in The Great Hall. The meal begins with an Ode to Haggis. A bagpipist, instrument singing, leads in a waiter holding a plate of haggis aloft. The plate is adorned with napkins curled up like the ends of a viking long boat. The musician then recites Burns’s “Address to a Haggis“, in the most exaggerated accent he can muster.

The Edinburgh chapter of my travels is nearly at a close, and I will depart for London shortly after I post this. Pictures forthcoming once I settle in London and move them off my camera. I should really get a flickr account…

As we descend into Heathrow, I see raindrops streaking the windows. Ah, raining in London: how predictably quaint. It was my brilliant idea to take a train from the airport to Edinburgh, rather than flying, so that I could spice up my trip with a pleasant tour of the English countryside. I take a light rail to Paddington, and then the Tube to King’s Cross, from where my train is to depart. The station is packed with people, and most of them look cross, or concerned, or disappointed. I sidle through the crowd to check the light board for the next train to Scotland. It reads thus: Canceled, Canceled, Canceled, Canceled, and so on down the line. The enormous flat-screens flash BBC images of the severe flooding that has washed out roads, and railroad tracks, all up and down the flourishing, green countryside.

“We advise you not to travel to Edinburgh tonight,” a gentleman with the GNER tells me. I advise him, in my turn, that I will be ignoring his well-meant words, wholesale, and seeking passage to my destination. We dance the frustrated-customer-and-powerless-terminal-operator jig for a couple of minutes, and I emerge with a ticket to Edinburgh that will leave “sometime” and take “probably a very long time”. The flooded sections slow the train to a few miles per hour, I learn.

Ticket in hand, I go stand dutifully underneath the giant light board with the throngs of passengers awaiting further instructions. The amber colors flicker and a single train is announced: Edinburgh-bound, Track 5. I bob and weave through the current of people as they rush toward the train (FCFS), my giant suitcase trailing heavily behind me as I curse myself for packing like a woman. I dive into a car and slump down in my seat, exhausted and, I realize as the train pulls away from King’s cross, hungry.

Sufficiently displeased with my condition, airline-rested and fed as I am, I immerse myself in The Forever War, by Joe Haldeman. A revered classic, to be sure, but I also took a writing course from Joe at MIT and felt ashamed to have never read anything he wrote. I finish the book on the train, pleased with the experience. The rain takes a break to allow the summer sun to blaze down through my west-facing window, driving me into a sweat and completing the trifecta. I stare out the window, pointedly.

Soon, though, the clouds roll over the sun again and food service sates my animal hunger. My status downgraded to whiny, I write this post. The train ride takes about seven hours. Total travel time to Edinburgh from home is roughly 24 hours, subjective time.

The countryside really is quite lovely. Speckled with white sheep and rising near Edinburgh into seaside cliffs and crumbling stone walls. Rolling, green, and well-watered.

At least three of the trips (Edinburgh, DC, and Anaheim) involve me giving a talk of some form, which means I have to convincingly feign cognizance. Sean is tying the knot in Portland, which will either make him a married man or qualify him to become an eagle scout. Or a sailor. Vegas is the MIT Pi Reunion, roughly 3.14 years after our graduation; I’ve got a room at the Wynn and tickets to Cirque. Vermont is a revivification of an old tradition, except in a better house and with people who care about each others well-being. I haven’t seen my family in a solid while, so I’ll be stopping there afterwards to raid the fridge and reluctantly (but with secret glee) accept numerous hugs.

Preparations for Burning Man have been ongoing for months now, beginning with the building of a 40′ diameter geodesic dome out of metal conduit piping (our trial assembly). The current projects involve making a cover for the dome, so that we can live inside of it happily, and designing the art car, which will apparently have wings. I joined up with a camp called DeMaTerial, which has gone to Burning Man before and includes several of my friends. I’m a playa virgin, so this will be a new experience.

I’ve been making an effort to leave my comfort zone whenever possible. I think it helps me grow as a person and inspires new ideas. Somewhere between Scotch tasting in the Scottish highlands and living in the Nevada desert in a colorful hemisphere of pipes and hotel sheets, I ought to be planted firmly outside of that comfort zone.

I hope it inspires more than just discomfort.

This past weekend included Peter’s Australia Day party, during which time I ate copious amounts of guacamole, smoked salmon, and fairy bread. I know what you’re thinking: “Adam, you promised to fix your photo galleries and still haven’t done it. What gives?” Right after that, you’re thinking, “What the deuce is fairy bread?” It’s a popular dish in Australia, you uncultured buffoon. Now, bring me my brandy and sprinkles.

Before that, I spent a weekend at Sierra near Lake Tahoe, learning to snowboard. From what I now know of snowboarding, and from what Colin has told me about abusive homosexual relationships, I would say that snowboarding is a bit like an abusive homosexual relationship. You take a severe beating, convince yourself that its worth it and that you are happy, and at the end of the day your ass is killing you. There will be nonpornographic photos of the snowboarding and gorgeous Tahoe scenery posted as soon as I keep my word and fix the galleries. I hope this little metaphor didn’t offend anyone. Except Colin, obviously. I think it’s safe to say his panties will be considerably bunched.

I’ll leave you with today’s political gem. As you are aware, Hamas recently won in landslide Palestinian elections. These well-known supporters of terrorism were elected democratically, so you’d think the Bush administration would be thrilled. After all, bringing democracy to the Middle East was the reason for invading Iraq, right? Well, no, it was WMDs. Remember when Saddam was an imminent threat? Neither does the administration. Suddenly, they went from “we must fight for peace” to “fighting for peace makes no sense”:

“You cannot be on one hand dedicated to peace and on the other dedicated to violence. Those two things are irreconcilable.” – Condoleezza Rice, January 30th, 2006.

“I just want you to know that, when we talk about war, we’re really talking about peace.” – George W. Bush, June 18, 2002.

The last few weeks have been a bit surreal, actually. Aside from taking 13 hours of exams, I also meditated with the Dalai Llama while he visited Stanford, met Orkut (of orkut.com fame) at a bar in Mountain View (I believe my companions were more excited about this than I was, but that’s another story), celebrated International Jane Weekend, and a number of other minor adventures I’ll have to save for another time (because one of them is a surprise… shhhh). I was very impressed by the Dalai Llama, by the way. He was friendly, good-natured, intelligent, and, for someone whose English is by his own admission “a disgrace,” quite well-spoken. Had he not been selected at age two to become the religious leader of a third of a billion people, he said he would like to have been an engineer. His answers to audience questions (including one about genetic engineering) were structured with a respect for science, logic, and practical concerns. He talked about how, as a young man, he was obliged to memorize an ancient Buddhist text that exhaustively accounted for the movement and position of the heavens in a way that was meticulous, precise, and “completely wrong.” To paraphrase his conclusion, there was no sense ignoring what the world was telling us; blindly following the words of a religious text despite the evidence of your own senses is silly. He laughed heartily, and always a little longer than the situation merited. The world could use more religious men like him.

More tidbits: I’ve officially aligned with a research advisor, which roughly means that my bribe check cleared. But seriously, Alex Aiken has agreed to guide me through the doctoral jungle. From my few interactions with him so far, he lives up to his stellar reputation. I am very excited for the opportunity to work with him. I have temporarily put him on hold, however, to work on some residual topics related to my Master’s thesis. In particular, I am preparing a paper for a December 9th deadline. I am also working on a post about torture; keep an eye out for that, because I know you have nothing better to do. Tomorrow, I fly to Boston to spend the week with my girlfriend (Amanda-face!) and family for Thanksgiving. Can I get a “hell, yeah”?

I knew that I could. Time for me to retire now, and become a duck.