Behind the hum of a power-grid: human choices

When the full fury of power-grid chaos came rolling through America's Northeast last Thursday, leaving millions without power, Vermont didn't go black.

Part of the reason was fast-acting sensors that sealed off the state from the rest of the Eastern grid. But another factor was the decades-long experience - and rapid-fire ingenuity - of electrical engineers like Grant Adams.

A tall, bespectacled man - who took a 50 percent pay cut to do the job he loves - Mr. Adams saw the fluorescent lights in his utility-company office dim on Thursday. He then sprinted into the control room - which was buzzing with 300 alarms - to help stabilize the state's rapidly deteriorating grid. "We were right on the edge," he says.

For residents of the Green Mountain State, the tale ultimately had a happy ending that day, but it also highlights how central the human factor is in power-grid management. It appears increasingly likely that human actions - or lack thereof - will be at least partly blamed for the biggest blackout in US history. And while billions of dollars may be needed to modernize America's electricity infrastructure, it's also clear that less-costly issues of human communication and decisionmaking may be just as crucial in preventing future blackouts.

Running power grids "requires someone looking at screens who has a lot of knowledge and judgment - and can act," says Robert Burns, a electricity expert at Ohio State University. Communication between engineers is also key. "Only so much of the communication can be between computers. To some extent people have to be able to talk to each other," including before any emergencies, he says. "The what-if scenarios have to be talked out in advance."

That's just the kind of thing Adams loves to do. For years, he worked the night shift in the control center, a secure room in the basement of Central Vermont Public Service, from which engineers control much of the grid. Along one wall stands an 8-foot-by-30-foot electronic map of the state's power system. With thousands of interconnected lines and lights, it looks like giant, lit-up maze from a child's activity book.

During many late nights, Adams would stare at the map and ask himself, "What if that substation went down - how would I compensate?" And what if a particular transformer blew? And on and on. "About 3 a.m. you start to get a little squirrely," he says with a smile. These days, engineers keep a treadmill in the room as a late-night diversion.

Managing a power grid is a supreme balancing act. It's a bit like controlling a small wave pool with hundreds of wave machines around its edge. If all the machines are operating at about the same level, there's a basic equilibrium in the water.

But if some machines are churning faster or harder than others, rogue waves can form - and start swamping the entire system. The wilder the waves, the more power plants start disconnecting themselves from the system - to avoid damaging their equipment. But that only complicates the problem. Some wave-making power is needed to tame big waves and bring the pool back into equilibrium.

Last Thursday, as Adams was sitting in his office, the nearby power grids started to be swamped. To stabilize themselves, they apparently tried to suck power in from outside sources, including Vermont.

To switch metaphors, it's as if the other systems became black holes, pulling all available energy toward them. Since electricity travels at virtually the speed of light, no human could have reacted quickly enough. But Vermont's sensors instantly rebelled, walling off the state.

The sensors' action was a godsend. But it also left no outside power flowing into the state. That sent voltage levels - a key measurement of the system's stability - falling fast. Like an airplane losing speed, Vermont's grid risked stalling and crashing. Now the humans had to stabilize things.

Adams and the other engineers knew they had to get in quick contact with Vermont's other utilities. Fortunately, the control room has several "bat phones," including a red instant hotline to the Vermont Electric Power Company, which runs the state's big transmission lines.

Adams knew his counterparts there would want extra power to boost voltage. He knew he could fire up a nearby gas-turbine generator quickly. "Do you want it?" he barked into the red phone. "Yep," came the reply. Done.

Such communication and coordination is key. In fact, a report prepared in May by the national industry watchdog targeted the Midwest as especially susceptible to lack of cooperation.

"There is a continuing need for the reliability coordinators, transmission planners and operators to communicate and coordinate their actions to preserve the continued reliability of the [Midwest] system," said the report by the North American Electric Reliability Council. It continued, ominously, "As long as transmission limitations are identified and available operating procedures are implemented ... no cascading events are anticipated." Then just such an event occurred.

Now communication is a central focus of the probe. Some Ohio operators reportedly may have known of troubles in the system for at least an hour before the ultimate crash. If they did - and had alerted other nearby operators - it may have prevented the blackout.

One other key difference: Regional power coordinators in the Eastern US have more authority to demand cooperation between local utilities than their Midwestern counterparts. More centralized authority may help those systems have tighter response times.

In Vermont, Adams and others drew on everything they had. They cranked up power generators, including a dam. It helped that three weeks of steady rain had raised reservoir levels. But just to be sure, they sent operators to each plant and substation - ready to manually operate them if needed.

The crowning moment came as they watched the voltage creep back up. "We've got 108 volts," Adams counted, "112 volts, 116 volts...." It meant that Vermont had avoided - if just barely - the great blackout of 2003.

• Amanda Paulson contributed.