Fixing Spirit, from 125 million miles away

[Editor's note: The headline of the original version misstated the distance from Earth to Mars.]

At the moment, it seemed a glass-half-full grasp at optimism. Over a 24-hour period, America's model rover had gone completely mad. Spirit had astonished scientists with its flawless performance for weeks, but when it was asked to send scientific data last Wednesday, it spat out only jibberish. When asked to go to sleep, Spirit refused with the obstinacy of an 8-year-old at bedtime.

Amid the mounting clatter of despair and doom, though, mission manager Firouz Naderi paused as he addressed the media that day: "Gosh, don't give up on us yet."

It was the briefest look into a world unseen in all the recent photos of red rocks or craters of dun dust - yet in many ways no less remarkable. It was a glimpse into the can-do world of the engineers themselves, where disasters can be divined from endless strings of ones and zeroes, and cosmic fix-it jobs are performed from millions of miles away through keystrokes and creativity.

If the Russians became the gurus of space improvisation on manned missions, holding their Mir space station together with little more than duct tape and paper clips, then the Americans have become the Houdinis of robotic missions into the Great Beyond. Indeed, Spirit now appears to be on the road to an almost full recovery in two to three weeks.

But on numerous other missions as well, unmanned American spacecraft and their engineers have established a legacy of overcoming the unexpected - from jammed camera mounts to broken antennas - yet still returning groundbreaking science.

"It is a very common thread," says James Oberg, a former member of NASA mission control and now a commentator. "These spacecraft are built with a robustness that allows people to project their ingenuity millions of miles into space."

For most of the mission, the rover's robust design has been almost an afterthought. After Spirit arrived Jan. 4, it performed just as scientists had hoped it would, rolling off its lander, snapping pictures of Gusev Crater, and sampling the dust for signs that water was once there.

Halfway around Mars, its twin also landed perfectly Saturday night, coming to rest in a small crater on the dark and Gothic fields of Meridiani. The rover Opportunity should move off its lander in a week or so.

But one week ago, the wonder of scientific discovery was abruptly displaced by worldwide scrutiny of every circuit on the Spirit rover. One minute, the scientists' jolly robot geologist was preparing to use one of its sensors. The next, it had transformed into a recluse, crouched on its patch of Martian soil and offering up only jumbled strings of useless information.

The story of how engineers pieced together the fragmented clues of a mystery some 125 million miles away - and plan to fix it - is a parable of space-age derring-do in which bravado is measured in long hours of persistence and translated into lines of computer code.

At first, it was simply a process of engineers trying to concoct any possible scenario for the symptoms they saw. "It's an exercise in imagination," says Julie Townsend, an engineer at the Jet Propulsion Laboratory, which runs the two rovers. "You try to imagine all the sorts of things that could cause the signatures that you're seeing."

For those working to unravel Spirit's babblings, it meant hours in the "sandbox" - the test bed where engineers work with a rover replica, pushing and prodding it to see what happens.

And for the most part, the most telling clues come not from any movement or action obvious to the untrained eye. Instead, they come in the lines of code that pop up on a computer screen every time the rover follows a command.

In a programming language that distills actions down to a jumble of words, brackets, and keyboard symbols, the rovers reveal their secrets, showing what parts of the rover were used to complete which commands.

With little more to go on than random bursts of data, Spirit's engineers started their search wide. Eventually, they narrowed the primary culprit down to the rover's flash memory, which stores data while the rover is sleeping.

The rover, engineers discovered, sensed some flaw in its flash memory, and when it couldn't resolve it, it shut down and tried to restart. In three days, it tried - unsuccessfully - to reboot itself more than 60 times.

Yet even in this apparent frustration is the kernel of what has made many of America's robotic missions so resilient. Spirit's ability to keep itself alive in a "safe mode" after sensing a fatal flaw gave engineers the time needed to diagnose the problem and devise a solution. "Russian probes never had this ability," says Dr. Oberg. "They broke and they died."

Things, after all, inevitably go wrong. On Voyager 2, for instance, a part of the camera assembly broke, meaning the camera could not pan correctly. Engineers trained it to move more modestly.

On Galileo, the large antenna designed to transmit scientific data never unfurled, so scientists reprogrammed an onboard tape recorder to store images until they could be sent on a smaller antenna. When the tape jammed, they taught the rewind mechanism to stop before it reached the sticky portion of the tape.

With Spirit, engineers have jury-rigged a "cripple mode" in which Spirit can reboot without using the flash memory. While Spirit can't perform all its functions in cripple mode, it can respond to commands and go to sleep when ordered, giving engineers more time and help in isolating Spirit's problem.

It's a process that could take weeks as they probe all the relevant lines of code to pinpoint exactly where and how the problem started - without triggering it again. If it's a flaw in the software - as is expected - engineers can eventually send up a new set of codes to replace the corrupt ones. If it's a problem caused by something physical in the machine, engineers can program the rover to avoid those sections.

Either way, the process is one that engineers have come to know well. And though Dr. Townsend knows her job is to deliver a perfect rover, there's something of a Martian car mechanic in her nature, and she can't help but relish the chance to tinker under the hood.

"A perfect mission is great, but we engineers love jumping on stuff like this," she says. "It's exciting to figure it out and make the craft work again."