In this second episode, we explore the drama that comes with flying robots to Mars. We talk with Planetary Society President Dr. Jim Bell and Mars dust expert Dr. Kjartan Kinch about the Spirit Rover and the challenges involved in the mission.
Enjoy our short tale of mystery, frustration and triumph.
Click Here for Transcript
[♪ Gershon Kingsley playing Popcorn ♪]
Dr. Regina Barber DeGraaff: Welcome back to “Spacecraft Chronicles,” where we celebrate past, present, and future space missions. I’m your host: Regina Barber DeGraaff, astrophysicist and movie-lover. Today’s show is about challenges in space exploration. Think Gravity and The Martian. We all know that drama makes great stories. But when you have a problem in space, that’s even better. If this sounds interesting to you, check out the Spark Science review of the film The Martian by Mars rover scientist Dr. Melissa Rice in the episode “The Martian: Film and Science” from January, 2016.
Let’s get back to the mission challenge of today’s show. I interviewed many people from the Mars rover Mastcam Z-team this summer, and asked them the following question: “Can you recall a problem that arose on a Mars mission and how was it solved?” Two NASA scientists told me the same story of the Spirit rover flash memory anomaly. I will let the two men introduce themselves. Note about the audio: I talked to them at two different locations, a VAP reception and a lake shore.
Dr. Jim Bell: I’m Jim Bell. I’m a professor at Arizona State University in the school of Earth and Space exploration. And I’m also president of the planetary society.
Dr. Kjartan Kinch: My name is Kjartan Kinch. And I am from Denmark and I work at the University of Copenhagen in Denmark.
Dr. DeGraaff: After the success of the first Mars rover “Sojourner” in 1997, NASA was eager to continue exploration of Mars geology or “marsology” as Dr. Rice likes to call it. NASA developed the Mars exploration mission, which consisted of two twin rovers: Spirit and Opportunity. They were set to arrive on Mars 3 weeks apart in 2004. Our story begins soon after the first rover bounced onto the Mars surface.
Dr. Bell: After the Spirit rover landed on Mars in January of 2004, we had tested the whole landing process and what would happen afterwards, the pictures we would take, the way the rover would come off that lander and drive onto the surface. And we had tested it with a test rover, with other simulations. And the longest test that we had done, we had simulated 2 weeks, the first 2 weeks of the mission. And we had it down, did it multiple times. So, Spirit lands; the first 2 weeks of the mission, it’s right by the book.
Dr. Kinch: Everything had just been going swimmingly. And suddenly, it became uncommunicative, or it, like, sent these weird, garbled communications. And I was on the science team and I was a junior guy. I was not involved with working with problem. But we got updates and we saw the engineers running around. And it was actually really dangerous for the rover because it wasn’t shutting down when it was supposed to. It was staying awake at night, running down its battery and…
Dr. DeGraaff: It was like any grad student. It was burning the wick at both ends.
Professor: [Laughing.] Yeah. Exactly.
[♪♪♪]
Dr. DeGraaff: The rover was alive, but not transmitting any of the collected data to Earth. Not only was the Spirit not properly operating, there might have been the possibility it was not storing information in its memory, either. Because Spirit was unable to shut down properly, it was also in danger of overheating, as well as running out of power. That power could only be replenished by the rover’s solar array. There were 2 rechargeable lithium ion batteries weighing roughly 16 pounds each to use when the light was limited. But these batteries degrade over time and they are being used more often than anticipated.
Dr. Bell: It went into what it calls a “safe mode.” It hunkers down, buttons itself up and says, “Uhhh. I’m having a problem!” and it uses its emergency transponder to send a help signal. Part of the help signal is, “Here’s the information that’s in my memory” and, “Here’s all the software flags that were set” and all that.
Dr. Kinch: As they worked the problem, they figured out that it was kind of rebooting all the time. It was starting up and then it was sending some communication, and doing it a little bit, and then it was rebooting. And it kept rebooting, rebooting, rebooting.
Dr. DeGraaff: With this constant strain on the power source, soon, Spirit would not be operational at all, and the mission would be a bust.
[♪♪♪]
Keep in mind that Opportunity, Spirit’s twin rover was set to arrive on Mars within days. The team of NASA scientists, engineers, and mathematicians were frantically trying to save Spirit’s life. What if the same issue happened to Opportunity?
Dr. Bell: I never knew that there was an entire field of engineering called fault-tree analysis. So, people who are like detail-oriented and paranoid are perfect for these jobs because you have to basically come up with everything that could possibly go wrong. “So, if this happens…” that’s like one branch of the tree “and then if that happens,” that’s another branch. “And then if it’s a Tuesday and a full moon…” And you make sure that at the end of every branch, you have a leaf with a solution written on it. And there are professionals who do this for a living and a lot of them work for NASA because things go wrong and you don’t want to be caught by surprise.
Dr. DeGraaff: Surprises always happen. And it’s the job of the fault-tree to lay out all scenarios.
Dr. Kinch: After working the problem for a long time, they realized that it was related to some sort of limit on the number of data products, like basically the number of files it was allowed to have on its hard drive. And there was some limit that was huge. And they had never encountered this in testing before. But once they sent it to Mars and it started making little logs, and with everything it was making [inaudible], eventually it hit that limit. And it would start up and it would check “am I over the limit?” and it would like “oh no. I’m over the limit.” And that would lead it to reboot. And it was obviously like kind of a silly thing that it should have that limit. And so they were doing this detective work and they were doing it under real-time pressure.
Dr. Bell: It turned out that, because we had never tested the vehicle for this long, and the computer had never been running for this long, and we hadn’t ever taken this much data in the tests, we were filling up the disk. And there was a bug in the software that we would have found if we tested for 19 days. We tested for 14 days. It was a bug that, if you fill up the flash memory above a certain point, it starts to overwrite the operating system.
It was a freaky bug and everybody, the major mission managers and all that were like “aw man.” Trying to figure out what happened. “What’s going on?” And they found the person who did the fault-tree analysis. And they found this guy and said “okay. Tell me. What’s this switch set at? What’s that variable set at?” You could sort of see him going out on the branches. At one point, it was like “I know what happened.” And we built a back-door to fix that. This one guy saved the entire mission.
[♪♪♪]
Dr. DeGraaff: The drama that created excitement in these missions was actually foretold by the men and women that created these fault trees. The hope is that these challenges and solutions, no matter how stressful or dangerous, will be found on the branches of these trees. Even when the solutions are not prophesied, every mission, with failures both large and small, create more branches and more opportunities for success.
[♪♪♪]
[♪ Gershon Kingsley playing Popcorn ♪]
This episode of “Spacecraft Chronicles” was produced, written, and edited by Suzanne Blais, Lela Jacobs, Robert Clark, and Regina Barber DeGraaff at Western Washington University in Bellingham, Washington. Our song is Popcorn Song by Gerson Kingsley. Special thanks to Planetary Society president Dr. Jim Bell and world expert in Mars dust, Dr. Kjartan Kinch. Also, we would like to think Kay Marie and Planetary Society’s Casey Drier for their support. Join us again next time for another edition of “Spacecraft Chronicles.”
[♪ Gershon Kingsley playing Popcorn ♪]
[End of podcast.]