On the evening of January 3 this year, Spirit, the first of the two Mars Exploration Rovers, landed successfully on the surface of Mars. Back on earth, Kanna Rajan was anxiously waiting for the 385-pound Rover Spirit to execute its first instructions; not to mention he had another such exercise of precise execution awaiting a fortnight later. Rajan’s anxiousness had a reason.

As senior scientist of NASA’s Ames Research Center in Moffett Field, CA, Rajan and his team have labored more than three years to design an activity-planning tool called MAPGEN (Mixed-Initiative Activity Plan GENerator) that will send instructions to the Rover to perform hundreds of maneuvers and scientific tasks.

Each Rover—located at scientifically distinct sites—has an operational lifetime of about 90 Martian days (sols). NASA’s objective is to maximize the science return from both the rovers within each’s lifetime. So making best use of the scientific instruments—within limited resources—is a crucial goal of the mission. Rajan’s tool helps to increase the amount of science data returned from the rovers. How is it achieved?

Sleep-time Frenzy
Sol-operations personnel on Earth receive telemetry from the Rovers and on the basis of the downloaded data, the science team constructs, verifies, and uplinks a detailed sequence of commands for the next sol to the Rovers. “Between the time the Rover goes to sleep and wakes up next morning, Martian scientists are working frantically,” notes Rajan. “They formulate a viable sequence that satisfies the mission goals within tight deadlines.” To help address this critical need, MAPGEN automatically generates plans and schedules for science and associated engineering activities; assists in hypothesis testing—what-if analysis on various scenarios; supports plan editing; analyzes resource usage; and performs constraint enforcement and maintenance.

Once the MAPGEN finishes sequencing the instructions, the plan outputs to a file for use in the next up-link process phase.

The Conflict-free plan
Apart from generating a plan, MAPGEN actively enforces flight and mission rules with conflict resolution—forbidden activity, overlaps or resource violations. In fact, Rajan’s effort from the beginning has been towards producing a conflict-free plan.

There are approximately six hours of sunlight on Mars in a sol, and the Rovers are solar-based explorers. Temperatures during the day are -72C—a major concern with the possibility of running down the batteries by overworking the robots. “It is very important to leave sufficient battery power to conduct engineering activities, and to ensure that the robot is safe,” he says, “If we overwork the Rover, it might overheat, which would be disastrous. Without enough heat, it can become too cold and the Rover could freeze. Our technology is driven to optimize so that we get as much science response as possible.”

In light of all the constraints of the scientists, executing the maneuvers by the Rovers is a challenge. For example, MAPGEN automatically ensures that a PanCam picture is not taken while the rover is moving, since that may not only drain batteries, but could also blur pictures. “When we decide on a particular kind of science, for example, to use the Rover to do an atmospheric sky campaign (sampling the Martian sky every 20 minutes or so for about two hours), there are only certain windows for the execution. It doesn’t make sense to do such a campaign at night; but more so when the sun is either rising or setting. It is here that MAPGEN’s strength comes into the picture—it produces an initiation plan automatically,” explains Rajan.

Developing the MAPGEN
The proof of concept phase started with agreement of support from the Mars Exploration Rovers (MER) mission operations and science representative.

The MAPGEN team set out on the development using C++. They leveraged the strengths of APGEN—a multi-mission tool for a number of flight projects—and the Planner, a planning and scheduling system, which provided a comprehensive tool for planning and scheduling.

There were several delivery phases that included new modules. Ops/science panel reviewed each delivery phase, providing new directions or feedback.

For example, the first version of MAPGEN was designed to capture only flight/mission rules in its models. However, for scientists, capturing science-intent was crucial for the effectiveness of MER. The scientists started paying more attention to the commanding process and the MAPGEN team was asked to build a science-intent capture tool. The scientists suggested an automated planner that could use an electronic version of the intent information. This led to the development of a Constraint Editor (CE) that was incorporated into the MAPGEN. The MAPGEN team leveraged Planner’s database to enforce these preferences/constraints.

During the implementation stages, MAPGEN was rebuffed when the team tried squeezing it into the critical plan integration process. The MAPGEN team had to make sure that their software interfaced with other softwares used in the mission. “Though it started off chaotically, we kept polishing it, and as we went along, we discovered new capabilities in the tool and refined the MAPGEN,” says Rajan.

Testing
Ultimately scientists at the Pasadena JPL would use the tools these tools. Rajan and his team were at it till August 2003, leavig very little time to train JPL’s scientists on the tools. Consequently, in September, when the first of the timed Operational Readiness was conducted, Rajan’s team failed to generate a plan in the half-hour time limit.

“One of the reasons for failure was the lack of training. People who were supposed to use the tools had no idea on how to handle MAPGEN,” says Rajan as he stresses on the importance of training.

However, the MER team was clear that a “waterfall” approach of building plans didn’t work and parallelism was needed to build the command load. Finally a Manual Vs. Automated “bake off” was proposed, and some of the JPL scientists underwent “tools training.” In the final test MAPGEN took 30 minutes to generate a plan, whereas it took four hours manually.

Martian Bugs
Once the software was developed and implemented, it had to watch out for bugs that could crop up during execution. However, a bug was the last thing expected in a mission critical environment. Within two weeks after the Spirit Rover landed on Mars, one of the systems running MAPGEN crashed. After frantic analysis and testing, it became apparent that there was no technical fault in the software. It was more of an environment or compiler issue.

Fixing a bug is easy. All that one needs to do is mirror the operational environment in which the bug occurred, fix the bug in the mirror environment and finally activate the mirror environment so that the other users continue using without any impact.

But it isn’t that easy in a mission-control operation, especially at NASA, which is tightly regulated. The software developers have to appear before a panel—that includes the mission manager and the Rover management team to convince them about the impact-risk investment.
“We would need to have a complete plan even if it was to change a single character,” quips Rajan. It is this stringent change-control process that makes software developers at NASA more disciplined in their approach to developing a robust product.

The challenge
The overall cost for Mars Exploration Rover (MER) project was about $820 million. “A simple math would suggest $820 million for two Rovers for 90 Martian days would mean $4.5 million every sol. So if MAPGEN fails to execute a single day, sub-optimal commanding to the Rover will not be possible. This would mean a loss of $4.5 million for NASA!” observes Rajan. He adds, “My team’s mission was to make sure we don’t mess it up, and this is exactly the type of kind of challenge that I like to take on.”
Three years ago, when Rajan got on to this project, he asked himself, “Where do we go where no one has gone before?”

Now he knows.