3 things you should never do in space software

My mobile rang...

“Hi, Slavo. We have some problems with the satellite.” said my team lead.

“What happened?” I asked.

“This morning, it flew through the South-Atlantic Anomaly, and part of it stopped responding.”

My mind started buzzing. What may have happened? What does telemetry say? Do we have any telemetry at all? Can a simple reset save the day?

It was late Friday afternoon in July of 2017. The third week of SkCube-1 satellite in-orbit.

The point where your experience starts to build

It’s this kind of situation that will prove your satellite software design. Its robustness. Its flexibility. Its recovery scenarios and many other things.

From the experience above and from many other missions where I participated as a software developer, here are three things you should NEVER do in your space code.

#1 Do not build in too many restrictions

Space is hard enough to make your software stubborn. If you build in too many functional restrictions and data range checks, it may work for nominal situations. But what about failure scenarios that are so common in space missions?

A good example where out-of-range operations saved the mission was the all-famous Hubble Space Telescope. Its lens didn’t have a perfect shape from the beginning. Operating software out of normal ranges for image processing compensated for the faulty shape of the lens, and scientists could get their images.

There are many other scenarios where space probes were lost because the software didn’t allow operators to write out-of-range configurations that would save the mission.

To reflect it into your space C code – it may be as simple as null pointers checks. Do the null pointer checks, but do not switch off the device if the null pointer occurs. Have a recovery scenario for that case. Notify the operator of the failure in your telemetry, but design the recovery so that your system will survive the mission. That’s why we call most of the space systems mission-critical :-).

#2 Do not save on telemetry

Today’s software development community is all about savings. Savings on documentation, savings on proper fault handling, savings everywhere. One area where you should definitely not save money and time is telemetry reporting.

There is no one in the orbit to see whether the LED blinks on your system or whether a log has been created. It’s behind the Point of No Return. Unless you get a meaningful piece of telemetry down to Ground Station, you have no clue how your system is doing. And that is a scary feeling – trust me. You have invested too much of your life and money into developing it.

Make your telemetry shorter for periodic reports and more detailed for on-demand reports. But invest in both. Invest into telemetry at the different levels – simple GPIO line for system life-sign, simple ping over the data bus, more extensive CCSDS / ECSS PUS telemetry with the health and run-time information.

You will use your telemetry not only for failure scenarios but also for the development of the better second generation of your space system.

#3 Do not omit periodic reset

When our SkCube-1 satellite flew through South-Atlantic Anomaly on that sad day in 2010 – all we needed was a master (hardware) reset line. It was simply not there. The software reset feature didn’t help because we didn’t have a connection to some parts of the satellite over the databus.

Having an automated periodic reset is a fantastic design feature. It not only helps in the failure scenarios but also helps to mitigate software failures that only manifest after a long period of running your system, e.g., when the value of incrementing counter wrap-around after days or weeks of running.

Make sure that the automatic reset does not happen during your critical data processing. But make sure the periodic auto-reset it’s there. Count how many times the reset happened and make it a part of your telemetry.

There are many other points to make about space software, but for now, you can start with the three suggestions above and see how your design improves.

Slavo

contact@12gfs.com