Mitigating and Recovering from Radiation Induced Faults in Non-Hardened Spacecraft Flash Memory

The European Space Agency (ESA)’s OPS-SAT-1 mission pioneered the application of edge computing for onboard flight operations. Its objective is to upload and test new cutting edge on-board software in space on a daily basis. This involves a much more intense use of the on-board control computer compared to traditional missions. After several years, radiation damage occurred that would have normally led to mission end. The mission control team was able to restore the core payload functionality by exploiting the reconfigurability of the computing platform. The Satellite’s Experimental Processing Platform (SEPP) is equipped with an 800MHz processor and a reconfigurable Field Programmable Gate Array (FPGA). The SEPP is not radiation-hardened and runs embedded Linux. Since the beginning of 2022, several instances of file corruption in the SEPP’s non-volatile embedded MultiMediaCard (eMMC) memory, physically a NAND flash, were observed. Earlier instances could be rectified with conceptually simple procedures like rebuilding the filesystem. However, an event in June 2022 resulted in a total loss of communication with the payload. A previously unused access route via UART had to be made active before the problem could be investigated. The team then narrowed down the origin of the problem and revealed that the eMMC was not totally lost but that writing to it could crash the controller. Novel ways of avoiding memory writes had to be found to continue the mission. This approach ushered in an entirely new fully volatile mode of SEPP operations. At the same time a recovery strategy, in case the eMMC memory completely failed, also had to be designed and implemented. This entailed storage optimisation and added redundancy. This paper delves into the novel eMMC defect and sheds light on the resourceful strategies that were employed for its recovery. It outlines various approaches and provides a comparative analysis to underscore the associated trade-offs. The SEPP was custom built for OPSSAT-1 at a time when commercial off-the-shelf (COTS) counterparts did not exist. Since then, similar payloads have become readily available COTS products and their adoption in small satellite missions has sky-rocketed. This paper aims to share the Mission Control Team (MCT)’s experiences with managing and recovering from radiation induced memory faults in spacebound edge computers and propose guidelines for future small satellite missions carrying similar equipment.