Open Source Summit + AI_dev Japan 2024

There is a lot of work around how to achieve good real-time on Linux, but not as much on how to simulate faults such as jitter in the system, deadline misses or other faults. Without this it is difficult to test how your application or entire system copes with these problems. As part of work with a number of clients, especially in the safety sphere, questions have come up on how to test processes which rely on real-time scheduling. If we have a way of injecting faults we can reliably test error handling and other mitigations. Mitigations such as throttling, restarting or some measured shutdown of services. We will go through some methods we evaluated for fault injection via both user and kernel space. How existing kernel features can be used and what needs to be done in the way of either configuring or extending kernel features. There will be discussion about how each method works and the comparative merits where overlaps exist. We hope that this can help to promote thinking and improvements on how the scheduler and particularly real-time scheduling is tested under Linux.