RT-HIL Framework Documentation

Ryan C. Underwood

Abstract

An open source framework for real-time hardware-in-the-loop simulation.


Table of Contents

System Software Requirements
Required Libraries
Utilities
TODO Items
Distribution and licensing
Links

System Software Requirements

In order to be a soft real time simulation, the simulation requires 1 logical processor for the HAL, 1 processor for the simulation process, and 1 processor for the user interface and any other subprocesses. If you implement additional processes such as controller threads that share a lock with the HAL, simulation, or any process that shares a lock with those processes, those processes (referred to as critical processes) will need 1 additional logical CPU each. A logical CPU can be a dual core CPU, or it can be a hyperthreaded Pentium 4, and the speed of the CPU does not matter.

What is important is that the time that the simulation waits on a lock is bounded. This can only be accomplished by ensuring that all critical processes always make progress while holding shared locks.

This implies that any process sharing a lock with a critical process itself becomes a critical process.

It also means that critical processes must never be scheduled out while holding a lock, which can be accomplished with process pinning (sched_setaffinity(2) in Linux). If a critical process is scheduled out while not holding a lock, then that process's result will be delayed, but the rest of the simulation can continue. If a critical process is scheduled out while holding a shared lock, however, the amount by which the lock delays the rest of the simulation is unbounded - unless the scheduler guarantees fair time slicing (a typical non-RTOS scheduler does not).

Critical processes must also be shielded from hardware interrupts (for example, accomplished using the shield(1) utility on Red Hawk Linux from Concurrent). If a hardware device produced frequent enough hardware interrupts, a critical process could be stalled.

Note that the simulation driver program can optionally log data from the simulation. If this logging is performed, the simulation cannot guarantee a real time response, because the operating system must be used for I/O. This rule goes for any other operation performed during the simulation loop; any trap to a non-real-time operating system function in a critical process invalidates the real time guarantee of the system. Examples of operations that could trap to the operating system include file I/O, management of dynamic memory, or resource locking and synchronization.

Sometimes operating system access can be very hidden inside libraries. For example, the C library file I/O functions sometimes, but not always, call the operating system to perform the I/O. The same goes for C heap allocation functions.

If you are having mysterious real-time performance problems, ensure that no critical process ever calls the operating system unless absolutely necessary. There are a few ways to audit the simulation for operating system calls. You can use strace(1), logging the output to a file; every operating system call will be logged therein. Another approach is to use a disassembler on the compiled object file, as in objdump -D simcore.o, and examine the output. On IA-32 Linux, the operating system is invoked by an int 0x80 instruction. Note that you will have to audit all libraries that are called from inside the simulation loop in the same fashion.

Required Libraries

In order to build the simulation GUI, the GTK+ 2 development libraries and the Glade 2 development libraries are required.

Utilities

  • unnice.c - Nices the process to -10

  • ipcrmall - attempts to remove all leftover SysV IPC objects

TODO Items

Simulation:

  • Figure out how to make the simulation core real-time with GCC.

  • Wrapper for system-specific set affinity call (autoconf).

Distribution and licensing

The RT-HIL framework can be downloaded from the Sourceforge project page. Since it is an alpha quality project, the files are only available via Subversion access.

Unlike other so-called "open" real-time HIL platforms which provide no source code and only can run on marketing-selected platforms, RT-HIL employs a liberal copying and redistribution license which can be found in the COPYING file in the distribution.

Links