Linux Real Time Patch Review - Vanilla vs. RT patch comparison

Linux Real Time Patch Review - Vanilla vs. RT patch comparison

Hardly any standard operating system kernel is hard real time capable. At best, they're soft real time capable, with latencies up to milliseconds.

RTAI and Xenomai provide hard real time, which means that those operating systems (inserted between the hardware and Linux) can assure interrupt response times in the microsecond range. Not so with that standard Linux kernel. The 2.6 kernel - in comparison to the 2.4 kernel - has been much improved in regard to latencies, but still has some bottlenecks. Ingo Molnar provides a patch to compensate for those bottlenecks.
Ingo Molnar RealTime Preempt Patch

This patch improves the responsiveness of the kernel dramatically.
The tests below have been conducted with a tool provided by Andrew Morton. Within the amlat package, there is a tool called "realfeel" which also works on 2.6 kernels, despite the "amlat" tool which, well, doesn't even work on 2.4 kernels on recent systems.
Linux scheduling latency (Andrew Morton)

Here is a small tar.gz which provides just the "realfeel" tool from the AKPM package and the famous "hackbench" (tweaked to acually compile):
captest.tar.gz (2k)

Compile with "make".

# ./realfeel histogramfilename.hist
1400.038 MHz
secondsPerTick=0.000000
ticksPerSecond=1400037759.748198
2048 Hz
0.451 msec
0.361 msec
-0.004 msec
-0.029 msec
-0.146 msec
-0.205 msec

For the hackbench stress test call it like this:

# ./hackbench 100

Adjust the parameter - WARNING: if the parameter is too high, your system might freeze!

If you get something like this with "realfeel":

ioctl(RTC_IRQP_SET) failed: Invalid argument

do this:

# rmmod genrtc
# modprobe rtc

This removes the module "genrtc" and loads the "rtc" (real time clock driver).

In any case, you should unload the module "genrtc" and load the module "rtc".

THE RESULTS

The result with a vanilla 2.6.15 kernel:
(first column is in milliseconds; second column is the number of RTC (real time clock) interrupts received)

0.0 54410
0.1 32
0.2 8
0.3 5
0.4 3
0.5 1
0.6 1
0.8 1
1.0 1
1.4 1
1.5 1
2.0 1
3.7 1

As you can see, the max. latency of the RTC interrupt was about 3.7 milliseconds. (UPDATE: with "hackbench" I've even got more than ONE SECOND!!!)

Results with the rt16 RT patch:

0.0 308821
0.1 832
0.2 104

As you can see, the max. latency was just about 200 microseconds. (UPDATE: with "hackbench" I got a max. of about 400us)

If there is no or almost no load on your system, take into account that SMI (System Maintenance Interrupts on e.g. Intel 82845 845 (Brookdale) motherboards) can and will produce latencies of > 200us. Also see: Xenomai: High latencies with SMI not disabled

CONCLUSIONS:

If you need to have a low latency system for audio, music or video processing, the Molnar low latency patch might be the right thing for you.
But still, if you need hard real time performance, where latencies must not be higher than a few tens of microseconds, you'll better off with RTAI or XENOMAI.

Appendix:

Here are a few stress testing routines proven to be good for determining the worst case latencies.

# dd if=/dev/zero of=bigfile bs=1024000 count=1024

# ping -l 100000 -q -s 10 -f localhost

# ping -f -s 1400 & dd if=/dev/urandom of=/dev/null & ls -lR /

# - a=0; while "true"; do cat /proc/interrupts; a=`expr $a + 1`; echo $a; done

# while :; do dd if=/dev/zero of=/tmp/foo bs=1M count=500; sync; rm /tmp/foo; done

If you want to combine some of those:

#!/bin/sh
dd if=/dev/zero of=bigfile1 bs=1024000 count=1000 &
ping -l 100000 -q -s 10 -f localhost &
du / &

-or one from Paolo Mantegazza (RTAI.org)

a=0; while "true"; do dd if=/dev/hda1 of=/dev/null bs=1M count=1000;
 a=`expr $a + 1`; date; echo $a; done

(this is just one line!)

If you use "hackbench" you can try this:

#!/bin/sh
while true; do dd if=/dev/zero of=bigfile bs=1024000 count=1024; done &
while true; do killall hackbench; sleep 5; done &
while true; do ./hackbench 20; done &
ping -l 100000 -q -s 10 -f localhost &
du / &

Most hardcore test
Philippe Gerum (Ingo Molnar):
http://lkml.org/lkml/2005/6/22/347

while true; do dd if=/dev/zero of=bigfile bs=1024000 count=1024; done &
while true; do killall hackbench; sleep 5; done &
while true; do ./hackbench 20; done &
( cd ltp-full-20040707; while true; do ./runalltests.sh -x 40; done; ) &
ping -l 100000 -q -s 10 -f localhost &
du / &

And last but not least, here is a process listing of the processes on a RT preempt patch patched kernel and a vanilla kernel just after booting on a debian sarge system ...
... to illustrate the differences between the rt-patch and vanilla kernel system:

2.6.15-rt16

# ps -A
 PID TTY          TIME CMD
   1 ?        00:00:00 init
   2 ?        00:00:00 softirq-high/0
   3 ?        00:00:00 softirq-timer/0
   4 ?        00:00:00 softirq-net-tx/
   5 ?        00:00:00 softirq-net-rx/
   6 ?        00:00:00 softirq-scsi/0
   7 ?        00:00:00 softirq-tasklet
   8 ?        00:00:00 desched/0
   9 ?        00:00:03 events/0
  10 ?        00:00:00 khelper
  11 ?        00:00:00 kthread
  12 ?        00:00:00 kblockd/0
  13 ?        00:00:00 pdflush
  14 ?        00:00:00 pdflush
  16 ?        00:00:00 aio/0
  15 ?        00:00:00 kswapd0
  17 ?        00:00:00 kseriod
  18 ?        00:00:00 IRQ 12
  19 ?        00:00:00 IRQ 14
  20 ?        00:00:00 kjournald
  21 ?        00:00:00 IRQ 1
 378 ?        00:00:00 IRQ 19
 389 ?        00:00:00 IRQ 4
 392 ?        00:00:00 IRQ 3
 544 ?        00:00:00 syslogd
 547 ?        00:00:00 klogd
 579 ?        00:00:00 exim4
 584 ?        00:00:00 inetd
 592 ?        00:00:00 sshd
 597 ?        00:00:00 atd
 600 ?        00:00:00 cron
 606 tty1     00:00:00 getty
 607 tty2     00:00:00 getty
 608 tty3     00:00:00 getty
 609 tty4     00:00:00 getty
 610 tty5     00:00:00 getty
 611 tty6     00:00:00 getty
 612 ?        00:00:00 sshd
 614 pts/0    00:00:00 bash
 618 pts/0    00:00:00 ps

2.6.15 vanilla

# ps -A
 PID TTY          TIME CMD
   1 ?        00:00:00 init
   2 ?        00:00:00 ksoftirqd/0
   3 ?        00:00:04 events/0
   4 ?        00:00:00 khelper
   5 ?        00:00:00 kthread
   6 ?        00:00:00 kblockd/0
   7 ?        00:00:00 pdflush
   8 ?        00:00:00 pdflush
  10 ?        00:00:00 aio/0
   9 ?        00:00:00 kswapd0
  11 ?        00:00:00 kseriod
  12 ?        00:00:00 kjournald
 532 ?        00:00:00 syslogd
 535 ?        00:00:00 klogd
 567 ?        00:00:00 exim4
 572 ?        00:00:00 inetd
 580 ?        00:00:00 sshd
 585 ?        00:00:00 atd
 588 ?        00:00:00 cron
 594 tty1     00:00:00 getty
 595 tty2     00:00:00 getty
 596 tty3     00:00:00 getty
 597 tty4     00:00:00 getty
 598 tty5     00:00:00 getty
 599 tty6     00:00:00 getty
 602 ?        00:00:00 sshd
 604 pts/0    00:00:00 bash
 609 pts/0    00:00:00 ps

Last-Modified: Thu, 09 Feb 2006 19:14:41 GMT