> I've recently switched to the latest rt kernel
> available from http://people.centos.org/hughesjr/kernel-rt
> And one of the two problems I'm having with this new kernel
> is that the logs have been flooded with nagios alerts
> which indicate high percentage (50-60) of packet loss
> when pinging localhost. The problem shows up several hours
> after the system reboot.
The problem also shows up with the latest kernel-rt-2.6.24.7-132
and seems to be triggered by scheduled cron jobs. That's probably
the only time when this system might be under any significant load.
Below are the relevant excerpts from /var/log/messages indicating
when the system booted into which kernel and the first 10 ping alerts
after system reboot.
Aug 28 20:03:40 angara kernel: Linux version 2.6.24.7-65.el5rt.centos
Aug 29 02:12:54 PING CRITICAL - Packet loss = 60%, RTA = 0.13 ms
Aug 29 02:13:44 PING WARNING - Packet loss = 44%, RTA = 0.09 ms
Aug 29 02:14:54 PING WARNING - Packet loss = 54%, RTA = 0.08 ms
Aug 29 02:15:45 PING WARNING - Packet loss = 44%, RTA = 0.09 ms
Aug 29 03:10:54 PING CRITICAL - Packet loss = 70%, RTA = 0.11 ms
Aug 29 03:20:46 PING WARNING - Packet loss = 44%, RTA = 0.09 ms
Aug 29 03:30:56 PING CRITICAL - Packet loss = 72%, RTA = 0.09 ms
Aug 29 03:35:45 PING WARNING - Packet loss = 44%, RTA = 0.12 ms
Aug 29 03:40:44 PING OK - Packet loss = 16%, RTA = 0.10 ms
Aug 29 03:45:44 PING WARNING - Packet loss = 50%, RTA = 0.09 ms
...
Aug 29 11:30:56 angara kernel: Linux version 2.6.24.7-65.el5rt.centos
Aug 31 03:43:13 PING WARNING - Packet loss = 28%, RTA = 0.10 ms
Aug 31 03:44:13 PING WARNING - Packet loss = 37%, RTA = 0.10 ms
Aug 31 03:45:23 PING WARNING - Packet loss = 54%, RTA = 0.08 ms
Aug 31 03:46:23 PING CRITICAL - Packet loss = 60%, RTA = 0.12 ms
Aug 31 03:51:13 PING WARNING - Packet loss = 44%, RTA = 0.10 ms
Aug 31 04:31:23 PING CRITICAL - Packet loss = 60%, RTA = 0.09 ms
Aug 31 04:41:14 PING WARNING - Packet loss = 50%, RTA = 0.11 ms
Aug 31 06:16:13 PING OK - Packet loss = 16%, RTA = 0.11 ms
Aug 31 06:21:13 PING WARNING - Packet loss = 44%, RTA = 0.10 ms
Aug 31 06:22:13 PING WARNING - Packet loss = 50%, RTA = 0.10 ms
...
Sep 3 23:44:20 angara kernel: Linux version 2.6.24.7-132.el5.local
Sep 5 04:19:53 PING WARNING - Packet loss = 50%, RTA = 0.14 ms
Sep 5 04:21:01 PING CRITICAL - Packet loss = 60%, RTA = 0.10 ms
Sep 5 04:21:51 PING WARNING - Packet loss = 28%, RTA = 0.12 ms
Sep 5 04:22:56 PING WARNING - Packet loss = 50%, RTA = 0.11 ms
Sep 5 04:28:01 PING CRITICAL - Packet loss = 80%, RTA = 0.11 ms
Sep 5 04:32:53 PING WARNING - Packet loss = 44%, RTA = 0.10 ms
Sep 5 04:37:51 PING OK - Packet loss = 16%, RTA = 0.12 ms
Sep 5 04:42:51 PING WARNING - Packet loss = 50%, RTA = 0.11 ms
Sep 5 04:44:02 PING CRITICAL - Packet loss = 60%, RTA = 0.09 ms
Sep 5 04:45:01 PING CRITICAL - Packet loss = 70%, RTA = 0.10 ms
...
When it happens the pattern of ping failures remains
basically the same until the system reboots.
# ping -n -c10 localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.074 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.075 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.074 ms
64 bytes from 127.0.0.1: icmp_seq=8 ttl=64 time=0.097 ms
64 bytes from 127.0.0.1: icmp_seq=10 ttl=64 time=0.067 ms
--- localhost.localdomain ping statistics ---
10 packets transmitted, 5 received, 50% packet loss, time 9004ms
rtt min/avg/max/mdev = 0.067/0.077/0.097/0.012 ms
Interestingly enough flood pings yeild much better results.
# ping -nf -c10000 localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
.......
--- localhost.localdomain ping statistics ---
10000 packets transmitted, 9993 received, 0% packet loss, time 646ms
rtt min/avg/max/mdev = 0.019/0.025/0.119/0.010 ms, ipg/ewma 0.064/0.020
ms
Any ideas or suggestions welcome.
Thanks,
Sasha