I had the dom0 which unfortunately didn''t have a console on it hit a race condition and saw oom errors on it also. This happened after it was running for over 36 hours with a domU whose load average was avg was around 3 most of the time. changeset: 7396:9b51e7637676 Dom0 - UP i686, Centos 4.1, 768 megs domU-1 92 megs snmpd domU-2 92 megs snmpdd domU-3 410 megs postgres, tomcat 5.5, opennms head on java 5 Got this from syslog Oct 18 10:31:28 tarkus kernel: peth0: received packet with own address as source address Oct 18 10:32:03 tarkus kernel: peth0: received packet with own address as source address Oct 18 10:33:53 tarkus kernel: oom-killer: gfp_mask=0xd2 Oct 18 10:33:53 tarkus kernel: DMA per-cpu: Oct 18 10:33:53 tarkus kernel: cpu 0 hot: low 30, high 90, batch 15 Oct 18 10:33:53 tarkus kernel: cpu 0 cold: low 0, high 30, batch 15 Oct 18 10:33:53 tarkus kernel: Normal per-cpu: empty Oct 18 10:33:53 tarkus kernel: HighMem per-cpu: empty Oct 18 10:33:53 tarkus kernel: Oct 18 10:33:53 tarkus kernel: Free pages: 1948kB (0kB HighMem) Oct 18 10:34:02 tarkus kernel: Active:899 inactive:695 dirty:0 writeback:7 unstable:0 free:487 slab:28555 mapped:815 pagetables:206 Oct 18 10:34:09 tarkus crond(pam_unix)[25117]: session closed for user root Oct 18 10:34:16 tarkus kernel: DMA free:1948kB min:1480kB low:1848kB high:2220kB active:3596kB inactive:2780kB present:137200kB pages_scanned:1480 all_unreclaimable? no Oct 18 10:34:42 tarkus kernel: lowmem_reserve[]: 0 0 0 Oct 18 10:34:53 tarkus kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Oct 18 10:35:01 tarkus kernel: lowmem_reserve[]: 0 0 0 Oct 18 10:35:02 tarkus kernel: HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Oct 18 10:35:10 tarkus kernel: lowmem_reserve[]: 0 0 0 Oct 18 10:35:17 tarkus crond(pam_unix)[25252]: session opened for user root by (uid=0) Oct 18 10:35:22 tarkus kernel: DMA: 113*4kB 9*8kB 3*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1948kB Oct 18 10:35:28 tarkus kernel: Normal: empty Oct 18 10:35:37 tarkus kernel: HighMem: empty Oct 18 10:35:43 tarkus kernel: Swap cache: add 164344, delete 163838, find 45906/87552, race 0+46 Oct 18 10:35:43 tarkus kernel: Free swap = 1553068kB Oct 18 10:35:43 tarkus kernel: Total swap = 1572856kB Oct 18 10:35:43 tarkus kernel: Out of Memory: Killed process 25124 (sendmail). Oct 18 10:35:52 tarkus snmpd[3925]: send response: (if_nameindex() failed) Oct 18 10:36:06 tarkus snmpd[3925]: send response: Oct 18 10:36:26 tarkus snmpd[3925]: send response: Oct 18 10:37:00 tarkus last message repeated 4 times Oct 18 10:37:02 tarkus kernel: peth0: received packet with own address as source address Regards, Ted _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I had the dom0 which unfortunately didn''t have a console on it hit a > race condition and saw oom errors on it also. This happened > after it was > running for over 36 hours with a domU whose load average was avg was > around 3 most of the time.Were you running anything in the dom0 other than xend etc? It looks like the slabcache it dom0 is over 100MB, so having the ouput of /proc/slabinfo would be interesting when the machine is in this state. It might be good to look at this overtime and see if something is being leaked. Ian> changeset: 7396:9b51e7637676 > > Dom0 - UP i686, Centos 4.1, 768 megs > > domU-1 92 megs snmpd > domU-2 92 megs snmpdd > domU-3 410 megs postgres, tomcat 5.5, opennms head on java 5 > > Got this from syslog > > Oct 18 10:31:28 tarkus kernel: peth0: received packet with > own address > as source address > Oct 18 10:32:03 tarkus kernel: peth0: received packet with > own address > as source address > Oct 18 10:33:53 tarkus kernel: oom-killer: gfp_mask=0xd2 > Oct 18 10:33:53 tarkus kernel: DMA per-cpu: > Oct 18 10:33:53 tarkus kernel: cpu 0 hot: low 30, high 90, batch 15 > Oct 18 10:33:53 tarkus kernel: cpu 0 cold: low 0, high 30, batch 15 > Oct 18 10:33:53 tarkus kernel: Normal per-cpu: empty > Oct 18 10:33:53 tarkus kernel: HighMem per-cpu: empty > Oct 18 10:33:53 tarkus kernel: > Oct 18 10:33:53 tarkus kernel: Free pages: 1948kB (0kB HighMem) > Oct 18 10:34:02 tarkus kernel: Active:899 inactive:695 dirty:0 > writeback:7 unstable:0 free:487 slab:28555 mapped:815 pagetables:206 > Oct 18 10:34:09 tarkus crond(pam_unix)[25117]: session closed for user > root > Oct 18 10:34:16 tarkus kernel: DMA free:1948kB min:1480kB low:1848kB > high:2220kB active:3596kB inactive:2780kB present:137200kB > pages_scanned:1480 all_unreclaimable? no > Oct 18 10:34:42 tarkus kernel: lowmem_reserve[]: 0 0 0 > Oct 18 10:34:53 tarkus kernel: Normal free:0kB min:0kB > low:0kB high:0kB > active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? > no > Oct 18 10:35:01 tarkus kernel: lowmem_reserve[]: 0 0 0 > Oct 18 10:35:02 tarkus kernel: HighMem free:0kB min:128kB low:160kB > high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 > all_unreclaimable? no > Oct 18 10:35:10 tarkus kernel: lowmem_reserve[]: 0 0 0 > Oct 18 10:35:17 tarkus crond(pam_unix)[25252]: session opened for user > root by (uid=0) > Oct 18 10:35:22 tarkus kernel: DMA: 113*4kB 9*8kB 3*16kB 1*32kB 1*64kB > 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1948kB > Oct 18 10:35:28 tarkus kernel: Normal: empty > Oct 18 10:35:37 tarkus kernel: HighMem: empty > Oct 18 10:35:43 tarkus kernel: Swap cache: add 164344, delete 163838, > find 45906/87552, race 0+46 > Oct 18 10:35:43 tarkus kernel: Free swap = 1553068kB > Oct 18 10:35:43 tarkus kernel: Total swap = 1572856kB > Oct 18 10:35:43 tarkus kernel: Out of Memory: Killed process 25124 > (sendmail). > Oct 18 10:35:52 tarkus snmpd[3925]: send response: (if_nameindex() > failed) > Oct 18 10:36:06 tarkus snmpd[3925]: send response: > Oct 18 10:36:26 tarkus snmpd[3925]: send response: > Oct 18 10:37:00 tarkus last message repeated 4 times > Oct 18 10:37:02 tarkus kernel: peth0: received packet with > own address > as source address > > Regards, > Ted > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2005-10-18 at 23:10 +0100, Ian Pratt wrote:> > I had the dom0 which unfortunately didn''t have a console on it hit a > > race condition and saw oom errors on it also. This happened > > after it was > > running for over 36 hours with a domU whose load average was avg was > > around 3 most of the time. > > Were you running anything in the dom0 other than xend etc? > > It looks like the slabcache it dom0 is over 100MB, so having the ouput > of /proc/slabinfo would be interesting when the machine is in this > state. It might be good to look at this overtime and see if something is > being leaked.zebra, ospfd, bgpd, smartd, cupsd, snmpd, acpid, normally I am below 100 megs usage. [root@tarkus ~]# free total used free shared buffers cached Mem: 132212 96796 35416 0 12308 40124 -/+ buffers/cache: 44364 87848 Swap: 1572856 0 1572856 These usage is fairly consistent, don''t have any snaps right before it happened :-) Anything specific in slabinfo to look for ? Regards, Ted> > Ian > > > > changeset: 7396:9b51e7637676 > > > > Dom0 - UP i686, Centos 4.1, 768 megs > > > > domU-1 92 megs snmpd > > domU-2 92 megs snmpdd > > domU-3 410 megs postgres, tomcat 5.5, opennms head on java 5 > > > > Got this from syslog > > > > Oct 18 10:31:28 tarkus kernel: peth0: received packet with > > own address > > as source address > > Oct 18 10:32:03 tarkus kernel: peth0: received packet with > > own address > > as source address > > Oct 18 10:33:53 tarkus kernel: oom-killer: gfp_mask=0xd2 > > Oct 18 10:33:53 tarkus kernel: DMA per-cpu: > > Oct 18 10:33:53 tarkus kernel: cpu 0 hot: low 30, high 90, batch 15 > > Oct 18 10:33:53 tarkus kernel: cpu 0 cold: low 0, high 30, batch 15 > > Oct 18 10:33:53 tarkus kernel: Normal per-cpu: empty > > Oct 18 10:33:53 tarkus kernel: HighMem per-cpu: empty > > Oct 18 10:33:53 tarkus kernel: > > Oct 18 10:33:53 tarkus kernel: Free pages: 1948kB (0kB HighMem) > > Oct 18 10:34:02 tarkus kernel: Active:899 inactive:695 dirty:0 > > writeback:7 unstable:0 free:487 slab:28555 mapped:815 pagetables:206 > > Oct 18 10:34:09 tarkus crond(pam_unix)[25117]: session closed for user > > root > > Oct 18 10:34:16 tarkus kernel: DMA free:1948kB min:1480kB low:1848kB > > high:2220kB active:3596kB inactive:2780kB present:137200kB > > pages_scanned:1480 all_unreclaimable? no > > Oct 18 10:34:42 tarkus kernel: lowmem_reserve[]: 0 0 0 > > Oct 18 10:34:53 tarkus kernel: Normal free:0kB min:0kB > > low:0kB high:0kB > > active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? > > no > > Oct 18 10:35:01 tarkus kernel: lowmem_reserve[]: 0 0 0 > > Oct 18 10:35:02 tarkus kernel: HighMem free:0kB min:128kB low:160kB > > high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 > > all_unreclaimable? no > > Oct 18 10:35:10 tarkus kernel: lowmem_reserve[]: 0 0 0 > > Oct 18 10:35:17 tarkus crond(pam_unix)[25252]: session opened for user > > root by (uid=0) > > Oct 18 10:35:22 tarkus kernel: DMA: 113*4kB 9*8kB 3*16kB 1*32kB 1*64kB > > 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1948kB > > Oct 18 10:35:28 tarkus kernel: Normal: empty > > Oct 18 10:35:37 tarkus kernel: HighMem: empty > > Oct 18 10:35:43 tarkus kernel: Swap cache: add 164344, delete 163838, > > find 45906/87552, race 0+46 > > Oct 18 10:35:43 tarkus kernel: Free swap = 1553068kB > > Oct 18 10:35:43 tarkus kernel: Total swap = 1572856kB > > Oct 18 10:35:43 tarkus kernel: Out of Memory: Killed process 25124 > > (sendmail). > > Oct 18 10:35:52 tarkus snmpd[3925]: send response: (if_nameindex() > > failed) > > Oct 18 10:36:06 tarkus snmpd[3925]: send response: > > Oct 18 10:36:26 tarkus snmpd[3925]: send response: > > Oct 18 10:37:00 tarkus last message repeated 4 times > > Oct 18 10:37:02 tarkus kernel: peth0: received packet with > > own address > > as source address > > > > Regards, > > Ted > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2005-10-18 at 23:10 +0100, Ian Pratt wrote:> > I had the dom0 which unfortunately didn''t have a console on it hit a > > race condition and saw oom errors on it also. This happened > > after it was > > running for over 36 hours with a domU whose load average was avg was > > around 3 most of the time. > > Were you running anything in the dom0 other than xend etc? >zebra, ospfd, bgpd, smartd, cupsd, snmpd, acpid, normally I am below 100 megs usage. [root@tarkus ~]# free total used free shared buffers cached Mem: 132212 96796 35416 0 12308 40124 -/+ buffers/cache: 44364 87848 Swap: 1572856 0 1572856 That is fairly consistent, don''t have any snaps right before it happened :-) Regards, Ted _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > It looks like the slabcache it dom0 is over 100MB, so > having the ouput > > of /proc/slabinfo would be interesting when the machine is in this > > state. It might be good to look at this overtime and see if> Anything specific in slabinfo to look for ?Look for an object type with an abnormally large number of objects in it (e.g. where size*num > 40MB). In particularly, keep an eye out for an object type which seems to grow steadily over time. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Apparently Analagous Threads
- 2.6 Kernel Out of memory
- XEN Dom 0 boot and Out of Memory issue
- [Bug 93458] New: page allocation failure: order:5, mode:0x240c0c0
- Xen CentOS 7.3 server + CentOS 7.3 VM fails to boot after CR updates (applied to VM)!
- Xen CentOS 7.3 server + CentOS 7.3 VM fails to boot after CR updates (applied to VM)!