James Pifer
2009-Dec-31 04:39 UTC
[Xen-users] Memory problems persist... Cannot allocate memory
I have always had a problem with my xen servers having memory problems after a random period of time. For example, even if there''s free memory available, if you try to add a new domU or restart one, virt-manager says it can''t allocate memory. I was really hoping I had gotten past this issue on my latest SLES11 servers running xen 3.4 by specifying domU memory and disabling ballooning. Unfortunately the problem has reappeared on one of these servers. My only solution is a restart, which affects 12-15 servers. Are others having this problem? There has to be a resolution to this. I will try restarting xend and see if that resolves it, but I need to wait until I can restart dom0 in case it completely hoses it. I also just made another post that has me scared to death where I completely lost a whole domU from my ocfs2 volume when I rebooted a server, so I''m a bit gun shy right now! Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2009-Dec-31 14:05 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Wed, Dec 30, 2009 at 11:39:20PM -0500, James Pifer wrote:> I have always had a problem with my xen servers having memory problems > after a random period of time. For example, even if there''s free memory > available, if you try to add a new domU or restart one, virt-manager > says it can''t allocate memory. I was really hoping I had gotten past > this issue on my latest SLES11 servers running xen 3.4 by specifying > domU memory and disabling ballooning. >How do you check this ''free memory''? from dom0, or from Xen hypervisor "xm info" ?> Unfortunately the problem has reappeared on one of these servers. My > only solution is a restart, which affects 12-15 servers. Are others > having this problem? There has to be a resolution to this. >This could be because you haven''t limited dom0 memory to a dedicated amount, and now you need to balloon down dom0 memory every time you start a new guest..> I will try restarting xend and see if that resolves it, but I need to > wait until I can restart dom0 in case it completely hoses it. > > I also just made another post that has me scared to death where I > completely lost a whole domU from my ocfs2 volume when I rebooted a > server, so I''m a bit gun shy right now! >Have you dedicated memory to dom0? ie. have you specified dom0_mem=512M (or whatever you need) in grub.conf for xen.gz ? In addition to dom0_mem for Xen hypervisor, you should also have dom0-min-mem= specified in /etc/xen/xend-config.sxp to prevent dom0 from being ballooned down. If you don''t have dom0_mem set for xen.gz, then dom0 will have all the memory visible to it, and many dom0 Linux kernel parameters and buffers are calculated/allocated based on that initial amount of memory - and then during runtime when you start guests dom0 memory needs to be ballooned down, and things start to go wrong at some point. Linux doesn''t really like when it''s memory goes to a only small fraction of what it had in the beginning (during startup). So yeah, make sure dom0 doesn''t need to be ballooned down during runtime. Dedicate fixed amount of memory for dom0. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2009-Dec-31 15:45 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> How do you check this ''free memory''? from dom0, or from Xen hypervisor > "xm info" ?Yes, from xm info.> Have you dedicated memory to dom0? ie. have you specified dom0_mem=512M > (or whatever you need) in grub.conf for xen.gz ?Yes: kernel /boot/xen.gz dom0_mem=2048M> > In addition to dom0_mem for Xen hypervisor, you should also have > dom0-min-mem= specified in /etc/xen/xend-config.sxp to prevent dom0 from > being ballooned down.# dom0-min-mem is the lowest permissible memory level (in MB) for dom0. # This is a minimum both for auto-ballooning (as enabled by # enable-dom0-ballooning below) and for xm mem-set when applied to dom0. (dom0-min-mem 512) # Whether to enable auto-ballooning of dom0 to allow domUs to be created. # If enable-dom0-ballooning = no, dom0 will never balloon out. (enable-dom0-ballooning no) Should this match the mem set at the kernel level or is this ok? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2009-Dec-31 16:46 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Thu, Dec 31, 2009 at 10:45:13AM -0500, James Pifer wrote:> > How do you check this ''free memory''? from dom0, or from Xen hypervisor > > "xm info" ? > > Yes, from xm info. > > > > Have you dedicated memory to dom0? ie. have you specified dom0_mem=512M > > (or whatever you need) in grub.conf for xen.gz ? > > Yes: > kernel /boot/xen.gz dom0_mem=2048M > > > > > In addition to dom0_mem for Xen hypervisor, you should also have > > dom0-min-mem= specified in /etc/xen/xend-config.sxp to prevent dom0 from > > being ballooned down. > > # dom0-min-mem is the lowest permissible memory level (in MB) for dom0. > # This is a minimum both for auto-ballooning (as enabled by > # enable-dom0-ballooning below) and for xm mem-set when applied to dom0. > (dom0-min-mem 512) > > # Whether to enable auto-ballooning of dom0 to allow domUs to be > created. > # If enable-dom0-ballooning = no, dom0 will never balloon out. > (enable-dom0-ballooning no) > > Should this match the mem set at the kernel level or is this ok? >Looks like you don''t have dom0 ballooning enabled, so then dom0 should always have the dedicated 2048 MB of memory. What was the error you got? check dom0 "dmesg", "xm dmesg" and "xm log" when creating 15/16+ guests.. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2009-Dec-31 18:44 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> Looks like you don''t have dom0 ballooning enabled, so then dom0 should > always have the dedicated 2048 MB of memory. > > What was the error you got? check dom0 "dmesg", "xm dmesg" and "xm log" > when creating 15/16+ guests.. > > -- PasiHere''s what I get in the log when I try to start a domain right now: [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2757) XendDomainInfo.destroy: domid=40 [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2227) Destroying device model [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2234) Releasing devices [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2247) Removing vif/0 [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2247) Removing vkbd/0 [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vkbd, device = vkbd/0 [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2247) Removing console/0 [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0 [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:2247) Removing vbd/768 [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768 [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:2247) Removing vbd/832 [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/832 [2009-12-31 11:42:23 5611] DEBUG (XendDomainInfo:2247) Removing vfb/0 [2009-12-31 11:42:23 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0 [2009-12-31 11:42:23 5611] ERROR (SrvBase:88) Request start failed. Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/web/SrvBase.py", line 85, in perform return op_method(op, req) File "/usr/lib64/python2.6/site-packages/xen/xend/server/SrvDomain.py", line 77, in op_start return self.xd.domain_start(self.dom.getName(), paused) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomain.py", line 1040, in domain_start dominfo.start(is_managed = True) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 459, in start XendTask.log_progress(31, 60, self._initDomain) File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2594, in _initDomain self._createDevices() File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2218, in _createDevices self.image.createDeviceModel() File "/usr/lib64/python2.6/site-packages/xen/xend/image.py", line 419, in createDeviceModel pid = os.fork() OSError: [Errno 12] Cannot allocate memory Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2009-Dec-31 19:11 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Thu, Dec 31, 2009 at 01:44:30PM -0500, James Pifer wrote:> > Looks like you don''t have dom0 ballooning enabled, so then dom0 should > > always have the dedicated 2048 MB of memory. > > > > What was the error you got? check dom0 "dmesg", "xm dmesg" and "xm log" > > when creating 15/16+ guests.. > > > > -- Pasi > > Here''s what I get in the log when I try to start a domain right now: > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2757) XendDomainInfo.destroy: domid=40 > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2227) Destroying device model > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2234) Releasing devices > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2247) Removing vif/0 > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2247) Removing vkbd/0 > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vkbd, device = vkbd/0 > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:2247) Removing console/0 > [2009-12-31 11:42:19 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0 > [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:2247) Removing vbd/768 > [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768 > [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:2247) Removing vbd/832 > [2009-12-31 11:42:22 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/832 > [2009-12-31 11:42:23 5611] DEBUG (XendDomainInfo:2247) Removing vfb/0 > [2009-12-31 11:42:23 5611] DEBUG (XendDomainInfo:1137) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0 > [2009-12-31 11:42:23 5611] ERROR (SrvBase:88) Request start failed. > Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/xen/web/SrvBase.py", line 85, in perform > return op_method(op, req) > File "/usr/lib64/python2.6/site-packages/xen/xend/server/SrvDomain.py", line 77, in op_start > return self.xd.domain_start(self.dom.getName(), paused) > File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomain.py", line 1040, in domain_start > dominfo.start(is_managed = True) > File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 459, in start > XendTask.log_progress(31, 60, self._initDomain) > File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress > retval = func(*args, **kwds) > File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2594, in _initDomain > self._createDevices() > File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2218, in _createDevices > self.image.createDeviceModel() > File "/usr/lib64/python2.6/site-packages/xen/xend/image.py", line 419, in createDeviceModel > pid = os.fork() > OSError: [Errno 12] Cannot allocate memory >So what does "xm info" say about free memory? How much does the guest you''re trying to start have memory configured for it? -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2009-Dec-31 19:48 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Thu, Dec 31, 2009 at 03:45:36PM -0500, James Pifer wrote:> > So what does "xm info" say about free memory? How much does the guest > > you''re trying to start have memory configured for it? > > > > -- Pasi > > The domU only has 512meg. >Ok. Really weird. How about dom0, does it have free memory? This was SLES11? -- Pasi> # xm info > host : xen06 > release : 2.6.27.19-5-xen > version : #1 SMP 2009-02-28 04:40:21 +0100 > machine : x86_64 > nr_cpus : 16 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 2 > cpu_mhz : 2533 > hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 > virt_caps : hvm > total_memory : 32757 > free_memory : 12769 > max_free_memory : 13280 > max_para_memory : 13276 > max_hvm_memory : 13236 > node_to_cpu : node0:0-15 > node_to_memory : node0:12769 > xen_major : 3 > xen_minor : 4 > xen_extra : .1_19718_04-26. > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : 19718 > cc_compiler : gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) > cc_compile_by : abuild > cc_compile_domain : > cc_compile_date : Wed Oct 14 00:02:21 UTC 2009 > xend_config_format : 4 > > > Thanks, > James >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2009-Dec-31 20:45 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> So what does "xm info" say about free memory? How much does the guest > you''re trying to start have memory configured for it? > > -- PasiThe domU only has 512meg. # xm info host : xen06 release : 2.6.27.19-5-xen version : #1 SMP 2009-02-28 04:40:21 +0100 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2533 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32757 free_memory : 12769 max_free_memory : 13280 max_para_memory : 13276 max_hvm_memory : 13236 node_to_cpu : node0:0-15 node_to_memory : node0:12769 xen_major : 3 xen_minor : 4 xen_extra : .1_19718_04-26. xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : 19718 cc_compiler : gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) cc_compile_by : abuild cc_compile_domain : cc_compile_date : Wed Oct 14 00:02:21 UTC 2009 xend_config_format : 4 Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2009-Dec-31 21:35 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Thu, 2009-12-31 at 21:48 +0200, Pasi Kärkkäinen wrote:> On Thu, Dec 31, 2009 at 03:45:36PM -0500, James Pifer wrote: > > > So what does "xm info" say about free memory? How much does the guest > > > you''re trying to start have memory configured for it? > > > > > > -- Pasi > > > > The domU only has 512meg. > > > > Ok. Really weird. > How about dom0, does it have free memory? > > This was SLES11? > > -- Pasi > > > > # xm info > > host : xen06 > > release : 2.6.27.19-5-xen > > version : #1 SMP 2009-02-28 04:40:21 +0100 > > machine : x86_64 > > nr_cpus : 16 > > nr_nodes : 1 > > cores_per_socket : 4 > > threads_per_core : 2 > > cpu_mhz : 2533 > > hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 > > virt_caps : hvm > > total_memory : 32757 > > free_memory : 12769 > > max_free_memory : 13280 > > max_para_memory : 13276 > > max_hvm_memory : 13236 > > node_to_cpu : node0:0-15 > > node_to_memory : node0:12769 > > xen_major : 3 > > xen_minor : 4 > > xen_extra : .1_19718_04-26. > > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > > xen_scheduler : credit > > xen_pagesize : 4096 > > platform_params : virt_start=0xffff800000000000 > > xen_changeset : 19718 > > cc_compiler : gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) > > cc_compile_by : abuild > > cc_compile_domain : > > cc_compile_date : Wed Oct 14 00:02:21 UTC 2009 > > xend_config_format : 4 > > > > > > Thanks, > > James > >Looks like dom0 is just about out of memory. It currently has 1 gb of RAM. Looks like I changed menu.lst on Dec 12 to give it 2 gb of RAM, but I haven''t had a chance to restart it. It''s been up for 29 days. It''s a dual quad core machine with 32gb of ram. It will probably run 12-18 domU''s when all is said and done ranging anywhere from 512M to 4gb of ram. Mixed Windows and Linux. Do you think 2gb of ram is enough for dom0 or should I go more, like 3 or 4gb? "top" gives me: top - 14:18:13 up 29 days, 17:41, 3 users, load average: 0.47, 0.77, 0.71 Tasks: 302 total, 3 running, 299 sleeping, 0 stopped, 0 zombie Cpu(s): 0.4%us, 0.1%sy, 0.0%ni, 98.4%id, 0.7%wa, 0.0%hi, 0.0%si, 0.4%st Mem: 1047040k total, 1036656k used, 10384k free, 7816k buffers Swap: 2104472k total, 2104156k used, 316k free, 53876k cached Thanks for everything, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-01 00:53 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Thu, Dec 31, 2009 at 04:35:07PM -0500, James Pifer wrote:> On Thu, 2009-12-31 at 21:48 +0200, Pasi Kärkkäinen wrote: > > On Thu, Dec 31, 2009 at 03:45:36PM -0500, James Pifer wrote: > > > > So what does "xm info" say about free memory? How much does the guest > > > > you''re trying to start have memory configured for it? > > > > > > > > -- Pasi > > > > > > The domU only has 512meg. > > > > > > > Ok. Really weird. > > How about dom0, does it have free memory? > > > > This was SLES11? > > > > -- Pasi > > > > > > > # xm info > > > host : xen06 > > > release : 2.6.27.19-5-xen > > > version : #1 SMP 2009-02-28 04:40:21 +0100 > > > machine : x86_64 > > > nr_cpus : 16 > > > nr_nodes : 1 > > > cores_per_socket : 4 > > > threads_per_core : 2 > > > cpu_mhz : 2533 > > > hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 > > > virt_caps : hvm > > > total_memory : 32757 > > > free_memory : 12769 > > > max_free_memory : 13280 > > > max_para_memory : 13276 > > > max_hvm_memory : 13236 > > > node_to_cpu : node0:0-15 > > > node_to_memory : node0:12769 > > > xen_major : 3 > > > xen_minor : 4 > > > xen_extra : .1_19718_04-26. > > > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > > > xen_scheduler : credit > > > xen_pagesize : 4096 > > > platform_params : virt_start=0xffff800000000000 > > > xen_changeset : 19718 > > > cc_compiler : gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) > > > cc_compile_by : abuild > > > cc_compile_domain : > > > cc_compile_date : Wed Oct 14 00:02:21 UTC 2009 > > > xend_config_format : 4 > > > > > > > > > Thanks, > > > James > > > > > Looks like dom0 is just about out of memory. It currently has 1 gb of > RAM. Looks like I changed menu.lst on Dec 12 to give it 2 gb of RAM, but > I haven''t had a chance to restart it. It''s been up for 29 days. It''s a > dual quad core machine with 32gb of ram. It will probably run 12-18 > domU''s when all is said and done ranging anywhere from 512M to 4gb of > ram. Mixed Windows and Linux. Do you think 2gb of ram is enough for dom0 > or should I go more, like 3 or 4gb? >2GB definitely should be enough.. afaik.> "top" gives me: > top - 14:18:13 up 29 days, 17:41, 3 users, load average: 0.47, 0.77, 0.71 > Tasks: 302 total, 3 running, 299 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.4%us, 0.1%sy, 0.0%ni, 98.4%id, 0.7%wa, 0.0%hi, 0.0%si, 0.4%st > Mem: 1047040k total, 1036656k used, 10384k free, 7816k buffers > Swap: 2104472k total, 2104156k used, 316k free, 53876k cached >So what''s eating all the memory and swap? -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-01 04:42 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> > > > Looks like dom0 is just about out of memory. It currently has 1 gb of > > RAM. Looks like I changed menu.lst on Dec 12 to give it 2 gb of RAM, but > > I haven''t had a chance to restart it. It''s been up for 29 days. It''s a > > dual quad core machine with 32gb of ram. It will probably run 12-18 > > domU''s when all is said and done ranging anywhere from 512M to 4gb of > > ram. Mixed Windows and Linux. Do you think 2gb of ram is enough for dom0 > > or should I go more, like 3 or 4gb? > > > > 2GB definitely should be enough.. afaik. > > > "top" gives me: > > top - 14:18:13 up 29 days, 17:41, 3 users, load average: 0.47, 0.77, 0.71 > > Tasks: 302 total, 3 running, 299 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.4%us, 0.1%sy, 0.0%ni, 98.4%id, 0.7%wa, 0.0%hi, 0.0%si, 0.4%st > > Mem: 1047040k total, 1036656k used, 10384k free, 7816k buffers > > Swap: 2104472k total, 2104156k used, 316k free, 53876k cached > > > > So what''s eating all the memory and swap? > > -- Pasi >What''s the best way to see what''s using all the memory and swap? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-01 13:06 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Thu, Dec 31, 2009 at 11:42:55PM -0500, James Pifer wrote:> > > > > > Looks like dom0 is just about out of memory. It currently has 1 gb of > > > RAM. Looks like I changed menu.lst on Dec 12 to give it 2 gb of RAM, but > > > I haven''t had a chance to restart it. It''s been up for 29 days. It''s a > > > dual quad core machine with 32gb of ram. It will probably run 12-18 > > > domU''s when all is said and done ranging anywhere from 512M to 4gb of > > > ram. Mixed Windows and Linux. Do you think 2gb of ram is enough for dom0 > > > or should I go more, like 3 or 4gb? > > > > > > > 2GB definitely should be enough.. afaik. > > > > > "top" gives me: > > > top - 14:18:13 up 29 days, 17:41, 3 users, load average: 0.47, 0.77, 0.71 > > > Tasks: 302 total, 3 running, 299 sleeping, 0 stopped, 0 zombie > > > Cpu(s): 0.4%us, 0.1%sy, 0.0%ni, 98.4%id, 0.7%wa, 0.0%hi, 0.0%si, 0.4%st > > > Mem: 1047040k total, 1036656k used, 10384k free, 7816k buffers > > > Swap: 2104472k total, 2104156k used, 316k free, 53876k cached > > > > > > > So what''s eating all the memory and swap? > > > > -- Pasi > > > > What''s the best way to see what''s using all the memory and swap? >For example "top", sort by memory usage by pressing shift+m. Or just the usual "ps aux" output. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-01 15:41 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Fri, Jan 01, 2010 at 10:59:00AM -0500, James Pifer wrote:> > For example "top", sort by memory usage by pressing shift+m. > > Or just the usual "ps aux" output. > > > > -- Pasi > > Happy New Year! >Happy new year to you too!> This morning I went to look at the memory being used and box was locked > up requiring a hard reset. Thankfully everything came back up ok. Upon > reboot dom0 now has 2gb of RAM. So I started bringing up my domUs. >You definitely have something wrong.. you really shouldn''t run out of ram in dom0 and definitely shouldn''t crash.> After starting about half of them the available RAM was already used!!! >Well WHAT in dom0 was using all the RAM? Please figure that out. Was it some process, or just memory cache?> Having plenty of available RAM with 32gb total, I shut down the domU''s > and bumped the RAM to 4gb to see what would happen. I now have all 15 > domUs started, and it sucked up all available RAM, except for the 2gb of > swap, which is mostly free. > > The server runs in runlevel 3, but I do run a vncserver session because > I''m remote. Even so, the gnome session does not appear to be taking up > much memory. As a matter of fact, if I do a ps aux and add up the memory > column, it only comes out to a little over 8%, so I don''t understand > what''s happening. > > top now shows me: > top - 08:52:34 up 24 min, 2 users, load average: 0.71, 1.46, 2.70 > Tasks: 303 total, 1 running, 302 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.9%us, 0.8%sy, 0.0%ni, 92.4%id, 5.4%wa, 0.0%hi, 0.0%si, 0.5%st > Mem: 4192768k total, 4154732k used, 38036k free, 78084k buffers > Swap: 2104472k total, 112k used, 2104360k free, 3565872k cached >Well you have 3,5 GB in disk caches (3565872k cached), so you don''t have much RAM at all in use in dom0. You know Linux uses all/most of the free memory as disk cache automatically? So if you grow dom0 memory the end result is that dom0 Linux will just use more memory as disk caches. You really need to figure out what was using all the memory and swap earlier.> My domU''s are: > Name Mem VCPUs > Domain-0 4094 16 > windows2k3 2048 2 > windows2k 1024 1 > sles10 2048 2 > sles10 1024 4 > sles10 2048 2 > windows2k3 1024 1 > windows2k3 2048 1 > sles10 1024 2 > win2k8 2048 4 > windows2k3 1536 1 > windows2k3 512 1 > windows2k3 512 1 > windows2k3 512 1 > windows2k3 1024 1 > windows2k3 512 1 > > Any thoughts? >You really need to start monitoring the server/dom0 in more detail. Are you using file: backend for domU disks, or tap:aio: ? -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-01 15:59 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> For example "top", sort by memory usage by pressing shift+m. > Or just the usual "ps aux" output. > > -- PasiHappy New Year! This morning I went to look at the memory being used and box was locked up requiring a hard reset. Thankfully everything came back up ok. Upon reboot dom0 now has 2gb of RAM. So I started bringing up my domUs. After starting about half of them the available RAM was already used!!! Having plenty of available RAM with 32gb total, I shut down the domU''s and bumped the RAM to 4gb to see what would happen. I now have all 15 domUs started, and it sucked up all available RAM, except for the 2gb of swap, which is mostly free. The server runs in runlevel 3, but I do run a vncserver session because I''m remote. Even so, the gnome session does not appear to be taking up much memory. As a matter of fact, if I do a ps aux and add up the memory column, it only comes out to a little over 8%, so I don''t understand what''s happening. top now shows me: top - 08:52:34 up 24 min, 2 users, load average: 0.71, 1.46, 2.70 Tasks: 303 total, 1 running, 302 sleeping, 0 stopped, 0 zombie Cpu(s): 0.9%us, 0.8%sy, 0.0%ni, 92.4%id, 5.4%wa, 0.0%hi, 0.0%si, 0.5%st Mem: 4192768k total, 4154732k used, 38036k free, 78084k buffers Swap: 2104472k total, 112k used, 2104360k free, 3565872k cached My domU''s are: Name Mem VCPUs Domain-0 4094 16 windows2k3 2048 2 windows2k 1024 1 sles10 2048 2 sles10 1024 4 sles10 2048 2 windows2k3 1024 1 windows2k3 2048 1 sles10 1024 2 win2k8 2048 4 windows2k3 1536 1 windows2k3 512 1 windows2k3 512 1 windows2k3 512 1 windows2k3 1024 1 windows2k3 512 1 Any thoughts? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-01 20:06 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> You really need to start monitoring the server/dom0 in more detail. > > Are you using file: backend for domU disks, or tap:aio: ? > > -- PasiThe domUs are all using file based storage. The memory did not fill up until I started the domUs. I can''t tell what''s using the ram. Here is an output of ps aux: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 1064 384 ? Ss 08:28 0:00 init [3] root 2 0.0 0.0 0 0 ? S< 08:28 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/0] root 4 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/1] root 6 0.0 0.0 0 0 ? S< 08:28 0:03 [ksoftirqd/1] root 7 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/2] root 8 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/2] root 9 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/3] root 10 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/3] root 11 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/4] root 12 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/4] root 13 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/5] root 14 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/5] root 15 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/6] root 16 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/6] root 17 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/7] root 18 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/7] root 19 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/8] root 20 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/8] root 21 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/9] root 22 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/9] root 23 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/10] root 24 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/10] root 25 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/11] root 26 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/11] root 27 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/12] root 28 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/12] root 29 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/13] root 30 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/13] root 31 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/14] root 32 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/14] root 33 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/15] root 34 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/15] root 35 0.0 0.0 0 0 ? S< 08:28 0:00 [events/0] root 36 0.0 0.0 0 0 ? S< 08:28 0:00 [events/1] root 37 0.0 0.0 0 0 ? S< 08:28 0:00 [events/2] root 38 0.0 0.0 0 0 ? S< 08:28 0:00 [events/3] root 39 0.0 0.0 0 0 ? S< 08:28 0:00 [events/4] root 40 0.0 0.0 0 0 ? S< 08:28 0:00 [events/5] root 41 0.0 0.0 0 0 ? S< 08:28 0:00 [events/6] root 42 0.0 0.0 0 0 ? S< 08:28 0:00 [events/7] root 43 0.0 0.0 0 0 ? S< 08:28 0:00 [events/8] root 44 0.0 0.0 0 0 ? S< 08:28 0:00 [events/9] root 45 0.0 0.0 0 0 ? S< 08:28 0:00 [events/10] root 46 0.0 0.0 0 0 ? S< 08:28 0:00 [events/11] root 47 0.0 0.0 0 0 ? S< 08:28 0:00 [events/12] root 48 0.0 0.0 0 0 ? S< 08:28 0:00 [events/13] root 49 0.0 0.0 0 0 ? S< 08:28 0:00 [events/14] root 50 0.0 0.0 0 0 ? S< 08:28 0:00 [events/15] root 51 0.0 0.0 0 0 ? S< 08:28 0:00 [khelper] root 52 0.0 0.0 0 0 ? S< 08:28 0:00 [xenwatch] root 53 0.0 0.0 0 0 ? S< 08:28 0:00 [xenbus] root 54 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/0] root 55 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/1] root 56 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/2] root 57 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/3] root 58 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/4] root 59 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/5] root 60 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/6] root 61 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/7] root 62 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/8] root 63 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/9] root 64 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/10] root 65 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/11] root 66 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/12] root 67 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/13] root 68 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/14] root 69 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/15] root 70 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/0] root 71 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/1] root 72 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/2] root 73 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/3] root 74 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/4] root 75 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/5] root 76 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/6] root 77 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/7] root 78 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/8] root 79 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/9] root 80 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/10] root 81 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/11] root 82 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/12] root 83 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/13] root 84 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/14] root 85 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/15] root 86 0.0 0.0 0 0 ? S< 08:28 0:00 [kacpid] root 87 0.0 0.0 0 0 ? S< 08:28 0:00 [kacpi_notify] root 88 0.0 0.0 0 0 ? S< 08:28 0:00 [cqueue] root 89 0.0 0.0 0 0 ? S< 08:28 0:00 [kseriod] root 90 0.0 0.0 0 0 ? S 08:28 0:00 [pdflush] root 91 0.0 0.0 0 0 ? S 08:28 0:00 [pdflush] root 92 0.0 0.0 0 0 ? S< 08:28 0:01 [kswapd0] root 93 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/0] root 94 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/1] root 95 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/2] root 96 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/3] root 97 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/4] root 98 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/5] root 99 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/6] root 100 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/7] root 101 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/8] root 102 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/9] root 103 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/10] root 104 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/11] root 105 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/12] root 106 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/13] root 107 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/14] root 108 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/15] root 109 0.0 0.0 0 0 ? S< 08:28 0:00 [kpsmoused] root 250 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/0] root 251 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/1] root 252 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/2] root 253 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/3] root 254 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/4] root 255 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/5] root 256 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/6] root 257 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/7] root 258 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/8] root 259 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/9] root 260 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/10] root 261 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/11] root 262 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/12] root 263 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/13] root 264 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/14] root 265 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/15] root 266 0.0 0.0 0 0 ? S< 08:28 0:00 [ata_aux] root 287 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_eh_0] root 288 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_eh_1] root 440 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/0] root 441 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/1] root 442 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/2] root 443 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/3] root 444 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/4] root 445 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/5] root 446 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/6] root 447 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/7] root 448 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/8] root 449 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/9] root 450 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/10] root 451 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/11] root 452 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/12] root 453 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/13] root 454 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/14] root 455 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/15] root 687 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/0] root 689 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/1] root 690 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/2] root 691 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/3] root 692 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/4] root 693 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/5] root 694 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/6] root 695 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/7] root 697 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/8] root 698 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/9] root 699 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/10] root 700 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/11] root 701 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/12] root 702 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/13] root 703 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/14] root 704 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/15] root 779 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_eh_2] root 867 0.0 0.0 0 0 ? S< 08:28 0:00 [ksuspend_usbd] root 869 0.0 0.0 0 0 ? S< 08:28 0:00 [khubd] root 1157 0.0 0.0 0 0 ? S< 08:28 0:00 [qla2xxx_2_dpc] root 1158 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_wq_2] root 1159 0.0 0.0 0 0 ? S< 08:28 0:00 [fc_wq_2] root 1160 0.0 0.0 0 0 ? S< 08:28 0:00 [fc_dl_2] root 1366 0.0 0.0 4760 912 pts/0 R+ 12:58 0:00 ps aux root 1527 0.0 0.0 0 0 ? S< 08:28 0:01 [kjournald] root 1600 0.0 0.0 17272 1404 ? S<s 08:28 0:00 /sbin/udevd --daemon root 2799 0.2 0.0 0 0 ? SN 08:28 0:45 [kipmi0] root 2864 0.0 0.0 0 0 ? S< 08:28 0:00 [kstriped] root 3600 0.0 0.0 19116 896 ? Ss 08:28 0:00 /sbin/syslog-ng root 3603 0.0 0.0 4092 688 ? Ss 08:28 0:00 /sbin/klogd -c 1 -x 100 3612 0.0 0.0 21500 1144 ? Ss 08:28 0:00 /bin/dbus-daemon --system root 3684 0.0 0.0 3824 596 ? Ss 08:28 0:00 /sbin/acpid 101 3880 0.0 0.1 34404 5160 ? Ss 08:28 0:01 /usr/sbin/hald --daemon=yes root 3932 0.0 0.0 98404 2348 ? Ssl 08:28 0:00 /usr/sbin/console-kit-daemon root 3935 0.0 0.0 17848 1232 ? S 08:28 0:00 hald-runner root 4290 0.0 0.0 24100 1224 ? S 08:28 0:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event2 /dev/input/event4 root 4380 0.0 0.0 24104 1224 ? S 08:28 0:00 hald-addon-storage: polling /dev/sr0 (every 16 sec) 101 4384 0.0 0.0 25856 1144 ? S 08:28 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket root 4575 0.0 0.0 29316 836 ? S<sl 08:28 0:00 /sbin/auditd -s disable root 4579 0.0 0.0 14248 784 ? S<sl 08:28 0:00 /sbin/audispd root 4580 0.0 0.0 0 0 ? S< 08:28 0:00 [kauditd] root 4610 0.0 0.0 18712 680 ? Ss 08:28 0:00 /sbin/rpcbind root 4699 0.0 0.0 0 0 ? S< 08:28 0:00 [user_dlm] root 4707 0.0 0.0 0 0 ? S< 08:28 0:01 [o2net] root 5088 0.0 0.0 8908 540 ? Ss 08:28 0:06 /usr/sbin/irqbalance root 5098 0.0 0.0 172516 1196 ? Ssl 08:28 0:00 /usr/sbin/nscd root 5168 0.0 0.0 0 0 ? S< 08:28 0:00 [o2hb-D255FD5483] root 5258 0.0 0.0 65876 2632 ? Ss 08:28 0:00 /usr/sbin/cupsd root 5274 0.0 0.0 51728 1224 ? Ss 08:28 0:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid root 5557 0.0 0.0 37920 2220 ? Ss 08:28 0:00 /usr/lib/postfix/master root 5587 0.0 0.0 14608 632 ? Ss 08:28 0:00 /usr/sbin/cron postfix 5589 0.0 0.0 39816 2204 ? S 08:28 0:00 qmgr -l -t fifo -u root 5608 0.6 0.0 8740 1284 ? S 08:28 1:42 xenstored --pid-file /var/run/xenstore.pid root 5613 0.0 0.2 107500 11504 ? S 08:28 0:00 /usr/bin/python2.6 /usr/sbin/xend start root 5614 1.4 0.7 442992 30484 ? Sl 08:28 3:54 /usr/bin/python2.6 /usr/sbin/xend start root 5615 0.0 0.0 20640 732 ? Ssl 08:28 0:00 blktapctrl root 5618 0.0 0.0 88252 660 ? Sl 08:28 0:00 xenconsoled root 5693 0.0 0.2 103392 11100 ? S 08:28 0:00 python /usr/lib64/python2.6/site-packages/xen/xend/server/HalDaemon.py root 5708 0.1 0.1 218620 5144 ? Sl 08:28 0:24 /usr/sbin/libvirtd -d -l root 5768 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2_wq] root 5770 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2dc] root 5771 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_thread] root 5772 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_reco_thread] root 5773 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_wq] root 5774 0.0 0.0 0 0 ? S< 08:28 0:10 [kjournald2] root 5775 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2cmt] root 5779 0.0 0.0 0 0 ? S< 08:28 0:00 [o2hb-A773AA5249] root 5780 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2dc] root 5781 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_thread] root 5782 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_reco_thread] root 5783 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_wq] root 5784 0.0 0.0 0 0 ? S< 08:28 0:01 [kjournald2] root 5785 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2cmt] root 5801 0.0 0.0 4316 724 tty1 Ss+ 08:28 0:00 /sbin/mingetty --noclear tty1 root 5802 0.0 0.0 4316 704 tty2 Ss+ 08:28 0:00 /sbin/mingetty tty2 root 5803 0.0 0.0 4316 704 tty3 Ss+ 08:28 0:00 /sbin/mingetty tty3 root 5804 0.0 0.0 4316 708 tty4 Ss+ 08:28 0:00 /sbin/mingetty tty4 root 5805 0.0 0.0 4316 704 tty5 Ss+ 08:28 0:00 /sbin/mingetty tty5 root 5806 0.0 0.0 4316 708 tty6 Ss+ 08:28 0:00 /sbin/mingetty tty6 root 5855 0.0 0.0 79912 3232 ? Rs 08:29 0:00 sshd: root@pts/0 root 5861 0.0 0.0 13616 2844 pts/0 Ss 08:29 0:00 -bash root 5899 0.6 0.6 88776 27676 pts/0 S 08:29 1:37 Xvnc :49 -desktop X -httpd /usr/share/vnc/classes -auth /root/.Xauthority -geometry 1024x768 -depth 24 -rfbwait 120000 -rfbauth /root/.vnc/passwd -rfbport 5949 - root 5910 0.0 0.1 63760 8028 pts/0 S 08:29 0:00 xterm -geometry 80x24+10+10 -ls -title X Desktop root 5911 0.0 0.2 209464 9260 pts/0 Sl 08:29 0:00 gnome-session root 5913 0.0 0.0 13616 2788 pts/1 Ss+ 08:29 0:00 -bash root 5928 0.0 0.0 22024 756 pts/0 S 08:29 0:00 dbus-launch --exit-with-session gnome-session root 5929 0.0 0.0 13408 1076 ? Ss 08:29 0:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session root 5932 0.0 0.1 39032 4964 ? S 08:29 0:00 /usr/lib/GConf/2/gconfd-2 root 5935 0.0 0.0 107620 4104 pts/0 S 08:29 0:00 /usr/lib64/gnome-session/helpers/gnome-keyring-daemon-wrapper root 5936 0.0 0.6 301588 28964 ? Ssl 08:29 0:01 /usr/lib/gnome-settings-daemon/gnome-settings-daemon root 5940 0.0 0.0 66556 2320 pts/0 S 08:29 0:00 /usr/bin/gnome-keyring-daemon root 5942 0.0 0.2 148268 11964 pts/0 S 08:29 0:00 /usr/bin/metacity root 5948 0.0 0.4 360768 19892 pts/0 S 08:29 0:01 gnome-panel root 5949 0.1 1.1 499372 48484 pts/0 S 08:29 0:24 nautilus --no-desktop --browser root 5951 0.0 0.0 40864 2284 ? S 08:29 0:00 /usr/lib64/gvfs/gvfsd root 5953 0.0 0.0 211472 3872 ? Ssl 08:29 0:00 /usr/lib/bonobo/bonobo-activation-server --ac-activate --ior-output-fd=17 root 5959 0.0 0.0 62356 2296 ? Ssl 08:29 0:00 /usr/lib64/gvfs//gvfs-fuse-daemon /root/.gvfs root 5971 0.0 0.4 221680 19064 pts/0 S 08:29 0:00 python /usr/lib64/python2.6/site-packages/system-config-printer/applet.py root 5984 0.0 0.2 216268 9616 ? Ss 08:29 0:00 gnome-power-manager root 5989 0.0 0.0 182788 3044 ? S<sl 08:29 0:00 /usr/bin/pulseaudio --start root 5998 0.0 0.0 41332 3120 ? S 08:29 0:00 /usr/lib64/gvfs/gvfs-hal-volume-monitor root 6002 0.0 0.0 38904 2640 ? S 08:29 0:00 /usr/lib64/gvfs/gvfs-gphoto2-volume-monitor root 6005 0.0 0.0 117232 2908 ? S 08:29 0:00 /usr/lib64/gvfs/gvfsd-trash --spawner :1.11 /org/gtk/gvfs/exec_spaw/0 root 6021 0.0 0.1 203308 6568 ? Ss 08:29 0:00 gnome-screensaver root 6028 0.0 0.4 311496 17552 ? S 08:29 0:02 /usr/lib/gnome-main-menu/main-menu --oaf-activate-iid=OAFIID:GNOME_MainMenu_Factory --oaf-ior-fd=21 root 6031 0.0 0.3 354104 16092 ? Sl 08:29 0:10 /usr/lib/gnome-panel/mixer_applet2 --oaf-activate-iid=OAFIID:GNOME_MixerApplet_Factory --oaf-ior-fd=27 root 6034 0.0 0.0 41076 2308 ? S 08:29 0:00 /usr/lib64/gvfs/gvfsd-burn --spawner :1.11 /org/gtk/gvfs/exec_spaw/1 root 6050 2.4 1.1 526332 46572 ? Ssl 08:29 6:38 python /usr/share/virt-manager/virt-manager.py -c xen:/// root 6260 1.1 0.1 91184 5308 ? Sl 08:29 3:01 /usr/lib/xen/bin/qemu-dm -d 2 -domain-name csg01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tabl root 6639 0.0 0.0 0 0 ? S< 08:29 0:00 [loop0] root 6853 0.8 0.1 94740 7252 ? Sl 08:29 2:18 /usr/lib/xen/bin/qemu-dm -d 3 -domain-name cwi01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tabl root 6981 0.0 0.0 0 0 ? S< 08:29 0:00 [loop1] root 7123 0.1 0.0 85272 3664 ? Sl 08:29 0:29 /usr/lib64/xen/bin/qemu-dm -d 4 -domain-name dsfw -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv root 7384 0.1 0.0 0 0 ? S< 08:29 0:18 [loop10] root 7483 0.0 0.0 0 0 ? S< 08:30 0:03 [blkback.4.xvda] root 7823 0.6 0.1 93304 7056 ? Sl 08:30 1:44 /usr/lib/xen/bin/qemu-dm -d 5 -domain-name ram01-tri -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr root 8131 0.0 0.0 0 0 ? S< 08:30 0:00 [loop11] root 8304 0.0 0.0 0 0 ? S< 08:30 0:00 [loop12] root 8468 0.9 0.1 95252 7804 ? Sl 08:30 2:29 /usr/lib/xen/bin/qemu-dm -d 6 -domain-name vfs01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tabl root 8766 0.0 0.0 0 0 ? S< 08:30 0:00 [loop13] root 8944 0.0 0.0 0 0 ? S< 08:30 0:00 [loop14] root 9072 0.9 0.1 91672 4708 ? Sl 08:30 2:29 /usr/lib/xen/bin/qemu-dm -d 7 -domain-name webapp -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tab root 9384 0.0 0.0 0 0 ? S< 08:30 0:00 [loop15] root 9574 0.0 0.0 0 0 ? S< 08:30 0:00 [loop16] root 9685 0.3 0.1 81964 4372 ? Sl 08:30 0:56 /usr/lib/xen/bin/qemu-dm -d 8 -domain-name vre01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00: root 10023 0.0 0.0 0 0 ? S< 08:30 0:02 [loop17] root 10226 0.0 0.0 0 0 ? S< 08:30 0:00 [loop18] root 10327 0.3 0.1 81964 4372 ? Sl 08:30 0:59 /usr/lib/xen/bin/qemu-dm -d 9 -domain-name vre03 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00: root 10658 0.0 0.0 0 0 ? S< 08:30 0:02 [loop19] root 10886 0.0 0.0 0 0 ? S< 08:30 0:00 [loop2] root 10991 0.3 0.1 81964 4376 ? Sl 08:30 0:58 /usr/lib/xen/bin/qemu-dm -d 10 -domain-name vre05 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00 root 11327 0.0 0.0 0 0 ? S< 08:30 0:02 [loop20] root 11566 0.0 0.0 0 0 ? S< 08:30 0:00 [loop21] root 11750 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.8.hda] root 11751 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.9.hda] root 11752 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.9.hdb] root 11753 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.8.hdb] root 11835 1.0 0.2 96496 8952 ? Sl 08:31 2:56 /usr/lib/xen/bin/qemu-dm -d 11 -domain-name nad01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tab root 12177 0.0 0.0 0 0 ? S< 08:31 0:00 [loop22] root 12432 0.0 0.0 0 0 ? S< 08:31 0:00 [loop23] root 12517 0.8 0.1 90916 5292 ? Sl 08:31 2:19 /usr/lib/xen/bin/qemu-dm -d 12 -domain-name tsl02 -videoram 4 -k en-us -vnc 127.0.0.1:0 -vncunused -vcpus 4 -boot c -localtime -serial pty -acpi -usb -usbd root 12850 0.0 0.0 0 0 ? S< 08:31 0:00 [loop24] root 12903 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.10.hda] root 12905 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.10.hdb] root 13761 0.1 0.0 85272 3660 ? Sl 08:32 0:29 /usr/lib64/xen/bin/qemu-dm -d 14 -domain-name rly01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv root 14100 0.0 0.0 0 0 ? S< 08:32 0:13 [loop27] root 14417 0.0 0.0 0 0 ? S< 08:32 0:00 [loop28] root 14435 0.0 0.0 0 0 ? S< 08:32 0:02 [blkback.14.xvda] root 14436 0.0 0.0 0 0 ? S< 08:32 0:00 [blkback.14.xvdb] root 14570 0.1 0.0 85148 3656 ? Sl 08:32 0:29 /usr/lib64/xen/bin/qemu-dm -d 15 -domain-name hpd02 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv root 14934 0.1 0.0 0 0 ? S< 08:32 0:19 [loop29] root 14973 0.0 0.0 0 0 ? S< 08:32 0:02 [blkback.15.xvda] root 15338 0.1 0.0 85148 3652 ? Sl 08:34 0:29 /usr/lib64/xen/bin/qemu-dm -d 16 -domain-name ids -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv root 15714 0.0 0.0 0 0 ? S< 08:34 0:07 [loop25] root 15751 0.0 0.0 0 0 ? S< 08:34 0:00 [blkback.16.xvda] root 16041 0.8 0.1 94628 7052 ? Sl 08:34 2:17 /usr/lib/xen/bin/qemu-dm -d 17 -domain-name pkt01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tablet root 16419 0.0 0.0 0 0 ? S< 08:34 0:00 [loop26] root 16716 0.0 0.0 0 0 ? S< 08:34 0:00 [loop3] postfix 20520 0.0 0.0 39768 2172 ? S 11:47 0:00 pickup -l -t fifo -u Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-01 20:51 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Fri, Jan 01, 2010 at 03:06:03PM -0500, James Pifer wrote:> > You really need to start monitoring the server/dom0 in more detail. > > > > Are you using file: backend for domU disks, or tap:aio: ? > > > > -- Pasi > > The domUs are all using file based storage. The memory did not fill up > until I started the domUs.Yes, exactly. When you start using file: backed domUs, then the dom0 Linux kernel will start caching the guest disk accesses into the memory. This is completely normal, and it''s exactly what should happen. Like said, dom0 Linux kernel will use as much memory for caching as there''s available to dom0 - so caching is not a reason to increase dom0 memory. 2048 MB definitely is enough for dom0.> I can''t tell what''s using the ram. Here is an > output of ps aux: >It seems gnome and virt-manager are using most of the memory in use. You should run "top" and sort by memory usage (shift+m), and monitor how it goes. Memory used for caching is not really ''used'' memory, because it''ll be available to application immediately when they need it. So you should monitor the memory usage, and see if there''s some application that starts to use the dom0 memory for real. Since this seems to be desktop, it could be X server, firefox, or whatever applicaton leaking memory. I can only recommend you to monitor the memory usage. Earlier you had all the memory plus all the swap in use - so some program clearly had eaten it all. -- Pasi> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 1 0.0 0.0 1064 384 ? Ss 08:28 0:00 init [3] > root 2 0.0 0.0 0 0 ? S< 08:28 0:00 [kthreadd] > root 3 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/0] > root 4 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/0] > root 5 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/1] > root 6 0.0 0.0 0 0 ? S< 08:28 0:03 [ksoftirqd/1] > root 7 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/2] > root 8 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/2] > root 9 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/3] > root 10 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/3] > root 11 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/4] > root 12 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/4] > root 13 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/5] > root 14 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/5] > root 15 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/6] > root 16 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/6] > root 17 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/7] > root 18 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/7] > root 19 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/8] > root 20 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/8] > root 21 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/9] > root 22 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/9] > root 23 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/10] > root 24 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/10] > root 25 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/11] > root 26 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/11] > root 27 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/12] > root 28 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/12] > root 29 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/13] > root 30 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/13] > root 31 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/14] > root 32 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/14] > root 33 0.0 0.0 0 0 ? S< 08:28 0:00 [migration/15] > root 34 0.0 0.0 0 0 ? S< 08:28 0:00 [ksoftirqd/15] > root 35 0.0 0.0 0 0 ? S< 08:28 0:00 [events/0] > root 36 0.0 0.0 0 0 ? S< 08:28 0:00 [events/1] > root 37 0.0 0.0 0 0 ? S< 08:28 0:00 [events/2] > root 38 0.0 0.0 0 0 ? S< 08:28 0:00 [events/3] > root 39 0.0 0.0 0 0 ? S< 08:28 0:00 [events/4] > root 40 0.0 0.0 0 0 ? S< 08:28 0:00 [events/5] > root 41 0.0 0.0 0 0 ? S< 08:28 0:00 [events/6] > root 42 0.0 0.0 0 0 ? S< 08:28 0:00 [events/7] > root 43 0.0 0.0 0 0 ? S< 08:28 0:00 [events/8] > root 44 0.0 0.0 0 0 ? S< 08:28 0:00 [events/9] > root 45 0.0 0.0 0 0 ? S< 08:28 0:00 [events/10] > root 46 0.0 0.0 0 0 ? S< 08:28 0:00 [events/11] > root 47 0.0 0.0 0 0 ? S< 08:28 0:00 [events/12] > root 48 0.0 0.0 0 0 ? S< 08:28 0:00 [events/13] > root 49 0.0 0.0 0 0 ? S< 08:28 0:00 [events/14] > root 50 0.0 0.0 0 0 ? S< 08:28 0:00 [events/15] > root 51 0.0 0.0 0 0 ? S< 08:28 0:00 [khelper] > root 52 0.0 0.0 0 0 ? S< 08:28 0:00 [xenwatch] > root 53 0.0 0.0 0 0 ? S< 08:28 0:00 [xenbus] > root 54 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/0] > root 55 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/1] > root 56 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/2] > root 57 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/3] > root 58 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/4] > root 59 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/5] > root 60 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/6] > root 61 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/7] > root 62 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/8] > root 63 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/9] > root 64 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/10] > root 65 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/11] > root 66 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/12] > root 67 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/13] > root 68 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/14] > root 69 0.0 0.0 0 0 ? S< 08:28 0:00 [kintegrityd/15] > root 70 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/0] > root 71 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/1] > root 72 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/2] > root 73 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/3] > root 74 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/4] > root 75 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/5] > root 76 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/6] > root 77 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/7] > root 78 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/8] > root 79 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/9] > root 80 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/10] > root 81 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/11] > root 82 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/12] > root 83 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/13] > root 84 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/14] > root 85 0.0 0.0 0 0 ? S< 08:28 0:00 [kblockd/15] > root 86 0.0 0.0 0 0 ? S< 08:28 0:00 [kacpid] > root 87 0.0 0.0 0 0 ? S< 08:28 0:00 [kacpi_notify] > root 88 0.0 0.0 0 0 ? S< 08:28 0:00 [cqueue] > root 89 0.0 0.0 0 0 ? S< 08:28 0:00 [kseriod] > root 90 0.0 0.0 0 0 ? S 08:28 0:00 [pdflush] > root 91 0.0 0.0 0 0 ? S 08:28 0:00 [pdflush] > root 92 0.0 0.0 0 0 ? S< 08:28 0:01 [kswapd0] > root 93 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/0] > root 94 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/1] > root 95 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/2] > root 96 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/3] > root 97 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/4] > root 98 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/5] > root 99 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/6] > root 100 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/7] > root 101 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/8] > root 102 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/9] > root 103 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/10] > root 104 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/11] > root 105 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/12] > root 106 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/13] > root 107 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/14] > root 108 0.0 0.0 0 0 ? S< 08:28 0:00 [aio/15] > root 109 0.0 0.0 0 0 ? S< 08:28 0:00 [kpsmoused] > root 250 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/0] > root 251 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/1] > root 252 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/2] > root 253 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/3] > root 254 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/4] > root 255 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/5] > root 256 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/6] > root 257 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/7] > root 258 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/8] > root 259 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/9] > root 260 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/10] > root 261 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/11] > root 262 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/12] > root 263 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/13] > root 264 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/14] > root 265 0.0 0.0 0 0 ? S< 08:28 0:00 [ata/15] > root 266 0.0 0.0 0 0 ? S< 08:28 0:00 [ata_aux] > root 287 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_eh_0] > root 288 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_eh_1] > root 440 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/0] > root 441 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/1] > root 442 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/2] > root 443 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/3] > root 444 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/4] > root 445 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/5] > root 446 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/6] > root 447 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/7] > root 448 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/8] > root 449 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/9] > root 450 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/10] > root 451 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/11] > root 452 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/12] > root 453 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/13] > root 454 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/14] > root 455 0.0 0.0 0 0 ? S< 08:28 0:00 [net_accel/15] > root 687 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/0] > root 689 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/1] > root 690 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/2] > root 691 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/3] > root 692 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/4] > root 693 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/5] > root 694 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/6] > root 695 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/7] > root 697 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/8] > root 698 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/9] > root 699 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/10] > root 700 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/11] > root 701 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/12] > root 702 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/13] > root 703 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/14] > root 704 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_tgtd/15] > root 779 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_eh_2] > root 867 0.0 0.0 0 0 ? S< 08:28 0:00 [ksuspend_usbd] > root 869 0.0 0.0 0 0 ? S< 08:28 0:00 [khubd] > root 1157 0.0 0.0 0 0 ? S< 08:28 0:00 [qla2xxx_2_dpc] > root 1158 0.0 0.0 0 0 ? S< 08:28 0:00 [scsi_wq_2] > root 1159 0.0 0.0 0 0 ? S< 08:28 0:00 [fc_wq_2] > root 1160 0.0 0.0 0 0 ? S< 08:28 0:00 [fc_dl_2] > root 1366 0.0 0.0 4760 912 pts/0 R+ 12:58 0:00 ps aux > root 1527 0.0 0.0 0 0 ? S< 08:28 0:01 [kjournald] > root 1600 0.0 0.0 17272 1404 ? S<s 08:28 0:00 /sbin/udevd --daemon > root 2799 0.2 0.0 0 0 ? SN 08:28 0:45 [kipmi0] > root 2864 0.0 0.0 0 0 ? S< 08:28 0:00 [kstriped] > root 3600 0.0 0.0 19116 896 ? Ss 08:28 0:00 /sbin/syslog-ng > root 3603 0.0 0.0 4092 688 ? Ss 08:28 0:00 /sbin/klogd -c 1 -x > 100 3612 0.0 0.0 21500 1144 ? Ss 08:28 0:00 /bin/dbus-daemon --system > root 3684 0.0 0.0 3824 596 ? Ss 08:28 0:00 /sbin/acpid > 101 3880 0.0 0.1 34404 5160 ? Ss 08:28 0:01 /usr/sbin/hald --daemon=yes > root 3932 0.0 0.0 98404 2348 ? Ssl 08:28 0:00 /usr/sbin/console-kit-daemon > root 3935 0.0 0.0 17848 1232 ? S 08:28 0:00 hald-runner > root 4290 0.0 0.0 24100 1224 ? S 08:28 0:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event2 /dev/input/event4 > root 4380 0.0 0.0 24104 1224 ? S 08:28 0:00 hald-addon-storage: polling /dev/sr0 (every 16 sec) > 101 4384 0.0 0.0 25856 1144 ? S 08:28 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket > root 4575 0.0 0.0 29316 836 ? S<sl 08:28 0:00 /sbin/auditd -s disable > root 4579 0.0 0.0 14248 784 ? S<sl 08:28 0:00 /sbin/audispd > root 4580 0.0 0.0 0 0 ? S< 08:28 0:00 [kauditd] > root 4610 0.0 0.0 18712 680 ? Ss 08:28 0:00 /sbin/rpcbind > root 4699 0.0 0.0 0 0 ? S< 08:28 0:00 [user_dlm] > root 4707 0.0 0.0 0 0 ? S< 08:28 0:01 [o2net] > root 5088 0.0 0.0 8908 540 ? Ss 08:28 0:06 /usr/sbin/irqbalance > root 5098 0.0 0.0 172516 1196 ? Ssl 08:28 0:00 /usr/sbin/nscd > root 5168 0.0 0.0 0 0 ? S< 08:28 0:00 [o2hb-D255FD5483] > root 5258 0.0 0.0 65876 2632 ? Ss 08:28 0:00 /usr/sbin/cupsd > root 5274 0.0 0.0 51728 1224 ? Ss 08:28 0:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid > root 5557 0.0 0.0 37920 2220 ? Ss 08:28 0:00 /usr/lib/postfix/master > root 5587 0.0 0.0 14608 632 ? Ss 08:28 0:00 /usr/sbin/cron > postfix 5589 0.0 0.0 39816 2204 ? S 08:28 0:00 qmgr -l -t fifo -u > root 5608 0.6 0.0 8740 1284 ? S 08:28 1:42 xenstored --pid-file /var/run/xenstore.pid > root 5613 0.0 0.2 107500 11504 ? S 08:28 0:00 /usr/bin/python2.6 /usr/sbin/xend start > root 5614 1.4 0.7 442992 30484 ? Sl 08:28 3:54 /usr/bin/python2.6 /usr/sbin/xend start > root 5615 0.0 0.0 20640 732 ? Ssl 08:28 0:00 blktapctrl > root 5618 0.0 0.0 88252 660 ? Sl 08:28 0:00 xenconsoled > root 5693 0.0 0.2 103392 11100 ? S 08:28 0:00 python /usr/lib64/python2.6/site-packages/xen/xend/server/HalDaemon.py > root 5708 0.1 0.1 218620 5144 ? Sl 08:28 0:24 /usr/sbin/libvirtd -d -l > root 5768 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2_wq] > root 5770 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2dc] > root 5771 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_thread] > root 5772 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_reco_thread] > root 5773 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_wq] > root 5774 0.0 0.0 0 0 ? S< 08:28 0:10 [kjournald2] > root 5775 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2cmt] > root 5779 0.0 0.0 0 0 ? S< 08:28 0:00 [o2hb-A773AA5249] > root 5780 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2dc] > root 5781 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_thread] > root 5782 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_reco_thread] > root 5783 0.0 0.0 0 0 ? S< 08:28 0:00 [dlm_wq] > root 5784 0.0 0.0 0 0 ? S< 08:28 0:01 [kjournald2] > root 5785 0.0 0.0 0 0 ? S< 08:28 0:00 [ocfs2cmt] > root 5801 0.0 0.0 4316 724 tty1 Ss+ 08:28 0:00 /sbin/mingetty --noclear tty1 > root 5802 0.0 0.0 4316 704 tty2 Ss+ 08:28 0:00 /sbin/mingetty tty2 > root 5803 0.0 0.0 4316 704 tty3 Ss+ 08:28 0:00 /sbin/mingetty tty3 > root 5804 0.0 0.0 4316 708 tty4 Ss+ 08:28 0:00 /sbin/mingetty tty4 > root 5805 0.0 0.0 4316 704 tty5 Ss+ 08:28 0:00 /sbin/mingetty tty5 > root 5806 0.0 0.0 4316 708 tty6 Ss+ 08:28 0:00 /sbin/mingetty tty6 > root 5855 0.0 0.0 79912 3232 ? Rs 08:29 0:00 sshd: root@pts/0 > root 5861 0.0 0.0 13616 2844 pts/0 Ss 08:29 0:00 -bash > root 5899 0.6 0.6 88776 27676 pts/0 S 08:29 1:37 Xvnc :49 -desktop X -httpd /usr/share/vnc/classes -auth /root/.Xauthority -geometry 1024x768 -depth 24 -rfbwait 120000 -rfbauth /root/.vnc/passwd -rfbport 5949 - > root 5910 0.0 0.1 63760 8028 pts/0 S 08:29 0:00 xterm -geometry 80x24+10+10 -ls -title X Desktop > root 5911 0.0 0.2 209464 9260 pts/0 Sl 08:29 0:00 gnome-session > root 5913 0.0 0.0 13616 2788 pts/1 Ss+ 08:29 0:00 -bash > root 5928 0.0 0.0 22024 756 pts/0 S 08:29 0:00 dbus-launch --exit-with-session gnome-session > root 5929 0.0 0.0 13408 1076 ? Ss 08:29 0:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session > root 5932 0.0 0.1 39032 4964 ? S 08:29 0:00 /usr/lib/GConf/2/gconfd-2 > root 5935 0.0 0.0 107620 4104 pts/0 S 08:29 0:00 /usr/lib64/gnome-session/helpers/gnome-keyring-daemon-wrapper > root 5936 0.0 0.6 301588 28964 ? Ssl 08:29 0:01 /usr/lib/gnome-settings-daemon/gnome-settings-daemon > root 5940 0.0 0.0 66556 2320 pts/0 S 08:29 0:00 /usr/bin/gnome-keyring-daemon > root 5942 0.0 0.2 148268 11964 pts/0 S 08:29 0:00 /usr/bin/metacity > root 5948 0.0 0.4 360768 19892 pts/0 S 08:29 0:01 gnome-panel > root 5949 0.1 1.1 499372 48484 pts/0 S 08:29 0:24 nautilus --no-desktop --browser > root 5951 0.0 0.0 40864 2284 ? S 08:29 0:00 /usr/lib64/gvfs/gvfsd > root 5953 0.0 0.0 211472 3872 ? Ssl 08:29 0:00 /usr/lib/bonobo/bonobo-activation-server --ac-activate --ior-output-fd=17 > root 5959 0.0 0.0 62356 2296 ? Ssl 08:29 0:00 /usr/lib64/gvfs//gvfs-fuse-daemon /root/.gvfs > root 5971 0.0 0.4 221680 19064 pts/0 S 08:29 0:00 python /usr/lib64/python2.6/site-packages/system-config-printer/applet.py > root 5984 0.0 0.2 216268 9616 ? Ss 08:29 0:00 gnome-power-manager > root 5989 0.0 0.0 182788 3044 ? S<sl 08:29 0:00 /usr/bin/pulseaudio --start > root 5998 0.0 0.0 41332 3120 ? S 08:29 0:00 /usr/lib64/gvfs/gvfs-hal-volume-monitor > root 6002 0.0 0.0 38904 2640 ? S 08:29 0:00 /usr/lib64/gvfs/gvfs-gphoto2-volume-monitor > root 6005 0.0 0.0 117232 2908 ? S 08:29 0:00 /usr/lib64/gvfs/gvfsd-trash --spawner :1.11 /org/gtk/gvfs/exec_spaw/0 > root 6021 0.0 0.1 203308 6568 ? Ss 08:29 0:00 gnome-screensaver > root 6028 0.0 0.4 311496 17552 ? S 08:29 0:02 /usr/lib/gnome-main-menu/main-menu --oaf-activate-iid=OAFIID:GNOME_MainMenu_Factory --oaf-ior-fd=21 > root 6031 0.0 0.3 354104 16092 ? Sl 08:29 0:10 /usr/lib/gnome-panel/mixer_applet2 --oaf-activate-iid=OAFIID:GNOME_MixerApplet_Factory --oaf-ior-fd=27 > root 6034 0.0 0.0 41076 2308 ? S 08:29 0:00 /usr/lib64/gvfs/gvfsd-burn --spawner :1.11 /org/gtk/gvfs/exec_spaw/1 > root 6050 2.4 1.1 526332 46572 ? Ssl 08:29 6:38 python /usr/share/virt-manager/virt-manager.py -c xen:/// > root 6260 1.1 0.1 91184 5308 ? Sl 08:29 3:01 /usr/lib/xen/bin/qemu-dm -d 2 -domain-name csg01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tabl > root 6639 0.0 0.0 0 0 ? S< 08:29 0:00 [loop0] > root 6853 0.8 0.1 94740 7252 ? Sl 08:29 2:18 /usr/lib/xen/bin/qemu-dm -d 3 -domain-name cwi01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tabl > root 6981 0.0 0.0 0 0 ? S< 08:29 0:00 [loop1] > root 7123 0.1 0.0 85272 3664 ? Sl 08:29 0:29 /usr/lib64/xen/bin/qemu-dm -d 4 -domain-name dsfw -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv > root 7384 0.1 0.0 0 0 ? S< 08:29 0:18 [loop10] > root 7483 0.0 0.0 0 0 ? S< 08:30 0:03 [blkback.4.xvda] > root 7823 0.6 0.1 93304 7056 ? Sl 08:30 1:44 /usr/lib/xen/bin/qemu-dm -d 5 -domain-name ram01-tri -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr > root 8131 0.0 0.0 0 0 ? S< 08:30 0:00 [loop11] > root 8304 0.0 0.0 0 0 ? S< 08:30 0:00 [loop12] > root 8468 0.9 0.1 95252 7804 ? Sl 08:30 2:29 /usr/lib/xen/bin/qemu-dm -d 6 -domain-name vfs01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tabl > root 8766 0.0 0.0 0 0 ? S< 08:30 0:00 [loop13] > root 8944 0.0 0.0 0 0 ? S< 08:30 0:00 [loop14] > root 9072 0.9 0.1 91672 4708 ? Sl 08:30 2:29 /usr/lib/xen/bin/qemu-dm -d 7 -domain-name webapp -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tab > root 9384 0.0 0.0 0 0 ? S< 08:30 0:00 [loop15] > root 9574 0.0 0.0 0 0 ? S< 08:30 0:00 [loop16] > root 9685 0.3 0.1 81964 4372 ? Sl 08:30 0:56 /usr/lib/xen/bin/qemu-dm -d 8 -domain-name vre01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00: > root 10023 0.0 0.0 0 0 ? S< 08:30 0:02 [loop17] > root 10226 0.0 0.0 0 0 ? S< 08:30 0:00 [loop18] > root 10327 0.3 0.1 81964 4372 ? Sl 08:30 0:59 /usr/lib/xen/bin/qemu-dm -d 9 -domain-name vre03 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00: > root 10658 0.0 0.0 0 0 ? S< 08:30 0:02 [loop19] > root 10886 0.0 0.0 0 0 ? S< 08:30 0:00 [loop2] > root 10991 0.3 0.1 81964 4376 ? Sl 08:30 0:58 /usr/lib/xen/bin/qemu-dm -d 10 -domain-name vre05 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00 > root 11327 0.0 0.0 0 0 ? S< 08:30 0:02 [loop20] > root 11566 0.0 0.0 0 0 ? S< 08:30 0:00 [loop21] > root 11750 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.8.hda] > root 11751 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.9.hda] > root 11752 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.9.hdb] > root 11753 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.8.hdb] > root 11835 1.0 0.2 96496 8952 ? Sl 08:31 2:56 /usr/lib/xen/bin/qemu-dm -d 11 -domain-name nad01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tab > root 12177 0.0 0.0 0 0 ? S< 08:31 0:00 [loop22] > root 12432 0.0 0.0 0 0 ? S< 08:31 0:00 [loop23] > root 12517 0.8 0.1 90916 5292 ? Sl 08:31 2:19 /usr/lib/xen/bin/qemu-dm -d 12 -domain-name tsl02 -videoram 4 -k en-us -vnc 127.0.0.1:0 -vncunused -vcpus 4 -boot c -localtime -serial pty -acpi -usb -usbd > root 12850 0.0 0.0 0 0 ? S< 08:31 0:00 [loop24] > root 12903 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.10.hda] > root 12905 0.0 0.0 0 0 ? S< 08:31 0:00 [blkback.10.hdb] > root 13761 0.1 0.0 85272 3660 ? Sl 08:32 0:29 /usr/lib64/xen/bin/qemu-dm -d 14 -domain-name rly01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv > root 14100 0.0 0.0 0 0 ? S< 08:32 0:13 [loop27] > root 14417 0.0 0.0 0 0 ? S< 08:32 0:00 [loop28] > root 14435 0.0 0.0 0 0 ? S< 08:32 0:02 [blkback.14.xvda] > root 14436 0.0 0.0 0 0 ? S< 08:32 0:00 [blkback.14.xvdb] > root 14570 0.1 0.0 85148 3656 ? Sl 08:32 0:29 /usr/lib64/xen/bin/qemu-dm -d 15 -domain-name hpd02 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv > root 14934 0.1 0.0 0 0 ? S< 08:32 0:19 [loop29] > root 14973 0.0 0.0 0 0 ? S< 08:32 0:02 [blkback.15.xvda] > root 15338 0.1 0.0 85148 3652 ? Sl 08:34 0:29 /usr/lib64/xen/bin/qemu-dm -d 16 -domain-name ids -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv > root 15714 0.0 0.0 0 0 ? S< 08:34 0:07 [loop25] > root 15751 0.0 0.0 0 0 ? S< 08:34 0:00 [blkback.16.xvda] > root 16041 0.8 0.1 94628 7052 ? Sl 08:34 2:17 /usr/lib/xen/bin/qemu-dm -d 17 -domain-name pkt01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tablet > root 16419 0.0 0.0 0 0 ? S< 08:34 0:00 [loop26] > root 16716 0.0 0.0 0 0 ? S< 08:34 0:00 [loop3] > postfix 20520 0.0 0.0 39768 2172 ? S 11:47 0:00 pickup -l -t fifo -u > > Thanks, > James >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-02 00:28 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Fri, Jan 01, 2010 at 10:51:48PM +0200, Pasi Kärkkäinen wrote:> On Fri, Jan 01, 2010 at 03:06:03PM -0500, James Pifer wrote: > > > You really need to start monitoring the server/dom0 in more detail. > > > > > > Are you using file: backend for domU disks, or tap:aio: ? > > > > > > -- Pasi > > > > The domUs are all using file based storage. The memory did not fill up > > until I started the domUs. > > Yes, exactly. When you start using file: backed domUs, then the dom0 > Linux kernel will start caching the guest disk accesses into the memory. > > This is completely normal, and it''s exactly what should happen. > > Like said, dom0 Linux kernel will use as much memory for caching as > there''s available to dom0 - so caching is not a reason to increase dom0 memory. > 2048 MB definitely is enough for dom0. >Just for reference, back in the Xen 2.0 days I used to have 192 MB of RAM for dom0.. and that was enough. Nowadays I usually give dom0 512 MB of RAM, or in bigger machines 1024 MB. That is usually enough for a Xen-only server machine (without any local desktop/X use). So.. you need to make sure there''s nothing leaking memory in dom0. Xen management tools will obviously require some available memory in dom0 to work correctly. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-02 02:32 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Sat, 2010-01-02 at 02:28 +0200, Pasi Kärkkäinen wrote:> On Fri, Jan 01, 2010 at 10:51:48PM +0200, Pasi Kärkkäinen wrote: > > On Fri, Jan 01, 2010 at 03:06:03PM -0500, James Pifer wrote: > > > > You really need to start monitoring the server/dom0 in more detail. > > > > > > > > Are you using file: backend for domU disks, or tap:aio: ? > > > > > > > > -- Pasi > > > > > > The domUs are all using file based storage. The memory did not fill up > > > until I started the domUs. > > > > Yes, exactly. When you start using file: backed domUs, then the dom0 > > Linux kernel will start caching the guest disk accesses into the memory. > > > > This is completely normal, and it''s exactly what should happen. > > > > Like said, dom0 Linux kernel will use as much memory for caching as > > there''s available to dom0 - so caching is not a reason to increase dom0 memory. > > 2048 MB definitely is enough for dom0. > > > > Just for reference, back in the Xen 2.0 days I used to have 192 MB of > RAM for dom0.. and that was enough. > > Nowadays I usually give dom0 512 MB of RAM, or in bigger machines 1024 MB. > That is usually enough for a Xen-only server machine (without any local desktop/X use). > > So.. you need to make sure there''s nothing leaking memory in dom0. > Xen management tools will obviously require some available memory in dom0 > to work correctly. > > -- Pasi >Thanks for all the help. I will just have to monitor it. Nothing ever runs (at least started by me and I''m the only one ever on these boxes) except for virt-manager. Let you know if I see anything!! James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-23 03:06 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> Just for reference, back in the Xen 2.0 days I used to have 192 MB of > RAM for dom0.. and that was enough. > > Nowadays I usually give dom0 512 MB of RAM, or in bigger machines 1024 MB. > That is usually enough for a Xen-only server machine (without any local desktop/X use). > > So.. you need to make sure there''s nothing leaking memory in dom0. > Xen management tools will obviously require some available memory in dom0 > to work correctly. > > -- Pasi >Here we go again.... I wanted to start another domU on one of me xen servers tonight and I''m getting not able to allocate memory again. Check out the following stats. If anyone sees anything I''d love to hear it... Please let me know if there''s anything else I should post. Thanks, James # top top - 20:02:11 up 39 days, 9:03, 2 users, load average: 0.11, 0.85, 1.25 Tasks: 256 total, 1 running, 255 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 98.5%id, 0.5%wa, 0.0%hi, 0.1%si, 0.8%st Mem: 2095616k total, 1689848k used, 405768k free, 11624k buffers Swap: 2104472k total, 1476856k used, 627616k free, 406620k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8351 root 20 0 1098m 22m 928 S 1 1.1 459:39.03 qemu-dm 5610 root 20 0 1859m 988m 1604 S 1 48.3 422:09.78 xend 10448 root 20 0 1537m 29m 896 S 1 1.4 458:10.77 qemu-dm 6 root 15 -5 0 0 0 S 0 0.0 78:19.80 ksoftirqd/1 12 root 15 -5 0 0 0 S 0 0.0 10:07.34 ksoftirqd/4 34 root 15 -5 0 0 0 S 0 0.0 10:15.54 ksoftirqd/15 5074 root 20 0 92884 980 808 S 0 0.0 42:25.98 qemu-dm 11674 root 20 0 88600 768 656 S 0 0.0 27:43.75 qemu-dm 13009 root 20 0 89140 972 804 S 0 0.0 66:57.23 qemu-dm 1 root 20 0 1064 84 52 S 0 0.0 0:29.88 init 2 root 15 -5 0 0 0 S 0 0.0 0:00.02 kthreadd 3 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/0 4 root 15 -5 0 0 0 S 0 0.0 33:37.15 ksoftirqd/0 5 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/1 7 root RT -5 0 0 0 S 0 0.0 0:00.30 migration/2 8 root 15 -5 0 0 0 S 0 0.0 56:19.57 ksoftirqd/2 9 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/3 10 root 15 -5 0 0 0 S 0 0.0 31:04.94 ksoftirqd/3 11 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/4 13 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/5 14 root 15 -5 0 0 0 S 0 0.0 7:34.80 ksoftirqd/5 15 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/6 16 root 15 -5 0 0 0 S 0 0.0 5:30.04 ksoftirqd/6 17 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/7 18 root 15 -5 0 0 0 S 0 0.0 3:32.41 ksoftirqd/7 19 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/8 20 root 15 -5 0 0 0 S 0 0.0 0:02.44 ksoftirqd/8 21 root RT -5 0 0 0 S 0 0.0 0:00.20 migration/9 22 root 15 -5 0 0 0 S 0 0.0 0:03.64 ksoftirqd/9 23 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/10 24 root 15 -5 0 0 0 S 0 0.0 1:53.54 ksoftirqd/10 25 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/11 26 root 15 -5 0 0 0 S 0 0.0 0:46.29 ksoftirqd/11 27 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/12 28 root 15 -5 0 0 0 S 0 0.0 7:16.48 ksoftirqd/12 29 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/13 30 root 15 -5 0 0 0 S 0 0.0 23:07.36 ksoftirqd/13 31 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/14 32 root 15 -5 0 0 0 S 0 0.0 31:54.13 ksoftirqd/14 33 root RT -5 0 0 0 S 0 0.0 0:00.12 migration/15 35 root 15 -5 0 0 0 S 0 0.0 1:27.02 events/0 36 root 15 -5 0 0 0 S 0 0.0 1:25.28 events/1 37 root 15 -5 0 0 0 S 0 0.0 1:23.20 events/2 38 root 15 -5 0 0 0 S 0 0.0 1:23.05 events/3 39 root 15 -5 0 0 0 S 0 0.0 1:25.72 events/4 40 root 15 -5 0 0 0 S 0 0.0 1:27.81 events/5 41 root 15 -5 0 0 0 S 0 0.0 1:25.48 events/6 42 root 15 -5 0 0 0 S 0 0.0 1:35.03 events/7 43 root 15 -5 0 0 0 S 0 0.0 1:41.28 events/8 44 root 15 -5 0 0 0 S 0 0.0 1:47.56 events/9 45 root 15 -5 0 0 0 S 0 0.0 1:27.21 events/10 46 root 15 -5 0 0 0 S 0 0.0 1:35.71 events/11 47 root 15 -5 0 0 0 S 0 0.0 1:28.55 events/12 # xm info host : serverxen03 release : 2.6.27.19-5-xen version : #1 SMP 2009-02-28 04:40:21 +0100 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2533 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32757 free_memory : 15432 max_free_memory : 16966 max_para_memory : 16962 max_hvm_memory : 16914 node_to_cpu : node0:0-15 node_to_memory : node0:15432 xen_major : 3 xen_minor : 4 xen_extra : .1_19718_04-26. xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : 19718 cc_compiler : gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) cc_compile_by : abuild cc_compile_domain : cc_compile_date : Wed Oct 14 00:02:21 UTC 2009 xend_config_format : 4 # ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 1064 88 ? Ss 2009 0:29 init [3] root 2 0.0 0.0 0 0 ? S< 2009 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S< 2009 0:00 [migration/0] root 4 0.0 0.0 0 0 ? S< 2009 33:37 [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S< 2009 0:00 [migration/1] root 6 0.1 0.0 0 0 ? S< 2009 78:19 [ksoftirqd/1] root 7 0.0 0.0 0 0 ? S< 2009 0:00 [migration/2] root 8 0.0 0.0 0 0 ? S< 2009 56:19 [ksoftirqd/2] root 9 0.0 0.0 0 0 ? S< 2009 0:00 [migration/3] root 10 0.0 0.0 0 0 ? S< 2009 31:04 [ksoftirqd/3] root 11 0.0 0.0 0 0 ? S< 2009 0:00 [migration/4] root 12 0.0 0.0 0 0 ? S< 2009 10:07 [ksoftirqd/4] root 13 0.0 0.0 0 0 ? S< 2009 0:00 [migration/5] root 14 0.0 0.0 0 0 ? S< 2009 7:34 [ksoftirqd/5] root 15 0.0 0.0 0 0 ? S< 2009 0:00 [migration/6] root 16 0.0 0.0 0 0 ? S< 2009 5:30 [ksoftirqd/6] root 17 0.0 0.0 0 0 ? S< 2009 0:00 [migration/7] root 18 0.0 0.0 0 0 ? S< 2009 3:32 [ksoftirqd/7] root 19 0.0 0.0 0 0 ? S< 2009 0:00 [migration/8] root 20 0.0 0.0 0 0 ? S< 2009 0:02 [ksoftirqd/8] root 21 0.0 0.0 0 0 ? S< 2009 0:00 [migration/9] root 22 0.0 0.0 0 0 ? S< 2009 0:03 [ksoftirqd/9] root 23 0.0 0.0 0 0 ? S< 2009 0:00 [migration/10] root 24 0.0 0.0 0 0 ? S< 2009 1:53 [ksoftirqd/10] root 25 0.0 0.0 0 0 ? S< 2009 0:00 [migration/11] root 26 0.0 0.0 0 0 ? S< 2009 0:46 [ksoftirqd/11] root 27 0.0 0.0 0 0 ? S< 2009 0:00 [migration/12] root 28 0.0 0.0 0 0 ? S< 2009 7:16 [ksoftirqd/12] root 29 0.0 0.0 0 0 ? S< 2009 0:00 [migration/13] root 30 0.0 0.0 0 0 ? S< 2009 23:07 [ksoftirqd/13] root 31 0.0 0.0 0 0 ? S< 2009 0:00 [migration/14] root 32 0.0 0.0 0 0 ? S< 2009 31:54 [ksoftirqd/14] root 33 0.0 0.0 0 0 ? S< 2009 0:00 [migration/15] root 34 0.0 0.0 0 0 ? S< 2009 10:15 [ksoftirqd/15] root 35 0.0 0.0 0 0 ? S< 2009 1:27 [events/0] root 36 0.0 0.0 0 0 ? S< 2009 1:25 [events/1] root 37 0.0 0.0 0 0 ? S< 2009 1:23 [events/2] root 38 0.0 0.0 0 0 ? S< 2009 1:23 [events/3] root 39 0.0 0.0 0 0 ? S< 2009 1:25 [events/4] root 40 0.0 0.0 0 0 ? S< 2009 1:27 [events/5] root 41 0.0 0.0 0 0 ? S< 2009 1:25 [events/6] root 42 0.0 0.0 0 0 ? S< 2009 1:35 [events/7] root 43 0.0 0.0 0 0 ? S< 2009 1:41 [events/8] root 44 0.0 0.0 0 0 ? S< 2009 1:47 [events/9] root 45 0.0 0.0 0 0 ? S< 2009 1:27 [events/10] root 46 0.0 0.0 0 0 ? S< 2009 1:35 [events/11] root 47 0.0 0.0 0 0 ? S< 2009 1:28 [events/12] root 48 0.0 0.0 0 0 ? S< 2009 1:28 [events/13] root 49 0.0 0.0 0 0 ? S< 2009 1:27 [events/14] root 50 0.0 0.0 0 0 ? S< 2009 1:42 [events/15] root 51 0.0 0.0 0 0 ? S< 2009 0:00 [khelper] root 52 0.0 0.0 0 0 ? S< 2009 0:00 [xenwatch] root 53 0.0 0.0 0 0 ? S< 2009 0:00 [xenbus] root 54 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/0] root 55 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/1] root 56 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/2] root 57 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/3] root 58 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/4] root 59 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/5] root 60 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/6] root 61 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/7] root 62 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/8] root 63 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/9] root 64 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/10] root 65 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/11] root 66 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/12] root 67 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/13] root 68 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/14] root 69 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/15] root 70 0.0 0.0 0 0 ? S< 2009 0:29 [kblockd/0] root 71 0.0 0.0 0 0 ? S< 2009 0:15 [kblockd/1] root 72 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/2] root 73 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/3] root 74 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/4] root 75 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/5] root 76 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/6] root 77 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/7] root 78 0.0 0.0 0 0 ? S< 2009 0:06 [kblockd/8] root 79 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/9] root 80 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/10] root 81 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/11] root 82 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/12] root 83 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/13] root 84 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/14] root 85 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/15] root 86 0.0 0.0 0 0 ? S< 2009 0:00 [kacpid] root 87 0.0 0.0 0 0 ? S< 2009 0:00 [kacpi_notify] root 88 0.0 0.0 0 0 ? S< 2009 0:00 [cqueue] root 89 0.0 0.0 0 0 ? S< 2009 0:00 [kseriod] root 92 0.0 0.0 0 0 ? S< 2009 3:20 [kswapd0] root 93 0.0 0.0 0 0 ? S< 2009 0:00 [aio/0] root 94 0.0 0.0 0 0 ? S< 2009 0:00 [aio/1] root 95 0.0 0.0 0 0 ? S< 2009 0:00 [aio/2] root 96 0.0 0.0 0 0 ? S< 2009 0:00 [aio/3] root 97 0.0 0.0 0 0 ? S< 2009 0:00 [aio/4] root 98 0.0 0.0 0 0 ? S< 2009 0:00 [aio/5] root 99 0.0 0.0 0 0 ? S< 2009 0:00 [aio/6] root 100 0.0 0.0 0 0 ? S< 2009 0:00 [aio/7] root 101 0.0 0.0 0 0 ? S< 2009 0:00 [aio/8] root 102 0.0 0.0 0 0 ? S< 2009 0:00 [aio/9] root 103 0.0 0.0 0 0 ? S< 2009 0:00 [aio/10] root 104 0.0 0.0 0 0 ? S< 2009 0:00 [aio/11] root 105 0.0 0.0 0 0 ? S< 2009 0:00 [aio/12] root 106 0.0 0.0 0 0 ? S< 2009 0:00 [aio/13] root 107 0.0 0.0 0 0 ? S< 2009 0:00 [aio/14] root 108 0.0 0.0 0 0 ? S< 2009 0:00 [aio/15] root 109 0.0 0.0 0 0 ? S< 2009 0:00 [kpsmoused] root 267 0.0 0.0 0 0 ? S< 2009 0:14 [ata/0] root 268 0.0 0.0 0 0 ? S< 2009 0:14 [ata/1] root 269 0.0 0.0 0 0 ? S< 2009 0:00 [ata/2] root 270 0.0 0.0 0 0 ? S< 2009 0:00 [ata/3] root 271 0.0 0.0 0 0 ? S< 2009 0:00 [ata/4] root 272 0.0 0.0 0 0 ? S< 2009 0:00 [ata/5] root 273 0.0 0.0 0 0 ? S< 2009 0:00 [ata/6] root 274 0.0 0.0 0 0 ? S< 2009 0:00 [ata/7] root 275 0.0 0.0 0 0 ? S< 2009 0:00 [ata/8] root 276 0.0 0.0 0 0 ? S< 2009 0:00 [ata/9] root 277 0.0 0.0 0 0 ? S< 2009 0:00 [ata/10] root 278 0.0 0.0 0 0 ? S< 2009 0:00 [ata/11] root 279 0.0 0.0 0 0 ? S< 2009 0:00 [ata/12] root 280 0.0 0.0 0 0 ? S< 2009 0:00 [ata/13] root 281 0.0 0.0 0 0 ? S< 2009 0:00 [ata/14] root 282 0.0 0.0 0 0 ? S< 2009 0:00 [ata/15] root 283 0.0 0.0 0 0 ? S< 2009 0:00 [ata_aux] root 304 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_eh_0] root 305 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_eh_1] root 457 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/0] root 458 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/1] root 459 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/2] root 460 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/3] root 461 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/4] root 462 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/5] root 463 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/6] root 464 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/7] root 465 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/8] root 466 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/9] root 467 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/10] root 468 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/11] root 469 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/12] root 470 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/13] root 471 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/14] root 472 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/15] root 711 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/0] root 712 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/1] root 713 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/2] root 717 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/3] root 718 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/4] root 719 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/5] root 721 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/6] root 722 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/7] root 723 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/8] root 724 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/9] root 725 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/10] root 727 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/11] root 728 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/12] root 729 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/13] root 730 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/14] root 731 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/15] root 805 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_eh_2] root 878 0.0 0.0 0 0 ? S< 2009 0:00 [ksuspend_usbd] root 883 0.0 0.0 0 0 ? S< 2009 0:00 [khubd] root 1164 0.0 0.0 0 0 ? S< 2009 0:00 [qla2xxx_2_dpc] root 1165 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_wq_2] root 1166 0.0 0.0 0 0 ? S< 2009 0:00 [fc_wq_2] root 1167 0.0 0.0 0 0 ? S< 2009 0:00 [fc_dl_2] root 1525 0.0 0.0 0 0 ? S< 2009 6:04 [kjournald] root 1599 0.0 0.0 17272 416 ? S<s 2009 0:00 /sbin/udevd --daemon root 2719 0.2 0.0 0 0 ? SN 2009 154:51 [kipmi0] root 2840 0.0 0.0 0 0 ? S< 2009 0:00 [kstriped] root 3579 0.0 0.0 19376 736 ? Ss 2009 0:01 /sbin/syslog-ng root 3582 0.0 0.0 4092 616 ? Ss 2009 0:00 /sbin/klogd -c 1 -x 100 3591 0.0 0.0 21628 1024 ? Ss 2009 0:07 /bin/dbus-daemon --system root 3760 0.0 0.0 3824 376 ? Ss 2009 0:00 /sbin/acpid 101 3955 0.0 0.1 34272 2100 ? Ss 2009 0:26 /usr/sbin/hald --daemon=yes root 3991 0.0 0.0 98404 1296 ? Ssl 2009 0:01 /usr/sbin/console-kit-daemon root 3993 0.0 0.0 17848 656 ? S 2009 0:00 hald-runner root 4349 0.0 0.1 85496 3288 ? Rs 13:54 0:00 sshd: root@pts/3 root 4366 0.0 0.0 24100 636 ? S 2009 0:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event2 /dev/input/event5 root 4382 0.0 0.1 22068 3160 pts/3 Ss 13:54 0:00 -bash root 4424 0.0 0.0 24104 840 ? S 2009 2:23 hald-addon-storage: polling /dev/sr0 (every 16 sec) 101 4427 0.0 0.0 25856 668 ? S 2009 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket root 4576 0.0 0.0 29316 468 ? S<sl 2009 0:01 /sbin/auditd -s disable root 4579 0.0 0.0 14248 488 ? S<sl 2009 0:03 /sbin/audispd root 4581 0.0 0.0 0 0 ? S< 2009 0:00 [kauditd] root 4669 0.0 0.0 18712 400 ? Ss 2009 0:03 /sbin/rpcbind root 4680 0.0 0.0 0 0 ? S< 2009 0:00 [user_dlm] root 4691 0.0 0.0 0 0 ? S< 2009 3:27 [o2net] root 5074 0.3 0.0 85708 980 ? Sl Jan14 42:26 /usr/lib/xen/bin/qemu-dm -d 16 -domain-name servervre06 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:16:3e:74: root 5081 0.0 0.0 8908 424 ? Ss 2009 24:36 /usr/sbin/irqbalance root 5116 0.0 0.0 0 0 ? S< 2009 2:30 [o2hb-D255FD5483] root 5255 0.0 0.0 65876 956 ? Ss 2009 0:00 /usr/sbin/cupsd root 5257 0.0 0.0 0 0 ? S< Jan14 6:39 [loop14] root 5287 0.0 0.0 51728 588 ? Ss 2009 0:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid root 5553 0.0 0.0 37920 724 ? Ss 2009 0:14 /usr/lib/postfix/master root 5561 0.0 0.0 0 0 ? S< Jan14 1:24 [loop15] root 5583 0.0 0.0 14608 460 ? Ss 2009 0:06 /usr/sbin/cron postfix 5585 0.0 0.0 39816 716 ? S 2009 0:02 qmgr -l -t fifo -u root 5604 2.0 0.0 8612 1016 ? S 2009 1138:10 xenstored --pid-file /var/run/xenstore.pid root 5609 0.0 0.0 107496 1004 ? S 2009 0:00 /usr/bin/python2.6 /usr/sbin/xend start root 5610 0.7 48.3 1903980 1012724 ? Sl 2009 422:09 /usr/bin/python2.6 /usr/sbin/xend start root 5611 0.0 0.0 20640 484 ? Ssl 2009 0:58 blktapctrl root 5613 0.0 0.0 22716 484 ? Sl 2009 0:00 xenconsoled root 5689 0.0 0.0 103396 1616 ? S 2009 0:00 python /usr/lib64/python2.6/site-packages/xen/xend/server/HalDaemon.py root 5704 0.1 0.1 220764 3424 ? Sl 2009 91:20 /usr/sbin/libvirtd -d -l root 5761 0.0 0.0 0 0 ? S< Jan14 0:19 [blkback.16.hda] root 5764 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2_wq] root 5765 0.0 0.0 0 0 ? S< Jan14 0:40 [blkback.16.hdb] root 5766 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2dc] root 5767 0.0 0.0 0 0 ? S< 2009 0:05 [dlm_thread] root 5768 0.0 0.0 0 0 ? S< 2009 0:04 [dlm_reco_thread] root 5769 0.0 0.0 0 0 ? S< 2009 0:00 [dlm_wq] root 5770 0.0 0.0 0 0 ? S< 2009 7:34 [kjournald2] root 5771 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2cmt] root 5775 0.0 0.0 0 0 ? S< 2009 2:31 [o2hb-A773AA5249] root 5776 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2dc] root 5777 0.0 0.0 0 0 ? S< 2009 0:05 [dlm_thread] root 5778 0.0 0.0 0 0 ? S< 2009 0:03 [dlm_reco_thread] root 5779 0.0 0.0 0 0 ? S< 2009 0:00 [dlm_wq] root 5780 0.0 0.0 0 0 ? S< 2009 2:45 [kjournald2] root 5781 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2cmt] root 5799 0.0 0.0 4316 420 tty1 Ss+ 2009 0:00 /sbin/mingetty --noclear tty1 root 5800 0.0 0.0 4316 420 tty2 Ss+ 2009 0:00 /sbin/mingetty tty2 root 5801 0.0 0.0 4316 408 tty3 Ss+ 2009 0:00 /sbin/mingetty tty3 root 5802 0.0 0.0 4316 408 tty4 Ss+ 2009 0:00 /sbin/mingetty tty4 root 5803 0.0 0.0 4316 408 tty5 Ss+ 2009 0:00 /sbin/mingetty tty5 root 5804 0.0 0.0 4316 408 tty6 Ss+ 2009 0:00 /sbin/mingetty tty6 root 7736 0.0 0.0 0 0 ? S 14:01 0:00 [pdflush] root 8351 0.8 1.0 217312 22920 ? Sl 2009 459:39 /usr/lib/xen/bin/qemu-dm -d 4 -domain-name servercwi02 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tablet -net ni root 8659 0.0 0.0 0 0 ? S< 2009 0:00 [loop10] root 8821 0.0 0.0 0 0 ? S< 2009 0:00 [loop11] root 8964 0.1 0.0 85272 756 ? Sl 2009 96:45 /usr/lib64/xen/bin/qemu-dm -d 5 -domain-name servernoc03 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv root 9241 0.0 0.0 0 0 ? S< 2009 2:27 [loop12] root 9309 0.0 0.0 0 0 ? S< 2009 0:17 [blkback.5.xvda] root 10448 0.8 1.4 592144 30420 ? Sl 2009 458:11 /usr/lib/xen/bin/qemu-dm -d 6 -domain-name serverpam2 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:50:8b:c2:6a root 10722 0.0 0.0 0 0 ? S< 2009 0:00 [loop13] root 11674 0.1 0.0 86540 768 ? Sl Jan11 27:43 /usr/lib64/xen/bin/qemu-dm -d 14 -domain-name serverrly05 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv root 11887 0.0 0.0 0 0 ? S< Jan11 4:43 [loop0] root 11924 0.0 0.1 85496 3340 ? Ss 19:44 0:00 sshd: root@pts/8 root 11950 0.0 0.0 0 0 ? S< Jan11 0:35 [blkback.14.xvda] root 11962 0.0 0.1 21816 2992 pts/8 Ss+ 19:44 0:00 -bash root 13009 0.3 0.0 81964 972 ? Sl Jan07 66:57 /usr/lib/xen/bin/qemu-dm -d 12 -domain-name servervre04 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:16:3e:18: root 13185 0.0 0.0 0 0 ? S< Jan07 2:05 [loop18] root 13456 0.0 0.0 0 0 ? S< Jan07 3:24 [loop19] root 13614 0.0 0.0 0 0 ? S< Jan07 0:30 [blkback.12.hda] root 13615 0.0 0.0 0 0 ? S< Jan07 0:26 [blkback.12.hdb] root 14024 0.2 0.0 85708 976 ? Sl Jan07 65:47 /usr/lib/xen/bin/qemu-dm -d 13 -domain-name servervre02 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:16:3e:2d: root 14319 0.0 0.0 0 0 ? S< Jan07 1:39 [loop16] root 14548 0.0 0.0 0 0 ? S< Jan07 0:36 [loop17] root 14683 0.0 0.0 0 0 ? S< Jan07 0:25 [blkback.13.hda] root 14684 0.0 0.0 0 0 ? S< Jan07 0:03 [blkback.13.hdb] postfix 18434 0.0 0.1 45400 2280 ? S 19:58 0:00 pickup -l -t fifo -u root 18476 0.0 0.0 12960 1052 pts/3 R+ 20:03 0:00 ps aux root 21073 0.1 0.0 84740 760 ? Sl Jan14 20:39 /usr/lib64/xen/bin/qemu-dm -d 15 -domain-name serverhpd01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv root 21254 0.0 0.0 0 0 ? S< Jan14 2:02 [loop1] root 21288 0.0 0.0 0 0 ? S< Jan14 0:14 [blkback.15.xvda] root 27014 0.0 0.0 0 0 ? S Jan20 0:01 [pdflush] _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-23 11:59 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Fri, Jan 22, 2010 at 10:06:47PM -0500, James Pifer wrote:> > Just for reference, back in the Xen 2.0 days I used to have 192 MB of > > RAM for dom0.. and that was enough. > > > > Nowadays I usually give dom0 512 MB of RAM, or in bigger machines 1024 MB. > > That is usually enough for a Xen-only server machine (without any local desktop/X use). > > > > So.. you need to make sure there''s nothing leaking memory in dom0. > > Xen management tools will obviously require some available memory in dom0 > > to work correctly. > > > > -- Pasi > > > > Here we go again.... I wanted to start another domU on one of me xen > servers tonight and I''m getting not able to allocate memory again. Check > out the following stats. > > If anyone sees anything I''d love to hear it... Please let me know if > there''s anything else I should post. > > Thanks, > James > > > # top > > top - 20:02:11 up 39 days, 9:03, 2 users, load average: 0.11, 0.85, 1.25 > Tasks: 256 total, 1 running, 255 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 98.5%id, 0.5%wa, 0.0%hi, 0.1%si, 0.8%st > Mem: 2095616k total, 1689848k used, 405768k free, 11624k buffers > Swap: 2104472k total, 1476856k used, 627616k free, 406620k cached >xend is using 1,8 GB of memory? Is that normal? Even RES is almost 1 GB.. Please post the following logs: - "xm log" - "xm dmesg" - dom0 "dmesg" - and if there''s something related in /var/log/messages or /var/log/syslog -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-23 14:59 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Sat, Jan 23, 2010 at 10:21:09AM -0500, James Pifer wrote:> > > > > > > > > # top > > > > > > top - 20:02:11 up 39 days, 9:03, 2 users, load average: 0.11, 0.85, 1.25 > > > Tasks: 256 total, 1 running, 255 sleeping, 0 stopped, 0 zombie > > > Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 98.5%id, 0.5%wa, 0.0%hi, 0.1%si, 0.8%st > > > Mem: 2095616k total, 1689848k used, 405768k free, 11624k buffers > > > Swap: 2104472k total, 1476856k used, 627616k free, 406620k cached > > > > > > > xend is using 1,8 GB of memory? Is that normal? Even RES is almost 1 GB.. > > > > Please post the following logs: > > > > - "xm log" > > - "xm dmesg" > > - dom0 "dmesg" > > - and if there''s something related in /var/log/messages or /var/log/syslog > > > > -- Pasi > > Pasi, > > Here''s the info. I don''t have a /var/log/syslog. What is dom0 "dmesg"? I > tried that exact command and dom0 dmesg. >Just run "dmesg" command in dom0 linux, and it''ll print the dom0 kernel logs.> I didn''t see anything relevant in /var/log/messages so I attached the whole thing. > > Thanks for the help. I would love to figure out why this keeps > happening. >[2010-01-22 19:53:18 5610] ERROR (SrvBase:88) Request start failed. Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/web/SrvBase.py", line 85, in perform return op_method(op, req) File "/usr/lib64/python2.6/site-packages/xen/xend/server/SrvDomain.py", line 77, in op_start return self.xd.domain_start(self.dom.getName(), paused) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomain.py", line 1040, in domain_start dominfo.start(is_managed = True) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 459, in start XendTask.log_progress(31, 60, self._initDomain) File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2594, in _initDomain self._createDevices() File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2218, in _createDevices self.image.createDeviceModel() File "/usr/lib64/python2.6/site-packages/xen/xend/image.py", line 419, in createDeviceModel pid = os.fork() OSError: [Errno 12] Cannot allocate memory Uhm.. I wonder what memory it fails to allocate. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-23 15:21 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> > > > > > # top > > > > top - 20:02:11 up 39 days, 9:03, 2 users, load average: 0.11, 0.85, 1.25 > > Tasks: 256 total, 1 running, 255 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 98.5%id, 0.5%wa, 0.0%hi, 0.1%si, 0.8%st > > Mem: 2095616k total, 1689848k used, 405768k free, 11624k buffers > > Swap: 2104472k total, 1476856k used, 627616k free, 406620k cached > > > > xend is using 1,8 GB of memory? Is that normal? Even RES is almost 1 GB.. > > Please post the following logs: > > - "xm log" > - "xm dmesg" > - dom0 "dmesg" > - and if there''s something related in /var/log/messages or /var/log/syslog > > -- PasiPasi, Here''s the info. I don''t have a /var/log/syslog. What is dom0 "dmesg"? I tried that exact command and dom0 dmesg. I didn''t see anything relevant in /var/log/messages so I attached the whole thing. Thanks for the help. I would love to figure out why this keeps happening. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-23 16:13 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Sat, Jan 23, 2010 at 11:56:35AM -0500, James Pifer wrote:> > > > > > [2010-01-22 19:53:18 5610] ERROR (SrvBase:88) Request start failed. > > Traceback (most recent call last): > > File "/usr/lib64/python2.6/site-packages/xen/web/SrvBase.py", line > > 85, in perform > > return op_method(op, req) > > File > > "/usr/lib64/python2.6/site-packages/xen/xend/server/SrvDomain.py", > > line 77, in op_start > > return self.xd.domain_start(self.dom.getName(), paused) > > File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomain.py", > > line 1040, in domain_start > > dominfo.start(is_managed = True) > > File > > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > > 459, in start > > XendTask.log_progress(31, 60, self._initDomain) > > File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line > > 209, in log_progress > > retval = func(*args, **kwds) > > File > > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > > 2594, in _initDomain > > self._createDevices() > > File > > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > > 2218, in _createDevices > > self.image.createDeviceModel() > > File "/usr/lib64/python2.6/site-packages/xen/xend/image.py", line > > 419, in createDeviceModel > > pid = os.fork() > > OSError: [Errno 12] Cannot allocate memory > > > > > > Uhm.. I wonder what memory it fails to allocate. > > Me too.... >Did you already open a case with Novell? You''re using an enterpise distribution with full support included so just open a bug case.. They might have more ideas about what''s happening.. I can''t see anything obviously wrong in the stuff you''ve provided. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-23 16:56 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> > > [2010-01-22 19:53:18 5610] ERROR (SrvBase:88) Request start failed. > Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/xen/web/SrvBase.py", line > 85, in perform > return op_method(op, req) > File > "/usr/lib64/python2.6/site-packages/xen/xend/server/SrvDomain.py", > line 77, in op_start > return self.xd.domain_start(self.dom.getName(), paused) > File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomain.py", > line 1040, in domain_start > dominfo.start(is_managed = True) > File > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > 459, in start > XendTask.log_progress(31, 60, self._initDomain) > File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line > 209, in log_progress > retval = func(*args, **kwds) > File > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > 2594, in _initDomain > self._createDevices() > File > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > 2218, in _createDevices > self.image.createDeviceModel() > File "/usr/lib64/python2.6/site-packages/xen/xend/image.py", line > 419, in createDeviceModel > pid = os.fork() > OSError: [Errno 12] Cannot allocate memory > > > Uhm.. I wonder what memory it fails to allocate.Me too.... Here''s dmesg: # dmesg Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.27.19-5-xen (geeko@buildhost) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) #1 SMP 2009-02-28 04:40:21 +0100 Command line: root=/dev/disk/by-id/scsi-200d0b29d2d004400-part2 resume=/dev/disk/by-id/scsi-200d0b29d2d004400-part1 splash=silent vga=0x317 KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls Xen-provided physical RAM map: Xen: 0000000000000000 - 0000000080800000 (usable) DMI 2.6 present. last_pfn = 0x80800 max_arch_pfn = 0x6ffffff init_memory_mapping 0000000000 - 0080800000 page 4k kernel direct mapping tables up to 80800000 @ 19eb000-1df3000 last_map_addr: 80800000 end: 80800000 RAMDISK: 0082e000 - 015d7400 ACPI: RSDP 000F4F00, 0024 (r2 HP ) ACPI: XSDT DF62FDC0, 00B4 (r1 HP ProLiant 2 � 162E) ACPI: FACP DF62FEC0, 00F4 (r3 HP ProLiant 2 � 162E) ACPI: DSDT DF62FFC0, 1F95 (r1 HP DSDT 1 INTL 20030228) ACPI: FACS DF62F100, 0040 ACPI: SPCR DF62F140, 0050 (r1 HP SPCRRBSU 1 � 162E) ACPI: MCFG DF62F1C0, 003C (r1 HP ProLiant 1 0) ACPI: HPET DF62F200, 0038 (r1 HP ProLiant 2 � 162E) ACPI: FFFF DF62F240, 0064 (r2 HP ProLiant 2 � 162E) ACPI: SPMI DF62F2C0, 0040 (r5 HP ProLiant 1 � 162E) ACPI: ERST DF62F300, 01D0 (r1 HP ProLiant 1 � 162E) ACPI: APIC DF62F500, 00DE (r1 HP ProLiant 2 0) ACPI: SRAT DF62F600, 0370 (r1 HP Proliant 1 � 162E) ACPI: FFFF DF62F980, 0176 (r1 HP ProLiant 1 � 162E) ACPI: BERT DF62FB00, 0030 (r1 HP ProLiant 1 � 162E) ACPI: HEST DF62FB40, 00BC (r1 HP ProLiant 1 � 162E) ACPI: DMAR DF62FC00, 012C (r1 HP ProLiant 1 � 162E) ACPI: SSDT DF631F80, 0125 (r3 HP CRSPCI0 2 HP 1) ACPI: SSDT DF6320C0, 01CF (r3 HP riser1a 2 INTL 20061109) ACPI: SSDT DF6322C0, 021B (r1 HP pcc 1 INTL 20090625) ACPI: SSDT DF632500, 0377 (r1 HP pmab 1 INTL 20090625) ACPI: SSDT DF632880, 3094 (r1 INTEL PPM RCM 1 INTL 20061109) (4 early reservations) ==> bootmem [0000000000 - 0080000000] #0 [0000200000 - 000082d0b8] TEXT DATA BSS ==> [0000200000 - 000082d0b8] #1 [000082e000 - 00019ea000] Xen provided ==> [000082e000 - 00019ea000] #2 [00019ea000 - 00019eb000] INITMAP ==> [00019ea000 - 00019eb000] #3 [00019eb000 - 0001df3000] PGTABLE ==> [00019eb000 - 0001df3000] found SMP MP-table at [ffffffffff5ecf80] 000f4f80 Zone PFN ranges: DMA 0x00000000 -> 0x00001000 DMA32 0x00001000 -> 0x00100000 Normal 0x00100000 -> 0x00100000 Movable zone start PFN for each node early_node_map[1] active PFN ranges 0: 0x00000000 -> 0x00080800 On node 0 totalpages: 526336 free_area_init_node: node 0, pgdat ffffffff80689100, node_mem_map ffff880001df3000 DMA zone: 4032 pages, LIFO batch:0 DMA32 zone: 514080 pages, LIFO batch:31 ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x14] enabled) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] enabled) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x16] enabled) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] enabled) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] enabled) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] enabled) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] enabled) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-23 ACPI: IOAPIC (id[0x00] address[0xfec80000] gsi_base[24]) IOAPIC[1]: apic_id 0, version 0, address 0xfec80000, GSI 24-47 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) ACPI: IRQ0 used by override. ACPI: IRQ2 used by override. ACPI: IRQ9 used by override. Using ACPI (MADT) for SMP configuration information Allocating PCI resources starting at e6000000 (gap: e4000000:1ac00000) PERCPU: Allocating 50848 bytes of per cpu data NR_CPUS: 32, nr_cpu_ids: 16, nr_node_ids 1 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 518112 Kernel command line: root=/dev/disk/by-id/scsi-200d0b29d2d004400-part2 resume=/dev/disk/by-id/scsi-200d0b29d2d004400-part1 splash=silent vga=0x317 bootsplash: silent mode. Initializing CPU#0 PID hash table entries: 4096 (order: 12, 32768 bytes) Extended CMOS year: 2000 Xen reported: 2533.432 MHz processor. Console: colour VGA+ 80x25 console [tty0] enabled console [xvc-1] enabled Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) Software IO TLB enabled: Aperture: 64 megabytes Kernel range: ffff880004227000 - ffff880008227000 Address size: 27 bits PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Memory: 1964796k/2105344k available (2483k kernel code, 131844k reserved, 2190k data, 284k init) CPA: page pool initialized 1 of 1 pages preallocated Calibrating delay using timer specific routine.. 5068.59 BogoMIPS (lpj=10137182) Security Framework initialized AppArmor: AppArmor initialized Mount-cache hash table entries: 256 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 0 SMP alternatives: switching to UP code ACPI: Core revision 20080609 Parsing all Control Methods: Table [DSDT](id 0001) - 300 Objects with 36 Devices 69 Methods 14 Regions Parsing all Control Methods: Table [SSDT](id 0002) - 2 Objects with 0 Devices 0 Methods 0 Regions Parsing all Control Methods: Table [SSDT](id 0003) - 48 Objects with 16 Devices 0 Methods 0 Regions Parsing all Control Methods: Table [SSDT](id 0004) - 17 Objects with 0 Devices 0 Methods 0 Regions Parsing all Control Methods: Table [SSDT](id 0005) - 15 Objects with 1 Devices 7 Methods 1 Regions Parsing all Control Methods: Table [SSDT](id 0006) - 144 Objects with 0 Devices 64 Methods 0 Regions tbxface-0596 [00] tb_load_namespace : ACPI Tables successfully acquired evxfevnt-0091 [00] enable : Transition to ACPI mode successful SMP alternatives: switching to SMP code Initializing CPU#1 Initializing CPU#2 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 2 Initializing CPU#3 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 2 Initializing CPU#4 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 1 Initializing CPU#5 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 1 Initializing CPU#6 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 3 Initializing CPU#7 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 3 Initializing CPU#8 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 0 Initializing CPU#9 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 0 Initializing CPU#10 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 2 Initializing CPU#11 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 2 Initializing CPU#12 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 1 Initializing CPU#13 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 1 Initializing CPU#14 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 3 Brought up 16 CPUs Initializing CPU#15 CPU0 attaching sched-domain: domain 0: span 0-15 level CPU groups: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 CPU1 attaching sched-domain: domain 0: span 0-15 level CPU groups: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 CPU2 attaching sched-domain: domain 0: span 0-15 level CPU groups: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 CPU3 attaching sched-domain: domain 0: span 0-15 level CPU groups: 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 CPU4 attaching sched-domain: domain 0: span 0-15 level CPU groups: 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 CPU: L1 I cache: 32K<7>CPU5 attaching sched-domain: domain 0: span 0-15 level CPU groups: 5 6 7 8, L1 D cache: 32K 9<6>CPU: L2 cache: 256K 10<6>CPU: L3 cache: 8192K 11 12 13 14 15 0 1 2 3 4 CPU6 attaching sched-domain: domain 0: span 0-15 level CPU groups:<6>CPU: Physical Processor ID: 1 CPU: Processor Core ID: 3 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 CPU7 attaching sched-domain: domain 0: span 0-15 level CPU groups: 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 CPU8 attaching sched-domain: domain 0: span 0-15 level CPU groups: 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 CPU9 attaching sched-domain: domain 0: span 0-15 level CPU groups: 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 CPU10 attaching sched-domain: domain 0: span 0-15 level CPU groups: 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 CPU11 attaching sched-domain: domain 0: span 0-15 level CPU groups: 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 CPU12 attaching sched-domain: domain 0: span 0-15 level CPU groups: 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 CPU13 attaching sched-domain: domain 0: span 0-15 level CPU groups: 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 CPU14 attaching sched-domain: domain 0: span 0-15 level CPU groups: 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 CPU15 attaching sched-domain: domain 0: span 0-15 level CPU groups: 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 8192K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 0 net_namespace: 1936 bytes NET: Registered protocol family 16 ACPI: bus type pci registered PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 63 PCI: MCFG area at e0000000 reserved in E820 PCI: Using MMCONFIG at e0000000 - e3ffffff PCI: Using configuration type 1 for base access PCI: HP ProLiant DL360 detected, enabling pci=bfsort. evgpeblk-0957 [00] ev_create_gpe_block : GPE 00 to 3F [_GPE] 8 regs on int 0x9 ACPI: EC: Look up EC in DSDT Completing Region/Field/Buffer/Package initialization:........................................................................................... Initialized 13/15 Regions 23/23 Fields 32/32 Buffers 23/23 Packages (535 nodes) Initializing Device/Processor/Thermal objects by executing _INI methods: Executed 0 _INI methods requiring 0 _STA executions (examined 72 objects) evgpeblk-1054 [00] ev_initialize_gpe_bloc: Found 0 Wake, Enabled 0 Runtime GPEs in this block ACPI: Interpreter enabled ACPI: (supports S0 S5) ACPI: Using IOAPIC for interrupt routing ACPI: PCI Root Bridge [PCI0] (0000:00) pci 0000:00:00.0: PME# supported from D0 D3hot D3cold pci 0000:00:00.0: PME# disabled pci 0000:00:01.0: PME# supported from D0 D3hot D3cold pci 0000:00:01.0: PME# disabled pci 0000:00:02.0: PME# supported from D0 D3hot D3cold pci 0000:00:02.0: PME# disabled pci 0000:00:03.0: PME# supported from D0 D3hot D3cold pci 0000:00:03.0: PME# disabled pci 0000:00:04.0: PME# supported from D0 D3hot D3cold pci 0000:00:04.0: PME# disabled pci 0000:00:05.0: PME# supported from D0 D3hot D3cold pci 0000:00:05.0: PME# disabled pci 0000:00:06.0: PME# supported from D0 D3hot D3cold pci 0000:00:06.0: PME# disabled pci 0000:00:07.0: PME# supported from D0 D3hot D3cold pci 0000:00:07.0: PME# disabled pci 0000:00:08.0: PME# supported from D0 D3hot D3cold pci 0000:00:08.0: PME# disabled pci 0000:00:09.0: PME# supported from D0 D3hot D3cold pci 0000:00:09.0: PME# disabled pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold pci 0000:00:0a.0: PME# disabled PCI: 0000:00:1d.0 reg 20 io port: [1000, 101f] PCI: 0000:00:1d.1 reg 20 io port: [1020, 103f] PCI: 0000:00:1d.2 reg 20 io port: [1040, 105f] PCI: 0000:00:1d.3 reg 20 io port: [1060, 107f] PCI: 0000:00:1f.2 reg 10 io port: [1080, 1087] PCI: 0000:00:1f.2 reg 14 io port: [1088, 108b] PCI: 0000:00:1f.2 reg 18 io port: [1090, 1097] PCI: 0000:00:1f.2 reg 1c io port: [1098, 109b] PCI: 0000:00:1f.2 reg 20 io port: [10a0, 10af] PCI: 0000:00:1f.2 reg 24 io port: [10b0, 10bf] PCI: 0000:03:00.0 reg 10 64bit mmio: [fb800000, fbbfffff] PCI: 0000:03:00.0 reg 18 64bit mmio: [fb7f0000, fb7f0fff] PCI: 0000:03:00.0 reg 20 io port: [4000, 40ff] PCI: 0000:03:00.0 reg 30 32bit mmio: [0, 7ffff] pci 0000:03:00.0: supports D1 pci 0000:03:00.0: PME# supported from D0 pci 0000:03:00.0: PME# disabled PCI: bridge 0000:00:01.0 io port: [4000, 4fff] PCI: bridge 0000:00:01.0 32bit mmio: [fb700000, fbbfffff] pci 0000:07:00.0: supports D1 pci 0000:07:00.0: PME# supported from D0 D1 D3hot D3cold pci 0000:07:00.0: PME# disabled PCI: 0000:07:00.1 reg 10 64bit mmio: [fbef0000, fbef007f] pci 0000:07:00.1: supports D1 pci 0000:07:00.1: PME# supported from D0 D1 D3hot D3cold pci 0000:07:00.1: PME# disabled Pre-1.1 PCIe device detected, disable ASPM for 0000:00:03.0. It can be enabled forcedly with ''pcie_aspm=force'' PCI: bridge 0000:00:03.0 io port: [5000, 5fff] PCI: bridge 0000:00:03.0 32bit mmio: [fbe00000, fbffffff] PCI: 0000:08:01.0 reg 10 io port: [5000, 50ff] PCI: 0000:08:01.0 reg 14 64bit mmio: [fbff0000, fbff0fff] PCI: 0000:08:01.0 reg 30 32bit mmio: [0, 1ffff] PCI: bridge 0000:07:00.0 io port: [5000, 5fff] PCI: bridge 0000:07:00.0 32bit mmio: [fbf00000, fbffffff] PCI: 0000:02:00.0 reg 10 64bit mmio: [f8000000, f9ffffff] PCI: 0000:02:00.0 reg 30 32bit mmio: [0, ffff] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold pci 0000:02:00.0: PME# disabled PCI: 0000:02:00.1 reg 10 64bit mmio: [f6000000, f7ffffff] PCI: 0000:02:00.1 reg 30 32bit mmio: [0, ffff] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold pci 0000:02:00.1: PME# disabled PCI: bridge 0000:00:08.0 32bit mmio: [f6000000, f9ffffff] PCI: 0000:01:03.0 reg 10 32bit mmio: [e8000000, efffffff] PCI: 0000:01:03.0 reg 14 io port: [3000, 30ff] PCI: 0000:01:03.0 reg 18 32bit mmio: [f5ff0000, f5ffffff] PCI: 0000:01:03.0 reg 30 32bit mmio: [0, 1ffff] pci 0000:01:03.0: supports D1 pci 0000:01:03.0: supports D2 PCI: 0000:01:04.0 reg 10 io port: [2800, 28ff] PCI: 0000:01:04.0 reg 14 32bit mmio: [f5fe0000, f5fe01ff] pci 0000:01:04.0: PME# supported from D0 D3hot D3cold pci 0000:01:04.0: PME# disabled PCI: 0000:01:04.2 reg 10 io port: [3400, 34ff] PCI: 0000:01:04.2 reg 14 32bit mmio: [f5fd0000, f5fd07ff] PCI: 0000:01:04.2 reg 18 32bit mmio: [f5fc0000, f5fc3fff] PCI: 0000:01:04.2 reg 1c 32bit mmio: [f5f00000, f5f7ffff] PCI: 0000:01:04.2 reg 30 32bit mmio: [0, ffff] pci 0000:01:04.2: PME# supported from D0 D3hot D3cold pci 0000:01:04.2: PME# disabled PCI: 0000:01:04.4 reg 20 io port: [3800, 381f] pci 0000:01:04.4: PME# supported from D0 D3hot D3cold pci 0000:01:04.4: PME# disabled PCI: 0000:01:04.6 reg 10 32bit mmio: [f5ef0000, f5ef00ff] pci 0000:01:04.6: PME# supported from D0 D3hot D3cold pci 0000:01:04.6: PME# disabled pci 0000:00:1e.0: transparent bridge PCI: bridge 0000:00:1e.0 io port: [2000, 3fff] PCI: bridge 0000:00:1e.0 32bit mmio: [f5e00000, f5ffffff] PCI: bridge 0000:00:1e.0 64bit mmio pref: [e8000000, efffffff] bus 00 -> node 0 ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.IP2P._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT01._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT02._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT03._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT04._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT05._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT06._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT07._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT08._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT09._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT0A._PRT] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 7 10 11) ACPI: PCI Interrupt Link [LNKB] (IRQs 5 7 *10 11) ACPI: PCI Interrupt Link [LNKC] (IRQs 5 7 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKD] (IRQs 5 7 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKE] (IRQs *5 7 10 11) ACPI: PCI Interrupt Link [LNKF] (IRQs 5 7 10 *11) ACPI: PCI Interrupt Link [LNKG] (IRQs 5 *7 10 11) ACPI: PCI Interrupt Link [LNKH] (IRQs 5 *7 10 11) Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI init ACPI: bus type pnp registered pnp: PnP ACPI: found 11 devices ACPI: ACPI bus type pnp unregistered xen_mem: Initialising balloon driver. PCI: Using ACPI for IRQ routing AppArmor: AppArmor Filesystem Enabled ACPI: RTC can wake from S4 system 00:01: ioport range 0x408-0x40f has been reserved system 00:01: ioport range 0x4d0-0x4d1 has been reserved system 00:01: ioport range 0x700-0x71f has been reserved system 00:01: ioport range 0x880-0x8ff has been reserved system 00:01: ioport range 0x900-0x97f has been reserved system 00:01: ioport range 0xc80-0xc83 has been reserved system 00:01: ioport range 0xcd4-0xcd7 has been reserved system 00:01: ioport range 0xf50-0xf58 has been reserved system 00:01: ioport range 0xca0-0xca1 has been reserved system 00:01: ioport range 0xca4-0xca5 has been reserved system 00:01: ioport range 0x2f8-0x2ff has been reserved system 00:01: iomem range 0xe0000000-0xe3ffffff could not be reserved system 00:01: iomem range 0xfe000000-0xfebfffff has been reserved system 00:01: iomem range 0xe7ffe000-0xe7ffffff has been reserved pci 0000:00:01.0: PCI bridge, secondary bus 0000:03 pci 0000:00:01.0: IO window: 0x4000-0x4fff pci 0000:00:01.0: MEM window: 0xfb700000-0xfbbfffff pci 0000:00:01.0: PREFETCH window: 0x000000e6000000-0x000000e60fffff pci 0000:00:02.0: PCI bridge, secondary bus 0000:0a pci 0000:00:02.0: IO window: disabled pci 0000:00:02.0: MEM window: disabled pci 0000:00:02.0: PREFETCH window: disabled pci 0000:07:00.0: PCI bridge, secondary bus 0000:08 pci 0000:07:00.0: IO window: 0x5000-0x5fff pci 0000:07:00.0: MEM window: 0xfbf00000-0xfbffffff pci 0000:07:00.0: PREFETCH window: 0x000000e6100000-0x000000e61fffff pci 0000:07:00.1: PCI bridge, secondary bus 0000:09 pci 0000:07:00.1: IO window: disabled pci 0000:07:00.1: MEM window: disabled pci 0000:07:00.1: PREFETCH window: disabled pci 0000:00:03.0: PCI bridge, secondary bus 0000:07 pci 0000:00:03.0: IO window: 0x5000-0x5fff pci 0000:00:03.0: MEM window: 0xfbe00000-0xfbffffff pci 0000:00:03.0: PREFETCH window: 0x000000e6100000-0x000000e61fffff pci 0000:00:04.0: PCI bridge, secondary bus 0000:0b pci 0000:00:04.0: IO window: disabled pci 0000:00:04.0: MEM window: disabled pci 0000:00:04.0: PREFETCH window: disabled pci 0000:00:05.0: PCI bridge, secondary bus 0000:0c pci 0000:00:05.0: IO window: disabled pci 0000:00:05.0: MEM window: disabled pci 0000:00:05.0: PREFETCH window: disabled pci 0000:00:06.0: PCI bridge, secondary bus 0000:0d pci 0000:00:06.0: IO window: disabled pci 0000:00:06.0: MEM window: disabled pci 0000:00:06.0: PREFETCH window: disabled pci 0000:00:07.0: PCI bridge, secondary bus 0000:0e pci 0000:00:07.0: IO window: disabled pci 0000:00:07.0: MEM window: disabled pci 0000:00:07.0: PREFETCH window: disabled pci 0000:00:08.0: PCI bridge, secondary bus 0000:02 pci 0000:00:08.0: IO window: disabled pci 0000:00:08.0: MEM window: 0xf6000000-0xf9ffffff pci 0000:00:08.0: PREFETCH window: 0x000000e6200000-0x000000e62fffff pci 0000:00:09.0: PCI bridge, secondary bus 0000:04 pci 0000:00:09.0: IO window: disabled pci 0000:00:09.0: MEM window: disabled pci 0000:00:09.0: PREFETCH window: disabled pci 0000:00:0a.0: PCI bridge, secondary bus 0000:0f pci 0000:00:0a.0: IO window: disabled pci 0000:00:0a.0: MEM window: disabled pci 0000:00:0a.0: PREFETCH window: disabled pci 0000:00:1e.0: PCI bridge, secondary bus 0000:01 pci 0000:00:1e.0: IO window: 0x2000-0x3fff pci 0000:00:1e.0: MEM window: 0xf5e00000-0xf5ffffff pci 0000:00:1e.0: PREFETCH window: 0x000000e8000000-0x000000efffffff pci 0000:00:01.0: setting latency timer to 64 pci 0000:00:02.0: setting latency timer to 64 pci 0000:00:03.0: setting latency timer to 64 pci 0000:07:00.0: setting latency timer to 64 vendor=8086 device=340a pci 0000:07:00.1: PCI INT A -> GSI 24 (level, low) -> IRQ 24 pci 0000:07:00.1: setting latency timer to 64 pci 0000:00:04.0: setting latency timer to 64 pci 0000:00:05.0: setting latency timer to 64 pci 0000:00:06.0: setting latency timer to 64 pci 0000:00:07.0: setting latency timer to 64 pci 0000:00:08.0: setting latency timer to 64 pci 0000:00:09.0: setting latency timer to 64 pci 0000:00:0a.0: setting latency timer to 64 pci 0000:00:1e.0: setting latency timer to 64 bus: 00 index 0 io port: [0, ffff] bus: 00 index 1 mmio: [0, ffffffffffffffff] bus: 03 index 0 io port: [4000, 4fff] bus: 03 index 1 mmio: [fb700000, fbbfffff] bus: 03 index 2 mmio: [e6000000, e60fffff] bus: 03 index 3 mmio: [0, 0] bus: 0a index 0 mmio: [0, 0] bus: 0a index 1 mmio: [0, 0] bus: 0a index 2 mmio: [0, 0] bus: 0a index 3 mmio: [0, 0] bus: 07 index 0 io port: [5000, 5fff] bus: 07 index 1 mmio: [fbe00000, fbffffff] bus: 07 index 2 mmio: [e6100000, e61fffff] bus: 07 index 3 mmio: [0, 0] bus: 08 index 0 io port: [5000, 5fff] bus: 08 index 1 mmio: [fbf00000, fbffffff] bus: 08 index 2 mmio: [e6100000, e61fffff] bus: 08 index 3 mmio: [0, 0] bus: 09 index 0 mmio: [0, 0] bus: 09 index 1 mmio: [0, 0] bus: 09 index 2 mmio: [0, 0] bus: 09 index 3 mmio: [0, 0] bus: 0b index 0 mmio: [0, 0] bus: 0b index 1 mmio: [0, 0] bus: 0b index 2 mmio: [0, 0] bus: 0b index 3 mmio: [0, 0] bus: 0c index 0 mmio: [0, 0] bus: 0c index 1 mmio: [0, 0] bus: 0c index 2 mmio: [0, 0] bus: 0c index 3 mmio: [0, 0] bus: 0d index 0 mmio: [0, 0] bus: 0d index 1 mmio: [0, 0] bus: 0d index 2 mmio: [0, 0] bus: 0d index 3 mmio: [0, 0] bus: 0e index 0 mmio: [0, 0] bus: 0e index 1 mmio: [0, 0] bus: 0e index 2 mmio: [0, 0] bus: 0e index 3 mmio: [0, 0] bus: 02 index 0 mmio: [0, 0] bus: 02 index 1 mmio: [f6000000, f9ffffff] bus: 02 index 2 mmio: [e6200000, e62fffff] bus: 02 index 3 mmio: [0, 0] bus: 04 index 0 mmio: [0, 0] bus: 04 index 1 mmio: [0, 0] bus: 04 index 2 mmio: [0, 0] bus: 04 index 3 mmio: [0, 0] bus: 0f index 0 mmio: [0, 0] bus: 0f index 1 mmio: [0, 0] bus: 0f index 2 mmio: [0, 0] bus: 0f index 3 mmio: [0, 0] bus: 01 index 0 io port: [2000, 3fff] bus: 01 index 1 mmio: [f5e00000, f5ffffff] bus: 01 index 2 mmio: [e8000000, efffffff] bus: 01 index 3 io port: [0, ffff] bus: 01 index 4 mmio: [0, ffffffffffffffff] NET: Registered protocol family 2 IP route cache hash table entries: 65536 (order: 7, 524288 bytes) TCP established hash table entries: 262144 (order: 10, 4194304 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) TCP: Hash tables configured (established 262144 bind 65536) TCP reno registered NET: Registered protocol family 1 Unpacking initramfs... done Freeing initrd memory: 13989k freed audit: initializing netlink socket (disabled) type=2000 audit(1260809941.848:1): initialized VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) msgmni has been set to 1030 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci 0000:01:03.0: Boot video device pcieport-driver 0000:00:01.0: setting latency timer to 64 pcieport-driver 0000:00:01.0: found MSI capability pci_express 0000:00:01.0:pcie00: allocate port service pci_express 0000:00:01.0:pcie01: allocate port service pcieport-driver 0000:00:02.0: setting latency timer to 64 pcieport-driver 0000:00:02.0: found MSI capability pci_express 0000:00:02.0:pcie00: allocate port service pci_express 0000:00:02.0:pcie01: allocate port service pcieport-driver 0000:00:03.0: setting latency timer to 64 pcieport-driver 0000:00:03.0: found MSI capability pci_express 0000:00:03.0:pcie00: allocate port service pci_express 0000:00:03.0:pcie01: allocate port service pcieport-driver 0000:00:04.0: setting latency timer to 64 pcieport-driver 0000:00:04.0: found MSI capability pci_express 0000:00:04.0:pcie00: allocate port service pci_express 0000:00:04.0:pcie01: allocate port service pcieport-driver 0000:00:05.0: setting latency timer to 64 pcieport-driver 0000:00:05.0: found MSI capability pci_express 0000:00:05.0:pcie00: allocate port service pci_express 0000:00:05.0:pcie01: allocate port service pcieport-driver 0000:00:06.0: setting latency timer to 64 pcieport-driver 0000:00:06.0: found MSI capability pci_express 0000:00:06.0:pcie00: allocate port service pci_express 0000:00:06.0:pcie01: allocate port service pcieport-driver 0000:00:07.0: setting latency timer to 64 pcieport-driver 0000:00:07.0: found MSI capability pci_express 0000:00:07.0:pcie00: allocate port service pci_express 0000:00:07.0:pcie01: allocate port service pcieport-driver 0000:00:08.0: setting latency timer to 64 pcieport-driver 0000:00:08.0: found MSI capability pci_express 0000:00:08.0:pcie00: allocate port service pci_express 0000:00:08.0:pcie01: allocate port service pcieport-driver 0000:00:09.0: setting latency timer to 64 pcieport-driver 0000:00:09.0: found MSI capability pci_express 0000:00:09.0:pcie00: allocate port service pci_express 0000:00:09.0:pcie01: allocate port service pcieport-driver 0000:00:0a.0: setting latency timer to 64 pcieport-driver 0000:00:0a.0: found MSI capability pci_express 0000:00:0a.0:pcie00: allocate port service pci_express 0000:00:0a.0:pcie01: allocate port service aer 0000:00:01.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:02.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:03.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:04.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:05.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:06.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:07.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:08.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:09.0:pcie01: AER service couldn''t init device: no _OSC support aer 0000:00:0a.0:pcie01: AER service couldn''t init device: no _OSC support Non-volatile memory driver v1.2 Xen virtual console successfully installed as xvc0 Event-channel device installed. PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12 serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mice: PS/2 mouse device common for all mice input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 TCP cubic registered registered taskstats version 1 Freeing unused kernel memory: 284k freed Write protecting the kernel read-only data: 4420k SCSI subsystem initialized HP CISS Driver (v 3.6.20) vendor=8086 device=3408 cciss 0000:03:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28 cciss0: <0x323a> at PCI 0000:03:00.0 IRQ 1267 using DAC ACPI: CPU0 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:00: registered as cooling_device0 ACPI: CPU1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:01: registered as cooling_device1 ACPI: CPU2 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:02: registered as cooling_device2 ACPI: CPU3 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:03: registered as cooling_device3 ACPI: CPU4 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:04: registered as cooling_device4 ACPI: CPU5 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:05: registered as cooling_device5 ACPI: CPU6 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:06: registered as cooling_device6 ACPI: CPU7 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:07: registered as cooling_device7 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:08: registered as cooling_device8 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:09: registered as cooling_device9 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:0a: registered as cooling_device10 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:0b: registered as cooling_device11 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:0c: registered as cooling_device12 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:0d: registered as cooling_device13 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:0e: registered as cooling_device14 ACPI: CPU-1 (power states: C1[C1] C2[C3] C3[C3]) processor ACPI_CPU:0f: registered as cooling_device15 thermal LNXTHERM:01: registered as thermal_zone0 ACPI: Thermal Zone [THM0] (8 C) ACPI: No dock devices found. libata version 3.00 loaded. ata_piix 0000:00:1f.2: version 2.12 ata_piix 0000:00:1f.2: PCI INT B -> GSI 17 (level, low) -> IRQ 17 ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ] ata_piix 0000:00:1f.2: setting latency timer to 64 scsi0 : ata_piix scsi1 : ata_piix ata1: SATA max UDMA/133 cmd 0x1080 ctl 0x1088 bmdma 0x10a0 irq 17 ata2: SATA max UDMA/133 cmd 0x1090 ctl 0x1098 bmdma 0x10a8 irq 17 input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input1 ata1.00: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata1.01: SATA link down (SStatus 4 SControl 300) ata1.00: ATAPI: HL-DT-STDVD-ROM GDR-D20N, 1.05, max UDMA/100 ata1.00: configured for UDMA/100 ata2.00: SATA link down (SStatus 4 SControl 300) ata2.01: SATA link down (SStatus 4 SControl 300) scsi 0:0:0:0: CD-ROM HL-DT-ST DVD-ROM GDR-D20N 1.05 PQ: 0 ANSI: 5 Uniform Multi-Platform E-IDE driver BIOS EDD facility v0.16 2004-Jun-25, 1 devices found netfront: Initialising virtual ethernet driver. udevd version 128 started QLogic Fibre Channel HBA Driver: 8.02.01.03.11.0-k9 vendor=8086 device=340a qla2xxx 0000:08:01.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34 qla2xxx 0000:08:01.0: Found an ISP2312, irq 34, iobase 0xffffc20000064000 qla2xxx 0000:08:01.0: Configuring PCI space... qla2xxx 0000:08:01.0: Configure NVRAM parameters... qla2xxx 0000:08:01.0: Verifying loaded RISC code... usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb firmware: requesting ql2300_fw.bin qla2xxx 0000:08:01.0: Allocated (412 KB) for firmware dump... scsi2 : qla2xxx qla2xxx 0000:08:01.0: LOOP UP detected (2 Gbps). scsi 2:0:0:0: Direct-Access XIOtech Magnitude 3D 3.00 PQ: 0 ANSI: 4 scsi 2:0:0:2: Direct-Access XIOtech Magnitude 3D 3.00 PQ: 0 ANSI: 4 scsi 2:0:0:3: Direct-Access XIOtech Magnitude 3D 3.00 PQ: 0 ANSI: 4 qla2xxx 0000:08:01.0: QLogic Fibre Channel HBA Driver: 8.02.01.03.11.0-k9 QLogic QLA2340 - 133MHz PCI-X to 2Gb FC, Single Channel ISP2312: PCI-X (133 MHz) @ 0000:08:01.0 hdma-, host#=2, fw=3.03.27 IPX USB Universal Host Controller Interface driver v3.0 uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20 uhci_hcd 0000:00:1d.0: setting latency timer to 64 uhci_hcd 0000:00:1d.0: UHCI Host Controller uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1 uhci_hcd 0000:00:1d.0: irq 20, io base 0x00001000 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected sd 2:0:0:0: [sda] 104882176 512-byte hardware sectors: (53.6GB/50.0GiB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 97 00 00 08 sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn''t support DPO or FUA sd 2:0:0:0: [sda] 104882176 512-byte hardware sectors: (53.6GB/50.0GiB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 97 00 00 08 sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn''t support DPO or FUA sda: sda1 sda2 sd 2:0:0:0: [sda] Attached SCSI disk sd 2:0:0:2: [sdb] 629293056 512-byte hardware sectors: (322GB/300GiB) sd 2:0:0:2: [sdb] Write Protect is off sd 2:0:0:2: [sdb] Mode Sense: 97 00 00 08 sd 2:0:0:2: [sdb] Write cache: disabled, read cache: enabled, doesn''t support DPO or FUA sd 2:0:0:2: [sdb] 629293056 512-byte hardware sectors: (322GB/300GiB) sd 2:0:0:2: [sdb] Write Protect is off sd 2:0:0:2: [sdb] Mode Sense: 97 00 00 08 sd 2:0:0:2: [sdb] Write cache: disabled, read cache: enabled, doesn''t support DPO or FUA sdb: unknown partition table sd 2:0:0:2: [sdb] Attached SCSI disk sd 2:0:0:3: [sdc] 629166080 512-byte hardware sectors: (322GB/300GiB) sd 2:0:0:3: [sdc] Write Protect is off sd 2:0:0:3: [sdc] Mode Sense: 97 00 00 08 sd 2:0:0:3: [sdc] Write cache: disabled, read cache: enabled, doesn''t support DPO or FUA sd 2:0:0:3: [sdc] 629166080 512-byte hardware sectors: (322GB/300GiB) sd 2:0:0:3: [sdc] Write Protect is off sd 2:0:0:3: [sdc] Mode Sense: 97 00 00 08 sd 2:0:0:3: [sdc] Write cache: disabled, read cache: enabled, doesn''t support DPO or FUA sdc: unknown partition table sd 2:0:0:3: [sdc] Attached SCSI disk usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: UHCI Host Controller usb usb1: Manufacturer: Linux 2.6.27.19-5-xen uhci_hcd usb usb1: SerialNumber: 0000:00:1d.0 uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 23 (level, low) -> IRQ 23 uhci_hcd 0000:00:1d.1: setting latency timer to 64 uhci_hcd 0000:00:1d.1: UHCI Host Controller uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2 uhci_hcd 0000:00:1d.1: irq 23, io base 0x00001020 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected usb usb2: New USB device found, idVendor=1d6b, idProduct=0001 usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb2: Product: UHCI Host Controller usb usb2: Manufacturer: Linux 2.6.27.19-5-xen uhci_hcd usb usb2: SerialNumber: 0000:00:1d.1 uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22 uhci_hcd 0000:00:1d.2: setting latency timer to 64 uhci_hcd 0000:00:1d.2: UHCI Host Controller uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3 uhci_hcd 0000:00:1d.2: irq 22, io base 0x00001040 usb usb3: configuration #1 chosen from 1 choice hub 3-0:1.0: USB hub found hub 3-0:1.0: 2 ports detected usb usb3: New USB device found, idVendor=1d6b, idProduct=0001 usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb3: Product: UHCI Host Controller usb usb3: Manufacturer: Linux 2.6.27.19-5-xen uhci_hcd usb usb3: SerialNumber: 0000:00:1d.2 uhci_hcd 0000:00:1d.3: PCI INT D -> GSI 23 (level, low) -> IRQ 23 uhci_hcd 0000:00:1d.3: setting latency timer to 64 uhci_hcd 0000:00:1d.3: UHCI Host Controller uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 4 uhci_hcd 0000:00:1d.3: irq 23, io base 0x00001060 usb usb4: configuration #1 chosen from 1 choice hub 4-0:1.0: USB hub found hub 4-0:1.0: 2 ports detected usb usb4: New USB device found, idVendor=1d6b, idProduct=0001 usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb4: Product: UHCI Host Controller usb usb4: Manufacturer: Linux 2.6.27.19-5-xen uhci_hcd usb usb4: SerialNumber: 0000:00:1d.3 vendor=8086 device=244e uhci_hcd 0000:01:04.4: PCI INT B -> GSI 22 (level, low) -> IRQ 22 uhci_hcd 0000:01:04.4: UHCI Host Controller uhci_hcd 0000:01:04.4: new USB bus registered, assigned bus number 5 uhci_hcd 0000:01:04.4: port count misdetected? forcing to 2 ports uhci_hcd 0000:01:04.4: irq 22, io base 0x00003800 usb usb5: configuration #1 chosen from 1 choice hub 5-0:1.0: USB hub found hub 5-0:1.0: 2 ports detected usb usb5: New USB device found, idVendor=1d6b, idProduct=0001 usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb5: Product: UHCI Host Controller usb usb5: Manufacturer: Linux 2.6.27.19-5-xen uhci_hcd usb usb5: SerialNumber: 0000:01:04.4 usb 5-1: new full speed USB device using uhci_hcd and address 2 usb 5-1: configuration #1 chosen from 1 choice usb 5-1: New USB device found, idVendor=03f0, idProduct=1027 usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 usb 5-1: Product: Virtual Keyboard usb 5-1: Manufacturer: HP usb 5-2: new full speed USB device using uhci_hcd and address 3 usb 5-2: configuration #1 chosen from 1 choice hub 5-2:1.0: USB hub found hub 5-2:1.0: 7 ports detected usb 5-2: New USB device found, idVendor=03f0, idProduct=1327 usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 usb 5-2: Product: Virtual Hub usb 5-2: Manufacturer: HP usbcore: registered new interface driver hiddev input: HP Virtual Keyboard as /devices/pci0000:00/0000:00:1e.0/0000:01:04.4/usb5/5-1/5-1:1.0/input/input2 input,hidraw0: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:01:04.4-1 input: HP Virtual Keyboard as /devices/pci0000:00/0000:00:1e.0/0000:01:04.4/usb5/5-1/5-1:1.1/input/input3 input,hidraw1: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:01:04.4-1 usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver kjournald starting. Commit interval 5 seconds EXT3 FS on sda2, internal journal EXT3-fs: mounted filesystem with ordered data mode. udevd version 128 started input: PC Speaker as /devices/platform/pcspkr/input/input4 pci_hotplug: PCI Hot Plug PCI Core version: 0.5 shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 vendor=8086 device=244e hpwdt 0000:01:04.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 hpwdt: New timer passed in is 30 seconds. hp Watchdog Timer Driver: 1.00, timer margin: 30 seconds (nowayout=0), allow kernel dump: OFF (default = 0/OFF). Serial: 8250/16550 driver8 ports, IRQ sharing disabled serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input5 sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray Uniform CD-ROM driver Revision: 3.20 sr 0:0:0:0: Attached scsi CD-ROM sr0 vendor=8086 device=244e hpilo 0000:01:04.2: PCI INT B -> GSI 22 (level, low) -> IRQ 22 Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.8.0 (Aug 14, 2008) vendor=8086 device=340f bnx2 0000:02:00.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31 bnx2 0000:02:00.0: setting latency timer to 64 eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 31, node addr 00:26:55:82:7d:d4 vendor=8086 device=340f bnx2 0000:02:00.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39 bnx2 0000:02:00.1: setting latency timer to 64 eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 39, node addr 00:26:55:82:7d:d6 ipmi message handler version 39.2 sr 0:0:0:0: Attached scsi generic sg0 type 5 sd 2:0:0:0: Attached scsi generic sg1 type 0 sd 2:0:0:2: Attached scsi generic sg2 type 0 sd 2:0:0:3: Attached scsi generic sg3 type 0 ACPI: Power Button (FF) [PWRF] IPMI System Interface driver. ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A rtc_cmos 00:0a: rtc core: registered rtc_cmos as rtc0 rtc0: alarms up to one year, y3k ipmi: Found new BMC (man_id: 0x00000b, prod_id: 0x0000, dev_id: 0x11) IPMI kcs interface initialized ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 ipmi_si: duplicate interface vendor=8086 device=244e ipmi_si 0000:01:04.6: PCI INT A -> GSI 21 (level, low) -> IRQ 21 ipmi_si: Trying PCI-specified kcs state machine at mem address 0xf5ef0000, slave address 0x0, irq 21 Using irq 21 ipmi: interfacing existing BMC (man_id: 0x00000b, prod_id: 0x0000, dev_id: 0x11) IPMI kcs interface initialized Adding 2104472k swap on /dev/sda1. Priority:-1 extents:1 across:2104472k device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com loop: module loaded fuse init (API version 7.9) type=1505 audit(1260809966.977:2): operation="profile_load" name="/bin/ping" name2="default" pid=3395 type=1505 audit(1260809967.017:3): operation="profile_load" name="/sbin/klogd" name2="default" pid=3396 type=1505 audit(1260809967.073:4): operation="profile_load" name="/sbin/syslog-ng" name2="default" pid=3397 type=1505 audit(1260809967.137:5): operation="profile_load" name="/sbin/syslogd" name2="default" pid=3398 type=1505 audit(1260809967.212:6): operation="profile_load" name="/usr/sbin/avahi-daemon" name2="default" pid=3407 type=1505 audit(1260809967.273:7): operation="profile_load" name="/usr/sbin/identd" name2="default" pid=3427 type=1505 audit(1260809967.341:8): operation="profile_load" name="/usr/sbin/mdnsd" name2="default" pid=3436 type=1505 audit(1260809967.424:9): operation="profile_load" name="/usr/sbin/nscd" name2="default" pid=3437 type=1505 audit(1260809967.553:10): operation="profile_load" name="/usr/sbin/ntpd" name2="default" pid=3460 type=1505 audit(1260809967.617:11): operation="profile_load" name="/usr/sbin/traceroute" name2="default" pid=3475 IA-32 Microcode Update Driver: v1.14a-xen <tigran@aivazian.fsnet.co.uk> firmware: requesting intel-ucode/06-1a-05 Bridge firewalling registered bnx2: eth0: using MSIX device eth0 entered promiscuous mode bnx2: eth1: using MSIX device eth1 entered promiscuous mode OCFS2 Node Manager 1.5.0 OCFS2 DLM 1.5.0 ocfs2: Registered cluster interface o2cb OCFS2 DLMFS 1.5.0 OCFS2 User DLM kernel interface loaded bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON br0: topology change detected, propagating br0: port 1(eth0) entering forwarding state suspend: event channel 112 o2net: connected to node serverxen04 (num 0) at 172.10.4.154:7777 o2net: accepted connection from node serverxen06 (num 3) at 172.10.4.156:7777 o2net: accepted connection from node serverxen02 (num 2) at 172.10.4.152:7777 OCFS2 1.5.0 ocfs2_dlm: Nodes in domain ("D255FD5483184C6289B283294A0C1D14"): 0 1 2 3 kjournald2 starting. Commit interval 5 seconds ocfs2: Mounting device (8,16) on (node 1, slot 2) with ordered data mode. ocfs2_dlm: Nodes in domain ("A773AA5249074DB59A644756B3CD41B4"): 1 2 3 kjournald2 starting. Commit interval 5 seconds ocfs2: Mounting device (8,32) on (node 1, slot 2) with ordered data mode. ISO 9660 Extensions: Microsoft Joliet Level 3 ISO 9660 Extensions: RRIP_1991A EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: dm-1: orphan cleanup on readonly fs ext3_orphan_cleanup: deleting unreferenced inode 34113 EXT3-fs: dm-1: 1 orphan inode deleted EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. device vif2.0 entered promiscuous mode br0: topology change detected, propagating br0: port 3(vif2.0) entering forwarding state ip_tables: (C) 2000-2006 Netfilter Core Team nf_conntrack version 0.5.0 (16384 buckets, 65536 max) CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Plase use nf_conntrack.acct=1 kernel paramater, acct=1 nf_conntrack module option or sysctl net.netfilter.nf_conntrack_acct=1 to enable it. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/2/51712 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. blkback: ring-ref 8, event-channel 17, protocol 1 (x86_64-abi) EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: dm-1: orphan cleanup on readonly fs ext3_orphan_cleanup: deleting unreferenced inode 786657 ext3_orphan_cleanup: deleting unreferenced inode 786576 EXT3-fs: dm-1: 2 orphan inodes deleted EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. device vif3.0 entered promiscuous mode br0: topology change detected, propagating br0: port 4(vif3.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/3/51712 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 blkback: ring-ref 8, event-channel 17, protocol 1 (x86_64-abi) device vif4.0 entered promiscuous mode br0: topology change detected, propagating br0: port 5(vif4.0) entering forwarding state tun: Universal TUN/TAP device driver, 1.6 tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. device tap4.0 entered promiscuous mode br0: topology change detected, propagating br0: port 6(tap4.0) entering forwarding state (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/4/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/4/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. device vif5.0 entered promiscuous mode br0: topology change detected, propagating br0: port 7(vif5.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/5/51712 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 blkback: ring-ref 522, event-channel 18, protocol 2 (x86_32-abi) device tap6.0 entered promiscuous mode br0: topology change detected, propagating br0: port 8(tap6.0) entering forwarding state device vif6.0 entered promiscuous mode br0: topology change detected, propagating br0: port 9(vif6.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/6/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 device tap7.0 entered promiscuous mode br0: topology change detected, propagating br0: port 10(tap7.0) entering forwarding state device vif7.0 entered promiscuous mode br0: topology change detected, propagating br0: port 11(vif7.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/7/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/7/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 v5 domctl worked ok: dom7 is 64-bit blkback: ring-ref 16383, event-channel 5, protocol 1 (unspecified) v5 domctl worked ok: dom7 is 64-bit blkback: ring-ref 16238, event-channel 6, protocol 1 (unspecified) device tap8.0 entered promiscuous mode br0: topology change detected, propagating br0: port 12(tap8.0) entering forwarding state device vif8.0 entered promiscuous mode br0: topology change detected, propagating br0: port 13(vif8.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/8/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/8/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 device tap9.0 entered promiscuous mode br0: topology change detected, propagating br0: port 14(tap9.0) entering forwarding state device vif9.0 entered promiscuous mode br0: topology change detected, propagating br0: port 15(vif9.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/9/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/9/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 v5 domctl worked ok: dom8 is 64-bit blkback: ring-ref 16383, event-channel 5, protocol 1 (unspecified) v5 domctl worked ok: dom8 is 64-bit blkback: ring-ref 16238, event-channel 6, protocol 1 (unspecified) v5 domctl worked ok: dom9 is 64-bit blkback: ring-ref 16383, event-channel 5, protocol 1 (unspecified) v5 domctl worked ok: dom9 is 64-bit blkback: ring-ref 16238, event-channel 6, protocol 1 (unspecified) br0: port 4(vif3.0) entering disabled state br0: port 4(vif3.0) entering disabled state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. device vif10.0 entered promiscuous mode br0: topology change detected, propagating br0: port 4(vif10.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/10/51712 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 blkback: ring-ref 8, event-channel 17, protocol 1 (x86_64-abi) br0: port 10(tap7.0) entering disabled state device tap7.0 left promiscuous mode br0: port 10(tap7.0) entering disabled state br0: port 11(vif7.0) entering disabled state br0: port 11(vif7.0) entering disabled state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. device vif11.0 entered promiscuous mode br0: topology change detected, propagating br0: port 10(vif11.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. device tap11.0 entered promiscuous mode br0: topology change detected, propagating br0: port 11(tap11.0) entering forwarding state (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/11/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/11/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 v5 domctl worked ok: dom11 is 64-bit blkback: ring-ref 16383, event-channel 5, protocol 1 (unspecified) v5 domctl worked ok: dom11 is 64-bit blkback: ring-ref 16238, event-channel 6, protocol 1 (unspecified) ocfs2_dlm: Node 2 leaves domain A773AA5249074DB59A644756B3CD41B4 ocfs2_dlm: Nodes in domain ("A773AA5249074DB59A644756B3CD41B4"): 1 3 ocfs2_dlm: Node 2 leaves domain D255FD5483184C6289B283294A0C1D14 ocfs2_dlm: Nodes in domain ("D255FD5483184C6289B283294A0C1D14"): 0 1 3 o2net: no longer connected to node serverxen02 (num 2) at 172.10.4.152:7777 o2net: accepted connection from node serverxen02 (num 2) at 172.10.4.152:7777 ocfs2_dlm: Node 2 joins domain D255FD5483184C6289B283294A0C1D14 ocfs2_dlm: Nodes in domain ("D255FD5483184C6289B283294A0C1D14"): 0 1 2 3 ocfs2_dlm: Node 2 joins domain A773AA5249074DB59A644756B3CD41B4 ocfs2_dlm: Nodes in domain ("A773AA5249074DB59A644756B3CD41B4"): 1 2 3 o2net: no longer connected to node serverxen06 (num 3) at 172.10.4.156:7777 (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,3):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,2):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,2):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,2):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,2):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,2):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,2):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,2):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,4):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,1):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (4691,0):o2net_connect_expired:1659 ERROR: no connection established with node 3 after 30.0 seconds, giving up and returning errors. (5116,0):o2dlm_eviction_cb:258 o2dlm has evicted node 3 from group D255FD5483184C6289B283294A0C1D14 (5116,0):o2dlm_eviction_cb:258 o2dlm has evicted node 3 from group A773AA5249074DB59A644756B3CD41B4 (4691,0):o2dlm_eviction_cb:258 o2dlm has evicted node 3 from group A773AA5249074DB59A644756B3CD41B4 (5768,2):dlm_get_lock_resource:839 D255FD5483184C6289B283294A0C1D14:$RECOVERY: at least one node (3) to recover before lock mastery can begin (5768,2):dlm_get_lock_resource:873 D255FD5483184C6289B283294A0C1D14: recovery map is not empty, but must master $RECOVERY lock now (5768,2):dlm_do_recovery:524 (5768) Node 1 is the Recovery Master for the Dead Node 3 for Domain D255FD5483184C6289B283294A0C1D14 o2net: accepted connection from node serverxen06 (num 3) at 172.10.4.156:7777 ocfs2_dlm: Node 3 joins domain D255FD5483184C6289B283294A0C1D14 ocfs2_dlm: Nodes in domain ("D255FD5483184C6289B283294A0C1D14"): 0 1 2 3 ocfs2_dlm: Node 3 joins domain A773AA5249074DB59A644756B3CD41B4 ocfs2_dlm: Nodes in domain ("A773AA5249074DB59A644756B3CD41B4"): 1 2 3 ocfs2_dlm: Node 3 leaves domain A773AA5249074DB59A644756B3CD41B4 ocfs2_dlm: Nodes in domain ("A773AA5249074DB59A644756B3CD41B4"): 1 2 o2net: connection to node serverxen06 (num 3) at 172.10.4.156:7777 has been idle for 30.0 seconds, shutting it down. (0,1):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1262355887.355289 now 1262355917.353719 dr 1262355887.355268 adv 1262355887.355290:1262355887.355290 func (4320c113:513) 1262355879.362648:1262355879.362664) o2net: no longer connected to node serverxen06 (num 3) at 172.10.4.156:7777 (5116,1):o2dlm_eviction_cb:258 o2dlm has evicted node 3 from group D255FD5483184C6289B283294A0C1D14 (5768,2):dlm_get_lock_resource:839 D255FD5483184C6289B283294A0C1D14:$RECOVERY: at least one node (3) to recover before lock mastery can begin (5768,2):dlm_get_lock_resource:873 D255FD5483184C6289B283294A0C1D14: recovery map is not empty, but must master $RECOVERY lock now (5768,2):dlm_do_recovery:524 (5768) Node 1 is the Recovery Master for the Dead Node 3 for Domain D255FD5483184C6289B283294A0C1D14 o2net: accepted connection from node serverxen06 (num 3) at 172.10.4.156:7777 ocfs2_dlm: Node 3 joins domain D255FD5483184C6289B283294A0C1D14 ocfs2_dlm: Nodes in domain ("D255FD5483184C6289B283294A0C1D14"): 0 1 2 3 ocfs2_dlm: Node 3 joins domain A773AA5249074DB59A644756B3CD41B4 ocfs2_dlm: Nodes in domain ("A773AA5249074DB59A644756B3CD41B4"): 1 2 3 qla2xxx 0000:08:01.0: scsi(2:0:2): Abort command issued -- 1 216e09d 2002. qla2xxx 0000:08:01.0: scsi(2:0:2): Abort command issued -- 1 216e09e 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 216e09f 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 216e0a0 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 216e0a1 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 216e0a2 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 216e0a3 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 216e0a4 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 216e0b2 2002. br0: port 14(tap9.0) entering disabled state device tap9.0 left promiscuous mode br0: port 14(tap9.0) entering disabled state br0: port 15(vif9.0) entering disabled state br0: port 15(vif9.0) entering disabled state device vif12.0 entered promiscuous mode br0: topology change detected, propagating br0: port 14(vif12.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. device tap12.0 entered promiscuous mode br0: topology change detected, propagating br0: port 15(tap12.0) entering forwarding state (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/12/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/12/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 v5 domctl worked ok: dom12 is 64-bit blkback: ring-ref 16383, event-channel 5, protocol 1 (unspecified) v5 domctl worked ok: dom12 is 64-bit blkback: ring-ref 16238, event-channel 6, protocol 1 (unspecified) br0: port 12(tap8.0) entering disabled state device tap8.0 left promiscuous mode br0: port 12(tap8.0) entering disabled state br0: port 13(vif8.0) entering disabled state br0: port 13(vif8.0) entering disabled state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. device vif13.0 entered promiscuous mode br0: topology change detected, propagating br0: port 12(vif13.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. device tap13.0 entered promiscuous mode br0: topology change detected, propagating br0: port 13(tap13.0) entering forwarding state (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/13/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/13/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 v5 domctl worked ok: dom13 is 64-bit blkback: ring-ref 16383, event-channel 5, protocol 1 (unspecified) v5 domctl worked ok: dom13 is 64-bit blkback: ring-ref 16238, event-channel 6, protocol 1 (unspecified) br0: port 3(vif2.0) entering disabled state br0: port 3(vif2.0) entering disabled state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. device vif14.0 entered promiscuous mode br0: topology change detected, propagating br0: port 3(vif14.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/14/51712 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 blkback: ring-ref 8, event-channel 17, protocol 1 (x86_64-abi) br0: port 4(vif10.0) entering disabled state br0: port 4(vif10.0) entering disabled state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. device vif15.0 entered promiscuous mode br0: topology change detected, propagating br0: port 4(vif15.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/15/51712 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 blkback: ring-ref 8, event-channel 17, protocol 1 (x86_64-abi) br0: port 11(tap11.0) entering disabled state device tap11.0 left promiscuous mode br0: port 11(tap11.0) entering disabled state br0: port 10(vif11.0) entering disabled state br0: port 10(vif11.0) entering disabled state device vif16.0 entered promiscuous mode br0: topology change detected, propagating br0: port 10(vif16.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. device tap16.0 entered promiscuous mode br0: topology change detected, propagating br0: port 11(tap16.0) entering forwarding state (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/16/832 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 (cdrom_add_media_watch() file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/16/768 (cdrom_is_type() file=drivers/xen/blkback/cdrom.c, line=95) type:0 v5 domctl worked ok: dom16 is 64-bit blkback: ring-ref 16383, event-channel 5, protocol 1 (unspecified) v5 domctl worked ok: dom16 is 64-bit blkback: ring-ref 16238, event-channel 6, protocol 1 (unspecified) qla2xxx 0000:08:01.0: scsi(2:0:2): Abort command issued -- 1 36a2046 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 36a204b 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 36a204c 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 36a204d 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 36a204e 2002. qla2xxx 0000:08:01.0: scsi(2:0:0): Abort command issued -- 1 36a204f 2002. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. device vif19.0 entered promiscuous mode br0: topology change detected, propagating br0: port 16(vif19.0) entering forwarding state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. br0: port 16(vif19.0) entering disabled state br0: port 16(vif19.0) entering disabled state physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. ocfs2_dlm: Node 0 leaves domain D255FD5483184C6289B283294A0C1D14 ocfs2_dlm: Nodes in domain ("D255FD5483184C6289B283294A0C1D14"): 1 2 3 o2net: no longer connected to node serverxen04 (num 0) at 172.10.4.154:7777 o2net: connected to node serverxen04 (num 0) at 172.10.4.154:7777 ocfs2_dlm: Node 0 joins domain D255FD5483184C6289B283294A0C1D14 ocfs2_dlm: Nodes in domain ("D255FD5483184C6289B283294A0C1D14"): 0 1 2 3 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
jpp@jppozzi.dyndns.org
2010-Jan-23 22:02 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
Hello, What is the output of "xm top" and the memory size for the crashing domU ? Regards JP Pozzi Le vendredi 22 janvier 2010 à 22:06 -0500, James Pifer a écrit :> > Just for reference, back in the Xen 2.0 days I used to have 192 MB of > > RAM for dom0.. and that was enough. > > > > Nowadays I usually give dom0 512 MB of RAM, or in bigger machines 1024 MB. > > That is usually enough for a Xen-only server machine (without any local desktop/X use). > > > > So.. you need to make sure there''s nothing leaking memory in dom0. > > Xen management tools will obviously require some available memory in dom0 > > to work correctly. > > > > -- Pasi > > > > Here we go again.... I wanted to start another domU on one of me xen > servers tonight and I''m getting not able to allocate memory again. Check > out the following stats. > > If anyone sees anything I''d love to hear it... Please let me know if > there''s anything else I should post. > > Thanks, > James > > > # top > > top - 20:02:11 up 39 days, 9:03, 2 users, load average: 0.11, 0.85, 1.25 > Tasks: 256 total, 1 running, 255 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 98.5%id, 0.5%wa, 0.0%hi, 0.1%si, 0.8%st > Mem: 2095616k total, 1689848k used, 405768k free, 11624k buffers > Swap: 2104472k total, 1476856k used, 627616k free, 406620k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 8351 root 20 0 1098m 22m 928 S 1 1.1 459:39.03 qemu-dm > 5610 root 20 0 1859m 988m 1604 S 1 48.3 422:09.78 xend > 10448 root 20 0 1537m 29m 896 S 1 1.4 458:10.77 qemu-dm > 6 root 15 -5 0 0 0 S 0 0.0 78:19.80 ksoftirqd/1 > 12 root 15 -5 0 0 0 S 0 0.0 10:07.34 ksoftirqd/4 > 34 root 15 -5 0 0 0 S 0 0.0 10:15.54 ksoftirqd/15 > 5074 root 20 0 92884 980 808 S 0 0.0 42:25.98 qemu-dm > 11674 root 20 0 88600 768 656 S 0 0.0 27:43.75 qemu-dm > 13009 root 20 0 89140 972 804 S 0 0.0 66:57.23 qemu-dm > 1 root 20 0 1064 84 52 S 0 0.0 0:29.88 init > 2 root 15 -5 0 0 0 S 0 0.0 0:00.02 kthreadd > 3 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/0 > 4 root 15 -5 0 0 0 S 0 0.0 33:37.15 ksoftirqd/0 > 5 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/1 > 7 root RT -5 0 0 0 S 0 0.0 0:00.30 migration/2 > 8 root 15 -5 0 0 0 S 0 0.0 56:19.57 ksoftirqd/2 > 9 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/3 > 10 root 15 -5 0 0 0 S 0 0.0 31:04.94 ksoftirqd/3 > 11 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/4 > 13 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/5 > 14 root 15 -5 0 0 0 S 0 0.0 7:34.80 ksoftirqd/5 > 15 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/6 > 16 root 15 -5 0 0 0 S 0 0.0 5:30.04 ksoftirqd/6 > 17 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/7 > 18 root 15 -5 0 0 0 S 0 0.0 3:32.41 ksoftirqd/7 > 19 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/8 > 20 root 15 -5 0 0 0 S 0 0.0 0:02.44 ksoftirqd/8 > 21 root RT -5 0 0 0 S 0 0.0 0:00.20 migration/9 > 22 root 15 -5 0 0 0 S 0 0.0 0:03.64 ksoftirqd/9 > 23 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/10 > 24 root 15 -5 0 0 0 S 0 0.0 1:53.54 ksoftirqd/10 > 25 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/11 > 26 root 15 -5 0 0 0 S 0 0.0 0:46.29 ksoftirqd/11 > 27 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/12 > 28 root 15 -5 0 0 0 S 0 0.0 7:16.48 ksoftirqd/12 > 29 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/13 > 30 root 15 -5 0 0 0 S 0 0.0 23:07.36 ksoftirqd/13 > 31 root RT -5 0 0 0 S 0 0.0 0:00.18 migration/14 > 32 root 15 -5 0 0 0 S 0 0.0 31:54.13 ksoftirqd/14 > 33 root RT -5 0 0 0 S 0 0.0 0:00.12 migration/15 > 35 root 15 -5 0 0 0 S 0 0.0 1:27.02 events/0 > 36 root 15 -5 0 0 0 S 0 0.0 1:25.28 events/1 > 37 root 15 -5 0 0 0 S 0 0.0 1:23.20 events/2 > 38 root 15 -5 0 0 0 S 0 0.0 1:23.05 events/3 > 39 root 15 -5 0 0 0 S 0 0.0 1:25.72 events/4 > 40 root 15 -5 0 0 0 S 0 0.0 1:27.81 events/5 > 41 root 15 -5 0 0 0 S 0 0.0 1:25.48 events/6 > 42 root 15 -5 0 0 0 S 0 0.0 1:35.03 events/7 > 43 root 15 -5 0 0 0 S 0 0.0 1:41.28 events/8 > 44 root 15 -5 0 0 0 S 0 0.0 1:47.56 events/9 > 45 root 15 -5 0 0 0 S 0 0.0 1:27.21 events/10 > 46 root 15 -5 0 0 0 S 0 0.0 1:35.71 events/11 > 47 root 15 -5 0 0 0 S 0 0.0 1:28.55 events/12 > > > # xm info > host : serverxen03 > release : 2.6.27.19-5-xen > version : #1 SMP 2009-02-28 04:40:21 +0100 > machine : x86_64 > nr_cpus : 16 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 2 > cpu_mhz : 2533 > hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 > virt_caps : hvm > total_memory : 32757 > free_memory : 15432 > max_free_memory : 16966 > max_para_memory : 16962 > max_hvm_memory : 16914 > node_to_cpu : node0:0-15 > node_to_memory : node0:15432 > xen_major : 3 > xen_minor : 4 > xen_extra : .1_19718_04-26. > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : 19718 > cc_compiler : gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) > cc_compile_by : abuild > cc_compile_domain : > cc_compile_date : Wed Oct 14 00:02:21 UTC 2009 > xend_config_format : 4 > > > # ps aux > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 1 0.0 0.0 1064 88 ? Ss 2009 0:29 init [3] > root 2 0.0 0.0 0 0 ? S< 2009 0:00 [kthreadd] > root 3 0.0 0.0 0 0 ? S< 2009 0:00 [migration/0] > root 4 0.0 0.0 0 0 ? S< 2009 33:37 [ksoftirqd/0] > root 5 0.0 0.0 0 0 ? S< 2009 0:00 [migration/1] > root 6 0.1 0.0 0 0 ? S< 2009 78:19 [ksoftirqd/1] > root 7 0.0 0.0 0 0 ? S< 2009 0:00 [migration/2] > root 8 0.0 0.0 0 0 ? S< 2009 56:19 [ksoftirqd/2] > root 9 0.0 0.0 0 0 ? S< 2009 0:00 [migration/3] > root 10 0.0 0.0 0 0 ? S< 2009 31:04 [ksoftirqd/3] > root 11 0.0 0.0 0 0 ? S< 2009 0:00 [migration/4] > root 12 0.0 0.0 0 0 ? S< 2009 10:07 [ksoftirqd/4] > root 13 0.0 0.0 0 0 ? S< 2009 0:00 [migration/5] > root 14 0.0 0.0 0 0 ? S< 2009 7:34 [ksoftirqd/5] > root 15 0.0 0.0 0 0 ? S< 2009 0:00 [migration/6] > root 16 0.0 0.0 0 0 ? S< 2009 5:30 [ksoftirqd/6] > root 17 0.0 0.0 0 0 ? S< 2009 0:00 [migration/7] > root 18 0.0 0.0 0 0 ? S< 2009 3:32 [ksoftirqd/7] > root 19 0.0 0.0 0 0 ? S< 2009 0:00 [migration/8] > root 20 0.0 0.0 0 0 ? S< 2009 0:02 [ksoftirqd/8] > root 21 0.0 0.0 0 0 ? S< 2009 0:00 [migration/9] > root 22 0.0 0.0 0 0 ? S< 2009 0:03 [ksoftirqd/9] > root 23 0.0 0.0 0 0 ? S< 2009 0:00 [migration/10] > root 24 0.0 0.0 0 0 ? S< 2009 1:53 [ksoftirqd/10] > root 25 0.0 0.0 0 0 ? S< 2009 0:00 [migration/11] > root 26 0.0 0.0 0 0 ? S< 2009 0:46 [ksoftirqd/11] > root 27 0.0 0.0 0 0 ? S< 2009 0:00 [migration/12] > root 28 0.0 0.0 0 0 ? S< 2009 7:16 [ksoftirqd/12] > root 29 0.0 0.0 0 0 ? S< 2009 0:00 [migration/13] > root 30 0.0 0.0 0 0 ? S< 2009 23:07 [ksoftirqd/13] > root 31 0.0 0.0 0 0 ? S< 2009 0:00 [migration/14] > root 32 0.0 0.0 0 0 ? S< 2009 31:54 [ksoftirqd/14] > root 33 0.0 0.0 0 0 ? S< 2009 0:00 [migration/15] > root 34 0.0 0.0 0 0 ? S< 2009 10:15 [ksoftirqd/15] > root 35 0.0 0.0 0 0 ? S< 2009 1:27 [events/0] > root 36 0.0 0.0 0 0 ? S< 2009 1:25 [events/1] > root 37 0.0 0.0 0 0 ? S< 2009 1:23 [events/2] > root 38 0.0 0.0 0 0 ? S< 2009 1:23 [events/3] > root 39 0.0 0.0 0 0 ? S< 2009 1:25 [events/4] > root 40 0.0 0.0 0 0 ? S< 2009 1:27 [events/5] > root 41 0.0 0.0 0 0 ? S< 2009 1:25 [events/6] > root 42 0.0 0.0 0 0 ? S< 2009 1:35 [events/7] > root 43 0.0 0.0 0 0 ? S< 2009 1:41 [events/8] > root 44 0.0 0.0 0 0 ? S< 2009 1:47 [events/9] > root 45 0.0 0.0 0 0 ? S< 2009 1:27 [events/10] > root 46 0.0 0.0 0 0 ? S< 2009 1:35 [events/11] > root 47 0.0 0.0 0 0 ? S< 2009 1:28 [events/12] > root 48 0.0 0.0 0 0 ? S< 2009 1:28 [events/13] > root 49 0.0 0.0 0 0 ? S< 2009 1:27 [events/14] > root 50 0.0 0.0 0 0 ? S< 2009 1:42 [events/15] > root 51 0.0 0.0 0 0 ? S< 2009 0:00 [khelper] > root 52 0.0 0.0 0 0 ? S< 2009 0:00 [xenwatch] > root 53 0.0 0.0 0 0 ? S< 2009 0:00 [xenbus] > root 54 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/0] > root 55 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/1] > root 56 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/2] > root 57 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/3] > root 58 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/4] > root 59 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/5] > root 60 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/6] > root 61 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/7] > root 62 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/8] > root 63 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/9] > root 64 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/10] > root 65 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/11] > root 66 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/12] > root 67 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/13] > root 68 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/14] > root 69 0.0 0.0 0 0 ? S< 2009 0:00 [kintegrityd/15] > root 70 0.0 0.0 0 0 ? S< 2009 0:29 [kblockd/0] > root 71 0.0 0.0 0 0 ? S< 2009 0:15 [kblockd/1] > root 72 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/2] > root 73 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/3] > root 74 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/4] > root 75 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/5] > root 76 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/6] > root 77 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/7] > root 78 0.0 0.0 0 0 ? S< 2009 0:06 [kblockd/8] > root 79 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/9] > root 80 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/10] > root 81 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/11] > root 82 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/12] > root 83 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/13] > root 84 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/14] > root 85 0.0 0.0 0 0 ? S< 2009 0:00 [kblockd/15] > root 86 0.0 0.0 0 0 ? S< 2009 0:00 [kacpid] > root 87 0.0 0.0 0 0 ? S< 2009 0:00 [kacpi_notify] > root 88 0.0 0.0 0 0 ? S< 2009 0:00 [cqueue] > root 89 0.0 0.0 0 0 ? S< 2009 0:00 [kseriod] > root 92 0.0 0.0 0 0 ? S< 2009 3:20 [kswapd0] > root 93 0.0 0.0 0 0 ? S< 2009 0:00 [aio/0] > root 94 0.0 0.0 0 0 ? S< 2009 0:00 [aio/1] > root 95 0.0 0.0 0 0 ? S< 2009 0:00 [aio/2] > root 96 0.0 0.0 0 0 ? S< 2009 0:00 [aio/3] > root 97 0.0 0.0 0 0 ? S< 2009 0:00 [aio/4] > root 98 0.0 0.0 0 0 ? S< 2009 0:00 [aio/5] > root 99 0.0 0.0 0 0 ? S< 2009 0:00 [aio/6] > root 100 0.0 0.0 0 0 ? S< 2009 0:00 [aio/7] > root 101 0.0 0.0 0 0 ? S< 2009 0:00 [aio/8] > root 102 0.0 0.0 0 0 ? S< 2009 0:00 [aio/9] > root 103 0.0 0.0 0 0 ? S< 2009 0:00 [aio/10] > root 104 0.0 0.0 0 0 ? S< 2009 0:00 [aio/11] > root 105 0.0 0.0 0 0 ? S< 2009 0:00 [aio/12] > root 106 0.0 0.0 0 0 ? S< 2009 0:00 [aio/13] > root 107 0.0 0.0 0 0 ? S< 2009 0:00 [aio/14] > root 108 0.0 0.0 0 0 ? S< 2009 0:00 [aio/15] > root 109 0.0 0.0 0 0 ? S< 2009 0:00 [kpsmoused] > root 267 0.0 0.0 0 0 ? S< 2009 0:14 [ata/0] > root 268 0.0 0.0 0 0 ? S< 2009 0:14 [ata/1] > root 269 0.0 0.0 0 0 ? S< 2009 0:00 [ata/2] > root 270 0.0 0.0 0 0 ? S< 2009 0:00 [ata/3] > root 271 0.0 0.0 0 0 ? S< 2009 0:00 [ata/4] > root 272 0.0 0.0 0 0 ? S< 2009 0:00 [ata/5] > root 273 0.0 0.0 0 0 ? S< 2009 0:00 [ata/6] > root 274 0.0 0.0 0 0 ? S< 2009 0:00 [ata/7] > root 275 0.0 0.0 0 0 ? S< 2009 0:00 [ata/8] > root 276 0.0 0.0 0 0 ? S< 2009 0:00 [ata/9] > root 277 0.0 0.0 0 0 ? S< 2009 0:00 [ata/10] > root 278 0.0 0.0 0 0 ? S< 2009 0:00 [ata/11] > root 279 0.0 0.0 0 0 ? S< 2009 0:00 [ata/12] > root 280 0.0 0.0 0 0 ? S< 2009 0:00 [ata/13] > root 281 0.0 0.0 0 0 ? S< 2009 0:00 [ata/14] > root 282 0.0 0.0 0 0 ? S< 2009 0:00 [ata/15] > root 283 0.0 0.0 0 0 ? S< 2009 0:00 [ata_aux] > root 304 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_eh_0] > root 305 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_eh_1] > root 457 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/0] > root 458 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/1] > root 459 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/2] > root 460 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/3] > root 461 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/4] > root 462 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/5] > root 463 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/6] > root 464 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/7] > root 465 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/8] > root 466 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/9] > root 467 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/10] > root 468 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/11] > root 469 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/12] > root 470 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/13] > root 471 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/14] > root 472 0.0 0.0 0 0 ? S< 2009 0:00 [net_accel/15] > root 711 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/0] > root 712 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/1] > root 713 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/2] > root 717 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/3] > root 718 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/4] > root 719 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/5] > root 721 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/6] > root 722 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/7] > root 723 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/8] > root 724 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/9] > root 725 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/10] > root 727 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/11] > root 728 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/12] > root 729 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/13] > root 730 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/14] > root 731 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_tgtd/15] > root 805 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_eh_2] > root 878 0.0 0.0 0 0 ? S< 2009 0:00 [ksuspend_usbd] > root 883 0.0 0.0 0 0 ? S< 2009 0:00 [khubd] > root 1164 0.0 0.0 0 0 ? S< 2009 0:00 [qla2xxx_2_dpc] > root 1165 0.0 0.0 0 0 ? S< 2009 0:00 [scsi_wq_2] > root 1166 0.0 0.0 0 0 ? S< 2009 0:00 [fc_wq_2] > root 1167 0.0 0.0 0 0 ? S< 2009 0:00 [fc_dl_2] > root 1525 0.0 0.0 0 0 ? S< 2009 6:04 [kjournald] > root 1599 0.0 0.0 17272 416 ? S<s 2009 0:00 /sbin/udevd --daemon > root 2719 0.2 0.0 0 0 ? SN 2009 154:51 [kipmi0] > root 2840 0.0 0.0 0 0 ? S< 2009 0:00 [kstriped] > root 3579 0.0 0.0 19376 736 ? Ss 2009 0:01 /sbin/syslog-ng > root 3582 0.0 0.0 4092 616 ? Ss 2009 0:00 /sbin/klogd -c 1 -x > 100 3591 0.0 0.0 21628 1024 ? Ss 2009 0:07 /bin/dbus-daemon --system > root 3760 0.0 0.0 3824 376 ? Ss 2009 0:00 /sbin/acpid > 101 3955 0.0 0.1 34272 2100 ? Ss 2009 0:26 /usr/sbin/hald --daemon=yes > root 3991 0.0 0.0 98404 1296 ? Ssl 2009 0:01 /usr/sbin/console-kit-daemon > root 3993 0.0 0.0 17848 656 ? S 2009 0:00 hald-runner > root 4349 0.0 0.1 85496 3288 ? Rs 13:54 0:00 sshd: root@pts/3 > root 4366 0.0 0.0 24100 636 ? S 2009 0:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event2 /dev/input/event5 > root 4382 0.0 0.1 22068 3160 pts/3 Ss 13:54 0:00 -bash > root 4424 0.0 0.0 24104 840 ? S 2009 2:23 hald-addon-storage: polling /dev/sr0 (every 16 sec) > 101 4427 0.0 0.0 25856 668 ? S 2009 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket > root 4576 0.0 0.0 29316 468 ? S<sl 2009 0:01 /sbin/auditd -s disable > root 4579 0.0 0.0 14248 488 ? S<sl 2009 0:03 /sbin/audispd > root 4581 0.0 0.0 0 0 ? S< 2009 0:00 [kauditd] > root 4669 0.0 0.0 18712 400 ? Ss 2009 0:03 /sbin/rpcbind > root 4680 0.0 0.0 0 0 ? S< 2009 0:00 [user_dlm] > root 4691 0.0 0.0 0 0 ? S< 2009 3:27 [o2net] > root 5074 0.3 0.0 85708 980 ? Sl Jan14 42:26 /usr/lib/xen/bin/qemu-dm -d 16 -domain-name servervre06 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:16:3e:74: > root 5081 0.0 0.0 8908 424 ? Ss 2009 24:36 /usr/sbin/irqbalance > root 5116 0.0 0.0 0 0 ? S< 2009 2:30 [o2hb-D255FD5483] > root 5255 0.0 0.0 65876 956 ? Ss 2009 0:00 /usr/sbin/cupsd > root 5257 0.0 0.0 0 0 ? S< Jan14 6:39 [loop14] > root 5287 0.0 0.0 51728 588 ? Ss 2009 0:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid > root 5553 0.0 0.0 37920 724 ? Ss 2009 0:14 /usr/lib/postfix/master > root 5561 0.0 0.0 0 0 ? S< Jan14 1:24 [loop15] > root 5583 0.0 0.0 14608 460 ? Ss 2009 0:06 /usr/sbin/cron > postfix 5585 0.0 0.0 39816 716 ? S 2009 0:02 qmgr -l -t fifo -u > root 5604 2.0 0.0 8612 1016 ? S 2009 1138:10 xenstored --pid-file /var/run/xenstore.pid > root 5609 0.0 0.0 107496 1004 ? S 2009 0:00 /usr/bin/python2.6 /usr/sbin/xend start > root 5610 0.7 48.3 1903980 1012724 ? Sl 2009 422:09 /usr/bin/python2.6 /usr/sbin/xend start > root 5611 0.0 0.0 20640 484 ? Ssl 2009 0:58 blktapctrl > root 5613 0.0 0.0 22716 484 ? Sl 2009 0:00 xenconsoled > root 5689 0.0 0.0 103396 1616 ? S 2009 0:00 python /usr/lib64/python2.6/site-packages/xen/xend/server/HalDaemon.py > root 5704 0.1 0.1 220764 3424 ? Sl 2009 91:20 /usr/sbin/libvirtd -d -l > root 5761 0.0 0.0 0 0 ? S< Jan14 0:19 [blkback.16.hda] > root 5764 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2_wq] > root 5765 0.0 0.0 0 0 ? S< Jan14 0:40 [blkback.16.hdb] > root 5766 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2dc] > root 5767 0.0 0.0 0 0 ? S< 2009 0:05 [dlm_thread] > root 5768 0.0 0.0 0 0 ? S< 2009 0:04 [dlm_reco_thread] > root 5769 0.0 0.0 0 0 ? S< 2009 0:00 [dlm_wq] > root 5770 0.0 0.0 0 0 ? S< 2009 7:34 [kjournald2] > root 5771 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2cmt] > root 5775 0.0 0.0 0 0 ? S< 2009 2:31 [o2hb-A773AA5249] > root 5776 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2dc] > root 5777 0.0 0.0 0 0 ? S< 2009 0:05 [dlm_thread] > root 5778 0.0 0.0 0 0 ? S< 2009 0:03 [dlm_reco_thread] > root 5779 0.0 0.0 0 0 ? S< 2009 0:00 [dlm_wq] > root 5780 0.0 0.0 0 0 ? S< 2009 2:45 [kjournald2] > root 5781 0.0 0.0 0 0 ? S< 2009 0:00 [ocfs2cmt] > root 5799 0.0 0.0 4316 420 tty1 Ss+ 2009 0:00 /sbin/mingetty --noclear tty1 > root 5800 0.0 0.0 4316 420 tty2 Ss+ 2009 0:00 /sbin/mingetty tty2 > root 5801 0.0 0.0 4316 408 tty3 Ss+ 2009 0:00 /sbin/mingetty tty3 > root 5802 0.0 0.0 4316 408 tty4 Ss+ 2009 0:00 /sbin/mingetty tty4 > root 5803 0.0 0.0 4316 408 tty5 Ss+ 2009 0:00 /sbin/mingetty tty5 > root 5804 0.0 0.0 4316 408 tty6 Ss+ 2009 0:00 /sbin/mingetty tty6 > root 7736 0.0 0.0 0 0 ? S 14:01 0:00 [pdflush] > root 8351 0.8 1.0 217312 22920 ? Sl 2009 459:39 /usr/lib/xen/bin/qemu-dm -d 4 -domain-name servercwi02 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -serial pty -acpi -usb -usbdevice tablet -net ni > root 8659 0.0 0.0 0 0 ? S< 2009 0:00 [loop10] > root 8821 0.0 0.0 0 0 ? S< 2009 0:00 [loop11] > root 8964 0.1 0.0 85272 756 ? Sl 2009 96:45 /usr/lib64/xen/bin/qemu-dm -d 5 -domain-name servernoc03 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv > root 9241 0.0 0.0 0 0 ? S< 2009 2:27 [loop12] > root 9309 0.0 0.0 0 0 ? S< 2009 0:17 [blkback.5.xvda] > root 10448 0.8 1.4 592144 30420 ? Sl 2009 458:11 /usr/lib/xen/bin/qemu-dm -d 6 -domain-name serverpam2 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 2 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:50:8b:c2:6a > root 10722 0.0 0.0 0 0 ? S< 2009 0:00 [loop13] > root 11674 0.1 0.0 86540 768 ? Sl Jan11 27:43 /usr/lib64/xen/bin/qemu-dm -d 14 -domain-name serverrly05 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv > root 11887 0.0 0.0 0 0 ? S< Jan11 4:43 [loop0] > root 11924 0.0 0.1 85496 3340 ? Ss 19:44 0:00 sshd: root@pts/8 > root 11950 0.0 0.0 0 0 ? S< Jan11 0:35 [blkback.14.xvda] > root 11962 0.0 0.1 21816 2992 pts/8 Ss+ 19:44 0:00 -bash > root 13009 0.3 0.0 81964 972 ? Sl Jan07 66:57 /usr/lib/xen/bin/qemu-dm -d 12 -domain-name servervre04 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:16:3e:18: > root 13185 0.0 0.0 0 0 ? S< Jan07 2:05 [loop18] > root 13456 0.0 0.0 0 0 ? S< Jan07 3:24 [loop19] > root 13614 0.0 0.0 0 0 ? S< Jan07 0:30 [blkback.12.hda] > root 13615 0.0 0.0 0 0 ? S< Jan07 0:26 [blkback.12.hdb] > root 14024 0.2 0.0 85708 976 ? Sl Jan07 65:47 /usr/lib/xen/bin/qemu-dm -d 13 -domain-name servervre02 -videoram 4 -vnc 127.0.0.1:0 -vncunused -vcpus 1 -boot c -localtime -acpi -usb -net nic,vlan=1,macaddr=00:16:3e:2d: > root 14319 0.0 0.0 0 0 ? S< Jan07 1:39 [loop16] > root 14548 0.0 0.0 0 0 ? S< Jan07 0:36 [loop17] > root 14683 0.0 0.0 0 0 ? S< Jan07 0:25 [blkback.13.hda] > root 14684 0.0 0.0 0 0 ? S< Jan07 0:03 [blkback.13.hdb] > postfix 18434 0.0 0.1 45400 2280 ? S 19:58 0:00 pickup -l -t fifo -u > root 18476 0.0 0.0 12960 1052 pts/3 R+ 20:03 0:00 ps aux > root 21073 0.1 0.0 84740 760 ? Sl Jan14 20:39 /usr/lib64/xen/bin/qemu-dm -d 15 -domain-name serverhpd01 -videoram 4 -vnc 127.0.0.1:0 -vncunused -serial pty -M xenpv > root 21254 0.0 0.0 0 0 ? S< Jan14 2:02 [loop1] > root 21288 0.0 0.0 0 0 ? S< Jan14 0:14 [blkback.15.xvda] > root 27014 0.0 0.0 0 0 ? S Jan20 0:01 [pdflush] > > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
jpp@jppozzi.dyndns.org
2010-Jan-23 22:16 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
Le samedi 23 janvier 2010 à 18:13 +0200, Pasi Kärkkäinen a écrit :> On Sat, Jan 23, 2010 at 11:56:35AM -0500, James Pifer wrote: > > > > > > > > > [2010-01-22 19:53:18 5610] ERROR (SrvBase:88) Request start failed. > > > Traceback (most recent call last): > > > File "/usr/lib64/python2.6/site-packages/xen/web/SrvBase.py", line > > > 85, in perform > > > return op_method(op, req) > > > File > > > "/usr/lib64/python2.6/site-packages/xen/xend/server/SrvDomain.py", > > > line 77, in op_start > > > return self.xd.domain_start(self.dom.getName(), paused) > > > File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomain.py", > > > line 1040, in domain_start > > > dominfo.start(is_managed = True) > > > File > > > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > > > 459, in start > > > XendTask.log_progress(31, 60, self._initDomain) > > > File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line > > > 209, in log_progress > > > retval = func(*args, **kwds) > > > File > > > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > > > 2594, in _initDomain > > > self._createDevices() > > > File > > > "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line > > > 2218, in _createDevices > > > self.image.createDeviceModel() > > > File "/usr/lib64/python2.6/site-packages/xen/xend/image.py", line > > > 419, in createDeviceModel > > > pid = os.fork() > > > OSError: [Errno 12] Cannot allocate memory > > > > > > > > > Uhm.. I wonder what memory it fails to allocate. > > > > Me too.... > > > > Did you already open a case with Novell? You''re using an enterpise distribution > with full support included so just open a bug case.. > > They might have more ideas about what''s happening.. > I can''t see anything obviously wrong in the stuff you''ve provided. > > -- Pasi >Hello, I have had "cannot allocate memory" problems after having a try whith the XEN 4.0 rc1 (which don''t work on my system) and reverting to 3.4.3. I had to reinstall all the 3.4 stuff and correct the "dom0_mem" in the grub.cfg and the "dom0_min_mem" in xend-config.sxp (making both of them equal and disabling balloonning ... I don''t know why, but I have not seen any "cannot allocate memory" since the update. Regards JPP _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-24 02:09 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> > > > Did you already open a case with Novell? You''re using an enterpise distribution > > with full support included so just open a bug case.. > > > > They might have more ideas about what''s happening.. > > I can''t see anything obviously wrong in the stuff you''ve provided. > > > > -- Pasi > > > > Hello, > > I have had "cannot allocate memory" problems after having a try whith > the XEN 4.0 rc1 (which don''t work on my system) and reverting to 3.4.3. > I had to reinstall all the 3.4 stuff and correct the "dom0_mem" in the > grub.cfg and the "dom0_min_mem" in xend-config.sxp (making both of them > equal and disabling balloonning ... I don''t know why, but I have not > seen any "cannot allocate memory" since the update. > > Regards > > JPP > >How long ago was this? I ask because the cannot allocate memory error can take a long time to show up. In this case it was 39 days. I have both of those mem settings set, so I don''t think that''s my issue. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-24 02:17 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Sat, 2010-01-23 at 23:02 +0100, jpp@jppozzi.dyndns.org wrote:> Hello, > > What is the output of "xm top" and the memory size for the crashing > domU ? > > Regards > > JP Pozzi >xentop - 19:14:32 Xen 3.4.1_19718_04-26. 9 domains: 1 running, 8 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown Mem: 33543988k total, 17741584k used, 15802404k free CPUs: 16 @ 2533MHz NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID Domain-0 -----r 1246289 109.5 2095616 6.2 no limit n/a 16 0 0 0 0 0 0 0 0 servercwi0 --b--- 75887 1.5 1052588 3.1 1052672 3.1 2 1 0 0 2 0 0 0 0 serverhpd0 --b--- 221480 0.0 4194304 12.5 4194304 12.5 4 1 180634125 732323572 1 0 42849 778081 0 servernoc0 --b--- 4407 0.0 3145728 9.4 3145728 9.4 4 1 33771 1808661 1 7 29602 912013 0 serverpam2 --b--- 237517 2.8 1052588 3.1 1052672 3.1 2 1 0 0 1 0 0 0 0 serverrly0 --b--- 169632 0.5 4194304 12.5 4194304 12.5 4 1 122085682 105533360 1 0 306821 1716024 0 servervre0 --b--- 32635 1.3 528300 1.6 528384 1.6 1 1 535378 13214394 2 28 190040 1548933 0 servervre0 --b--- 39299 1.0 528300 1.6 528384 1.6 1 1 890550 12268039 2 290 1143690 3083914 0 servervre0 --b--- 17685 1.4 528300 1.6 528384 1.6 1 1 1182612 4006256 2 0 1520182 5525180 0 By crashing I assume you mean the cannot allocate memory? Only 1024meg. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Moi même
2010-Jan-24 12:35 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
Le samedi 23 janvier 2010 à 21:17 -0500, James Pifer a écrit :> On Sat, 2010-01-23 at 23:02 +0100, jpp@jppozzi.dyndns.org wrote: > > Hello, > > > > What is the output of "xm top" and the memory size for the crashing > > domU ? > > > > Regards > > > > JP Pozzi > > > > xentop - 19:14:32 Xen 3.4.1_19718_04-26. > 9 domains: 1 running, 8 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown > Mem: 33543988k total, 17741584k used, 15802404k free CPUs: 16 @ 2533MHz > NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID > Domain-0 -----r 1246289 109.5 2095616 6.2 no limit n/a 16 0 0 0 0 0 0 0 0 > servercwi0 --b--- 75887 1.5 1052588 3.1 1052672 3.1 2 1 0 0 2 0 0 0 0 > serverhpd0 --b--- 221480 0.0 4194304 12.5 4194304 12.5 4 1 180634125 732323572 1 0 42849 778081 0 > servernoc0 --b--- 4407 0.0 3145728 9.4 3145728 9.4 4 1 33771 1808661 1 7 29602 912013 0 > serverpam2 --b--- 237517 2.8 1052588 3.1 1052672 3.1 2 1 0 0 1 0 0 0 0 > serverrly0 --b--- 169632 0.5 4194304 12.5 4194304 12.5 4 1 122085682 105533360 1 0 306821 1716024 0 > servervre0 --b--- 32635 1.3 528300 1.6 528384 1.6 1 1 535378 13214394 2 28 190040 1548933 0 > servervre0 --b--- 39299 1.0 528300 1.6 528384 1.6 1 1 890550 12268039 2 290 1143690 3083914 0 > servervre0 --b--- 17685 1.4 528300 1.6 528384 1.6 1 1 1182612 4006256 2 0 1520182 5525180 0 > > > By crashing I assume you mean the cannot allocate memory? Only 1024meg. > > Thanks, > James >Hello, I see that your Dom0 has 33543988k #32GB, is it all the memory in the server? If so it is possible that after some "ballooning" and memory allocation the memory is too unorganized that the allocator can''t find suitable chunks of memory. I had had that problem after a system upgrade where the "menu.lst" (or grub.fcg) was recreated without the tuning parameters. I think that 32GB is fa too much for the Dom0, usually Dom0 is limited to a value between 1024M and 3072M with the dom0_mem=xxxx in the grub.cfg, the memory of Dom0 is mainly used for buffer cache on behalf of the Domus. It is also often useful to use the parameter "dom0_min_mem=xxx" in the xend.config file. In my systems I use the same value eg : (dom0-min-mem 2048) (enable-dom0-ballooning no) in xend-config.sxo and multiboot /xen-3.4.gz /xen-3.4.gz noreboot dom0_mem=2048M in grub.cfg. I have had problems with "ballonning yes" if domUs are stopped and started many times without a server reboot (memory fargmentation ?). Regards JP Pozzi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-24 14:47 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> > > > Hello, > > I see that your Dom0 has 33543988k #32GB, is it all the memory in the > server? > If so it is possible that after some "ballooning" and memory allocation > the memory is too unorganized that the allocator can''t find suitable > chunks of memory. I had had that problem after a system upgrade where > the "menu.lst" (or grub.fcg) was recreated without the tuning > parameters. > I think that 32GB is fa too much for the Dom0, usually Dom0 is limited > to a value between 1024M and 3072M with the dom0_mem=xxxx in the > grub.cfg, the memory of Dom0 is mainly used for buffer cache on behalf > of the Domus. > It is also often useful to use the parameter "dom0_min_mem=xxx" in the > xend.config file. In my systems I use the same value eg : > (dom0-min-mem 2048) > (enable-dom0-ballooning no) > in xend-config.sxo > and > multiboot /xen-3.4.gz /xen-3.4.gz noreboot dom0_mem=2048M > in grub.cfg. > I have had problems with "ballonning yes" if domUs are stopped and > started many times without a server reboot (memory fargmentation ?). > > Regards > > JP PozziNo, dom0 has 4gb (was lower but was trying 4gb to see if it helped) dom0_min_mem is set to 512M Ballooning is disabled.>From menu.lst:###Don''t change this comment - YaST2 identifier: Original name: xen### title Xen -- SUSE Linux Enterprise Server 11 - 2.6.27.19-5 root (hd0,1) kernel /boot/xen.gz dom0_mem=4096M module /boot/vmlinuz-2.6.27.19-5-xen root=/dev/disk/by-id/scsi-200d0b29d2d004400-part2 resume=/dev/disk/by-id/scsi-200d0b29d2d004400-part1 splash=silent showopts vga=0x317 module /boot/initrd-2.6.27.19-5-xen See xen-config.sxp below. Thanks, James # cat xend-config.sxp # -*- sh -*- # # Xend configuration file. # # This example configuration is appropriate for an installation that # utilizes a bridged network configuration. Access to xend via http # is disabled. # Commented out entries show the default for that entry, unless otherwise # specified. #(logfile /var/log/xen/xend.log) #(loglevel DEBUG) # Uncomment the line below. Set the value to flask, acm, or dummy to # select a security module. #(xsm_module_name dummy) # The Xen-API server configuration. # # This value configures the ports, interfaces, and access controls for the # Xen-API server. Each entry in the list starts with either unix, a port # number, or an address:port pair. If this is "unix", then a UDP socket is # opened, and this entry applies to that. If it is a port, then Xend will # listen on all interfaces on that TCP port, and if it is an address:port # pair, then Xend will listen on the specified port, using the interface with # the specified address. # # The subsequent string configures the user-based access control for the # listener in question. This can be one of "none" or "pam", indicating either # that users should be allowed access unconditionally, or that the local # Pluggable Authentication Modules configuration should be used. If this # string is missing or empty, then "pam" is used. # # The final string gives the host-based access control for that listener. If # this is missing or empty, then all connections are accepted. Otherwise, # this should be a space-separated sequence of regular expressions; any host # with a fully-qualified domain name or an IP address that matches one of # these regular expressions will be accepted. # # Example: listen on TCP port 9363 on all interfaces, accepting connections # only from machines in example.com or localhost, and allow access through # the unix domain socket unconditionally: # # (xen-api-server ((9363 pam ''^localhost$ example\\.com$'') # (unix none))) # # Optionally, the TCP Xen-API server can use SSL by specifying the private # key and certificate location: # # (9367 pam '''' /etc/xen/xen-api.key /etc/xen/xen-api.crt) # # Default: # (xen-api-server ((unix))) #(xend-http-server no) (xend-unix-server yes) #(xend-tcp-xmlrpc-server no) #(xend-unix-xmlrpc-server yes) # Only enable xend-relocation-server on trusted networks as it lacks # encryption and authentication. #(xend-relocation-server no) #(xend-relocation-ssl-server no) #(xend-udev-event-server no) #(xend-unix-path /var/lib/xend/xend-socket) # Address and port xend should use for the legacy TCP XMLRPC interface, # if xend-tcp-xmlrpc-server is set. #(xend-tcp-xmlrpc-server-address ''localhost'') #(xend-tcp-xmlrpc-server-port 8006) # SSL key and certificate to use for the legacy TCP XMLRPC interface. # Setting these will mean that this port serves only SSL connections as # opposed to plaintext ones. #(xend-tcp-xmlrpc-server-ssl-key-file /etc/xen/xmlrpc.key) #(xend-tcp-xmlrpc-server-ssl-cert-file /etc/xen/xmlrpc.crt) # Port xend should use for the HTTP interface, if xend-http-server is set. #(xend-port 8000) # Port xend should use for the relocation interface, if xend-relocation-server # is set. #(xend-relocation-port 8002) # Port xend should use for the ssl relocation interface, if # xend-relocation-ssl-server is set. #(xend-relocation-ssl-port 8003) # SSL key and certificate to use for the ssl relocation interface, if # xend-relocation-ssl-server is set. #(xend-relocation-server-ssl-key-file /etc/xen/xmlrpc.key) #(xend-relocation-server-ssl-cert-file /etc/xen/xmlrpc.crt) # Whether to use ssl as default when relocating. #(xend-relocation-ssl no) # Address xend should listen on for HTTP connections, if xend-http-server is # set. # Specifying ''localhost'' prevents remote connections. # Specifying the empty string '''' (the default) allows all connections. #(xend-address '''') #(xend-address localhost) # Address xend should listen on for relocation-socket connections, if # xend-relocation-server is set. # Meaning and default as for xend-address above. #(xend-relocation-address '''') # The hosts allowed to talk to the relocation port. If this is empty (the # default), then all connections are allowed (assuming that the connection # arrives on a port and interface on which we are listening; see # xend-relocation-port and xend-relocation-address above). Otherwise, this # should be a space-separated sequence of regular expressions. Any host with # a fully-qualified domain name or an IP address that matches one of these # regular expressions will be accepted. # # For example: # (xend-relocation-hosts-allow ''^localhost$ ^.*\\.example\\.org$'') # #(xend-relocation-hosts-allow '''') (xend-relocation-hosts-allow ''^localhost$ ^localhost\\.localdomain$'') # The limit (in kilobytes) on the size of the console buffer #(console-limit 1024) ## # To bridge network traffic, like this: # # dom0: ----------------- bridge -> real eth0 -> the network # | # domU: fake eth0 -> vifN.0 -+ # # use # # (network-script network-bridge) # # Your default ethernet device is used as the outgoing interface, by default. # To use a different one (e.g. eth1) use # # (network-script ''network-bridge netdev=eth1'') # # The bridge takes on the ethernet device name by default. To rename the # bridge, use # # (network-script ''network-bridge bridge=<name>'') # # It is possible to use the network-bridge script in more complicated # scenarios, such as having two outgoing interfaces, with two bridges, and # two fake interfaces per guest domain. To do things like this, write # yourself a wrapper script, and call network-bridge from it, as appropriate. # # SuSE users note: # On openSUSE >= 11.1 and SLES >= 11, networks should be configured using # native platform tool - YaST. vif-bridge and qemu-ifup can be used to # connect vifs to the YaST-managed networks. #(network-script network-bridge) (network-script ) # The script used to control virtual interfaces. This can be overridden on a # per-vif basis when creating a domain or a configuring a new vif. The # vif-bridge script is designed for use with the network-bridge script, or # similar configurations. # # If you have overridden the bridge name using # (network-script ''network-bridge bridge=<name>'') then you may wish to do the # same here. The bridge name can also be set when creating a domain or # configuring a new vif, but a value specified here would act as a default. # # If you are using only one bridge, the vif-bridge script will discover that, # so there is no need to specify it explicitly. # (vif-script vif-bridge) ## Use the following if network traffic is routed, as an alternative to the # settings for bridged networking given above. #(network-script network-route) #(vif-script vif-route) ## Use the following if network traffic is routed with NAT, as an alternative # to the settings for bridged networking given above. #(network-script network-nat) #(vif-script vif-nat) # dom0-min-mem is the lowest permissible memory level (in MB) for dom0. # This is a minimum both for auto-ballooning (as enabled by # enable-dom0-ballooning below) and for xm mem-set when applied to dom0. (dom0-min-mem 512) # Whether to enable auto-ballooning of dom0 to allow domUs to be created. # If enable-dom0-ballooning = no, dom0 will never balloon out. (enable-dom0-ballooning no) # In SMP system, dom0 will use dom0-cpus # of CPUS # If dom0-cpus = 0, dom0 will take all cpus available (dom0-cpus 0) # Whether to enable core-dumps when domains crash. #(enable-dump no) # The tool used for initiating virtual TPM migration #(external-migration-tool '''') # The interface for VNC servers to listen on. Defaults # to 127.0.0.1 To restore old ''listen everywhere'' behaviour # set this to 0.0.0.0 #(vnc-listen ''127.0.0.1'') # The default password for VNC console on HVM domain. # Empty string is no authentication. (vncpasswd '''') # The VNC server can be told to negotiate a TLS session # to encryption all traffic, and provide x509 cert to # clients enalbing them to verify server identity. The # GTK-VNC widget, virt-viewer, virt-manager and VeNCrypt # all support the VNC extension for TLS used in QEMU. The # TightVNC/RealVNC/UltraVNC clients do not. # # To enable this create x509 certificates / keys in the # directory /etc/xen/vnc # # ca-cert.pem - The CA certificate # server-cert.pem - The Server certificate signed by the CA # server-key.pem - The server private key # # and then uncomment this next line # (vnc-tls 1) # The certificate dir can be pointed elsewhere.. # # (vnc-x509-cert-dir /etc/xen/vnc) # The server can be told to request & validate an x509 # certificate from the client. Only clients with a cert # signed by the trusted CA will be able to connect. This # is more secure the password auth alone. Passwd auth can # used at the same time if desired. To enable client cert # checking uncomment this: # # (vnc-x509-verify 1) # The default keymap to use for the VM''s virtual keyboard # when not specififed in VM''s configuration #(keymap ''en-us'') # Script to run when the label of a resource has changed. #(resource-label-change-script '''') # Rotation count of qemu-dm log file. #(qemu-dm-logrotate-count 10) # Path where persistent domain configuration is stored. # Default is /var/lib/xend/domains/ # #(xend-domains-path /var/lib/xend/domains) # Domain Locking # In a multihost environment, domain locking prevents simultaneously # running a domain on more than one host. # # If enabled, xend will execute a external lock utility (defined below) # on each domain start and stop event. Disabled by default. Set to yes # to enable domain locking. # #(xend-domain-lock no) # Path where domain lock is stored if xend-domain-lock is enabled. # Note: This path must be accessible to all VM Servers participating # in domain locking, e.g. by specifying a shared mount point. # Lock is placed in /<xend-domain-lock-path>/<domain-uuid>. # Default is /var/lib/xend/domains/ # #(xend-domain-lock-path /var/lib/xend/domains) # External locking utility called by xend for acquiring/releasing # domain lock. By default /etc/xen/scripts/domain-lock will be used # if xend-domain-lock is set to yes. Set to path of custom locking # utility to override the default. # # Synopsis of lock-util: # lock-util [-l|-u] -n <vm name> -i <vm uuid> -p <physical host> path" # -l Acquire (create) lock # -u Remove lock # -n vm-name Name of domain # -i vm-id Id or UUID of domain # -p phy-host Name of physical host (dom0) # path /<xend-domain-lock-path>/<vm-uuid> # Return 0 on success, non-zero on error. # # lock-util [-s] path" # -s Lock status. If lock is acquired, print any contents # on stdout and return 0. Return non-zero if lock is # available. # path /<xend-domain-lock-path>/<vm-uuid> # If lock is acquired, print any contents on stdout and return 0. # Return non-zero if lock is available. # # Default lock-util behavior: # On domain start event, domain-lock will create and flock(1) # /<xend-domain-lock-path>/<vm-uuid>/lock. Every two seconds it # will write <vm-name>, <vm-id>, <vm-host>, and <tick> to the lock. # <tick> is running counter. # On domain stop event, domain-lock will unlock and remove # /<xend-domain-lock-path>/<vm-uuid>/lock. # # Note: If xend-domain-lock-path is a cluster-unaware file system, # administrator intervention may be required to remove stale # locks. Consider two hosts using NFS for xend-domain-lock-path # when HostA, running vm1, crashes. HostB could not acquire a # lock for vm1 since the NFS server holds an exclusive lock # acquired by HostA. The lock file must be manually removed # before starting vm1 on HostA. # #(xend-domain-lock-utility domain-lock) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jean-Paul Pozzi
2010-Jan-25 10:25 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
----- Mail Original ----- De: "James Pifer" <jep@obrien-pifer.com> À: xen-users@lists.xensource.com Envoyé: Dimanche 24 Janvier 2010 15:47:58 Objet: Re: [Xen-users] Memory problems persist... Cannot allocate memory> > > > Hello, > > I see that your Dom0 has 33543988k #32GB, is it all the memory in the > server? > If so it is possible that after some "ballooning" and memory allocation > the memory is too unorganized that the allocator can''t find suitable > chunks of memory. I had had that problem after a system upgrade where > the "menu.lst" (or grub.fcg) was recreated without the tuning > parameters. > I think that 32GB is fa too much for the Dom0, usually Dom0 is limited > to a value between 1024M and 3072M with the dom0_mem=xxxx in the > grub.cfg, the memory of Dom0 is mainly used for buffer cache on behalf > of the Domus. > It is also often useful to use the parameter "dom0_min_mem=xxx" in the > xend.config file. In my systems I use the same value eg : > (dom0-min-mem 2048) > (enable-dom0-ballooning no) > in xend-config.sxo > and > multiboot /xen-3.4.gz /xen-3.4.gz noreboot dom0_mem=2048M > in grub.cfg. > I have had problems with "ballonning yes" if domUs are stopped and > started many times without a server reboot (memory fargmentation ?). > > Regards > > JP PozziNo, dom0 has 4gb (was lower but was trying 4gb to see if it helped) dom0_min_mem is set to 512M Ballooning is disabled.>From menu.lst:###Don''t change this comment - YaST2 identifier: Original name: xen### title Xen -- SUSE Linux Enterprise Server 11 - 2.6.27.19-5 root (hd0,1) kernel /boot/xen.gz dom0_mem=4096M module /boot/vmlinuz-2.6.27.19-5-xen root=/dev/disk/by-id/scsi-200d0b29d2d004400-part2 resume=/dev/disk/by-id/scsi-200d0b29d2d004400-part1 splash=silent showopts vga=0x317 module /boot/initrd-2.6.27.19-5-xen See xen-config.sxp below. Thanks, James Hello, If you set "dom0_mem=4096M" try to set "dom0_min_mem=4096" in xend-config.sxp You said that the system ran a long time before showing "can''t allocate memory" it could be a problem with memory fragmentation or a memory leak. I saw something about that setting a long time ago, but I always use it. Regards JP Pozzi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Jan-27 18:36 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> Hello, > > If you set "dom0_mem=4096M" try to set "dom0_min_mem=4096" in xend-config.sxp > You said that the system ran a long time before showing "can''t allocate memory" > it could be a problem with memory fragmentation or a memory leak. > I saw something about that setting a long time ago, but I always use it. >So you always set the dom0_min_mem equal to what you set the kernel parameter to? I''ll keep that in mind. I can''t change anything right now because I''ve opened a ticket with Novell and waiting to hear back. The strange thing right now is, there''s free memory in dom0, free memory in swap, and a lot of free ram in xm (from xm info). Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jan-27 20:22 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Wed, Jan 27, 2010 at 01:36:21PM -0500, James Pifer wrote:> > Hello, > > > > If you set "dom0_mem=4096M" try to set "dom0_min_mem=4096" in xend-config.sxp > > You said that the system ran a long time before showing "can''t allocate memory" > > it could be a problem with memory fragmentation or a memory leak. > > I saw something about that setting a long time ago, but I always use it. > > > > So you always set the dom0_min_mem equal to what you set the kernel > parameter to? I''ll keep that in mind. I can''t change anything right now > because I''ve opened a ticket with Novell and waiting to hear back. >Let us know how it goes..> The strange thing right now is, there''s free memory in dom0, free memory > in swap, and a lot of free ram in xm (from xm info). >Definitely sounds like a bug.. Just don''t reboot the box before they get what they need to debug it :) -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Andrew Lyon
2010-Jan-27 21:23 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
On Wed, Jan 27, 2010 at 8:22 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Wed, Jan 27, 2010 at 01:36:21PM -0500, James Pifer wrote: >> > Hello, >> > >> > If you set "dom0_mem=4096M" try to set "dom0_min_mem=4096" in xend-config.sxp >> > You said that the system ran a long time before showing "can''t allocate memory" >> > it could be a problem with memory fragmentation or a memory leak. >> > I saw something about that setting a long time ago, but I always use it. >> > >> >> So you always set the dom0_min_mem equal to what you set the kernel >> parameter to? I''ll keep that in mind. I can''t change anything right now >> because I''ve opened a ticket with Novell and waiting to hear back. >> > > Let us know how it goes.. > >> The strange thing right now is, there''s free memory in dom0, free memory >> in swap, and a lot of free ram in xm (from xm info). >> > > Definitely sounds like a bug.. Just don''t reboot the box before they > get what they need to debug it :) > > -- Pasi > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >I think I may have hit a similar bug running 3.4.2 , while testing save/restore/migrate I occasionally got "unable to allocate memory" errors and the domU failed to restore or migrate successfully, but immediately trying to start the same vm on the same system was successful, and having restarted it I was able to migrate it between two servers several times. So it appears that sometimes Xen will error that it cannot allocate memory, but a subsequent request for the same amount of memory is successful. Andy _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer
2010-Feb-01 22:13 UTC
Re: [Xen-users] Memory problems persist... Cannot allocate memory
> Let us know how it goes..So far this has not gotten very far. I''ve had a ticket open for a week and have repeatedly asked for status updates. I''m told it''s been escalated. So far the only thing I''ve been asked, outside of sending them the supportconfig they requested, was if this problem also happens if I assign 4gb of ram to dom0. First, the issue does still happen. Second, I shouldn''t have to give dom0 that much ram. We''ll see.... James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users