I haven''t seen this on the list before but I''m wondering if anyone else has experienced this or has any ideas on what I''m finding to be very strange (and bad) i/o behavior. I have soft hangs on four Debian lenny guests backed by LVM that I can trigger reliably with bonnie++ by running it and then firing off something in another session that isn''t cached and thus has to go to disk (like running ''who'' instead of ''w''). The second command will eventually come back but it takes about 90 seconds. Further testing showed that no i/o was actually going on during those 90 seconds. So as the scenario goes, start up `vmstat 1` in a shell and then `bonnie++ -r 1024` in another. Let that bonnie++ run for 15 seconds or so so you run out your buffer cache, then try to launch something else that has to read from disk. The vmstat counters will drop to 0 on bi and bo and anything requiring disk activity will hang for exactly 90 seconds. After 90 seconds, i/o picks up where it left off and everything will appear to be ok. I''ve played with i/o schedulers and various kernel versions (even running etch''s kernel) to no avail. Any ideas? This is on a CentOS 5.4 hypervisor, Debian 5 domU with 1GB memory, 2 vcpu''s, single LVM phy: disk. Thanks, John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Chris ''Xenon'' Hanson
2009-Nov-18 20:00 UTC
Re: [Xen-users] strange i/o hangs on debian lenny guests
John Madden wrote:> I haven''t seen this on the list before but I''m wondering if anyone else > has experienced this or has any ideas on what I''m finding to be very > strange (and bad) i/o behavior. I have soft hangs on four Debian lenny > guests backed by LVM that I can trigger reliably with bonnie++ by > running it and then firing off something in another session that isn''t > cached and thus has to go to disk (like running ''who'' instead of ''w''). > > This is on a CentOS 5.4 hypervisor, Debian 5 domU with 1GB memory, 2 > vcpu''s, single LVM phy: disk.I think I had some similar trouble until I upgraded my DomU''s kernel to something newer than what I had previously been using. What exact kernel is your DomU using and where did it come from? How about your DomU? See my thread "Ubuntu 8.04 DomU crash" on 5/12/2009 6:25 AM.> Thanks, > John-- Chris ''Xenon'' Hanson, omo sanza lettere Xenon AlphaPixel.com PixelSense Landsat processing now available! http://www.alphapixel.com/demos/ "There is no Truth. There is only Perception. To Perceive is to Exist." - Xen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Madden
2009-Nov-18 21:19 UTC
Re: [Xen-users] strange i/o hangs on debian lenny guests
>> This is on a CentOS 5.4 hypervisor, Debian 5 domU with 1GB memory, 2 >> vcpu''s, single LVM phy: disk. > > I think I had some similar trouble until I upgraded my DomU''s kernel to something newer > than what I had previously been using. What exact kernel is your DomU using and where did > it come from? How about your DomU?It''s 2.6.26-2-686-bigmem from Debian''s repositories, apparently the default and recommended kernel rev (if you were to install and apt-get upgrade you''d end up with this kernel). Maybe that''s significant, that it''s 32-bit on a 64-bit hypervisor, although I do have a couple other lenny domU''s in this state that don''t exhibit the issue. I haven''t been able to find any differences between them though. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users