Hi, Now that I''ve got that 3-day-long, let''s-waste-some-time-with-a- broken-dom0 episode, I can get back to the real problems that are preventing me from getting any use out of my Xen-based server. What is happening, basically, is that as soon as some guest domains get busy, the hard disks go south for the winter. I know this is not really a Xen issue (at least not directly) but as the problem only happens in a Xen setup, I thought I''d try to ask about it here. The server has four SATA hard disks. It has worked with the same hardware setup for many months, with processor- and disk-intensive applications, with no problems. Now, I can start dom0 and guest domains with no problem, and they run fine as long as they don''t try to do a lot. But as soon as they get busy - say, compiling something - the console shows an endless stream of: ata1: command 0x35 timeout, stat 0x50 host_stat 0x64 ata2: command 0x25 timeout, stat 0x50 host_stat 0x64 ata4: command 0x35 timeout, stat 0x50 host_stat 0x4 And the system stops responding, until I reset the server or throw it out the window (well, not yet, but it''s coming). I''ve seen posts saying that these errors are related to APIC, so I tried adding "noapic acpi=off" to the dom0 kernel command line. No change. If anyone''s wondering why ata3 isn''t in the list... it''s a different drive - a 250GB WD while 1, 2 and 4 are 400GB Seagates. Interesting? Any ideas/suggestions? TIA, Itai _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rick van der LInde
2006-May-11 07:22 UTC
Re: [Xen-users] HD timeouts with busy xen domains
Itai Tavor wrote:> Hi, > > Now that I''ve got that 3-day-long, let''s-waste-some-time-with-a- > broken-dom0 episode, I can get back to the real problems that are > preventing me from getting any use out of my Xen-based server. > > What is happening, basically, is that as soon as some guest domains > get busy, the hard disks go south for the winter. > > I know this is not really a Xen issue (at least not directly) but as > the problem only happens in a Xen setup, I thought I''d try to ask > about it here. > > The server has four SATA hard disks. It has worked with the same > hardware setup for many months, with processor- and disk-intensive > applications, with no problems. Now, I can start dom0 and guest > domains with no problem, and they run fine as long as they don''t try > to do a lot. But as soon as they get busy - say, compiling something > - the console shows an endless stream of: > > ata1: command 0x35 timeout, stat 0x50 host_stat 0x64 > ata2: command 0x25 timeout, stat 0x50 host_stat 0x64 > ata4: command 0x35 timeout, stat 0x50 host_stat 0x4 > > And the system stops responding, until I reset the server or throw it > out the window (well, not yet, but it''s coming). > > I''ve seen posts saying that these errors are related to APIC, so I > tried adding "noapic acpi=off" to the dom0 kernel command line. No > change. > > If anyone''s wondering why ata3 isn''t in the list... it''s a different > drive - a 250GB WD while 1, 2 and 4 are 400GB Seagates. Interesting? > > Any ideas/suggestions?Hi, Are you working with an ICH6 or ICH7 chipset for SATA connections. If zo. check your BIOS settings (IDE Operation Mode). I''ve experienced this before. Somehow the Enhanced Mode setting can cause troubles. You can have it set to "Enhanced Mode", but need to make sure only SATA is enabled (not SATA + PATA). After setting this to SATA, the PATA devices are still accessible and the timeou problems disappeared for me. Hope this may help you. Rick _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 11/05/2006, at 5:22 PM, Rick van der LInde wrote:> Itai Tavor wrote: > >> Hi, >> >> Now that I''ve got that 3-day-long, let''s-waste-some-time-with-a- >> broken-dom0 episode, I can get back to the real problems that are >> preventing me from getting any use out of my Xen-based server. >> >> What is happening, basically, is that as soon as some guest >> domains get busy, the hard disks go south for the winter. >> >> I know this is not really a Xen issue (at least not directly) but >> as the problem only happens in a Xen setup, I thought I''d try to >> ask about it here. >> >> The server has four SATA hard disks. It has worked with the same >> hardware setup for many months, with processor- and disk- >> intensive applications, with no problems. Now, I can start dom0 >> and guest domains with no problem, and they run fine as long as >> they don''t try to do a lot. But as soon as they get busy - say, >> compiling something - the console shows an endless stream of: >> >> ata1: command 0x35 timeout, stat 0x50 host_stat 0x64 >> ata2: command 0x25 timeout, stat 0x50 host_stat 0x64 >> ata4: command 0x35 timeout, stat 0x50 host_stat 0x4 >> >> And the system stops responding, until I reset the server or throw >> it out the window (well, not yet, but it''s coming). >> >> I''ve seen posts saying that these errors are related to APIC, so >> I tried adding "noapic acpi=off" to the dom0 kernel command line. >> No change. >> >> If anyone''s wondering why ata3 isn''t in the list... it''s a >> different drive - a 250GB WD while 1, 2 and 4 are 400GB Seagates. >> Interesting? >> >> Any ideas/suggestions? > > Hi, > > Are you working with an ICH6 or ICH7 chipset for SATA connections. > If zo. check your BIOS settings (IDE Operation Mode). I''ve > experienced this before. Somehow the Enhanced Mode setting can > cause troubles. You can have it set to "Enhanced Mode", but need to > make sure only SATA is enabled (not SATA + PATA). After setting > this to SATA, the PATA devices are still accessible and the timeou > problems disappeared for me. > > Hope this may help you.Thanks, Rick, but no luck... my drives are on silicon Image chips, separate from the IDE controller. Itai _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stefan Kaltenbrunner
2006-May-11 17:09 UTC
Re: [Xen-users] HD timeouts with busy xen domains
Itai Tavor wrote:> > On 11/05/2006, at 5:22 PM, Rick van der LInde wrote: > >> Itai Tavor wrote: >> >>> Hi, >>> >>> Now that I''ve got that 3-day-long, let''s-waste-some-time-with-a- >>> broken-dom0 episode, I can get back to the real problems that are >>> preventing me from getting any use out of my Xen-based server. >>> >>> What is happening, basically, is that as soon as some guest domains >>> get busy, the hard disks go south for the winter. >>> >>> I know this is not really a Xen issue (at least not directly) but >>> as the problem only happens in a Xen setup, I thought I''d try to >>> ask about it here. >>> >>> The server has four SATA hard disks. It has worked with the same >>> hardware setup for many months, with processor- and disk- intensive >>> applications, with no problems. Now, I can start dom0 and guest >>> domains with no problem, and they run fine as long as they don''t >>> try to do a lot. But as soon as they get busy - say, compiling >>> something - the console shows an endless stream of: >>> >>> ata1: command 0x35 timeout, stat 0x50 host_stat 0x64 >>> ata2: command 0x25 timeout, stat 0x50 host_stat 0x64 >>> ata4: command 0x35 timeout, stat 0x50 host_stat 0x4 >>> >>> And the system stops responding, until I reset the server or throw >>> it out the window (well, not yet, but it''s coming). >>> >>> I''ve seen posts saying that these errors are related to APIC, so I >>> tried adding "noapic acpi=off" to the dom0 kernel command line. No >>> change. >>> >>> If anyone''s wondering why ata3 isn''t in the list... it''s a >>> different drive - a 250GB WD while 1, 2 and 4 are 400GB Seagates. >>> Interesting? >>> >>> Any ideas/suggestions? >> >> >> Hi, >> >> Are you working with an ICH6 or ICH7 chipset for SATA connections. If >> zo. check your BIOS settings (IDE Operation Mode). I''ve experienced >> this before. Somehow the Enhanced Mode setting can cause troubles. >> You can have it set to "Enhanced Mode", but need to make sure only >> SATA is enabled (not SATA + PATA). After setting this to SATA, the >> PATA devices are still accessible and the timeou problems disappeared >> for me. >> >> Hope this may help you. > > > Thanks, Rick, but no luck... my drives are on silicon Image chips, > separate from the IDE controller.FWIW - I''m seeing the same issue (DMA timeouts under moderate IO loads - like building a kernel) here on both ICH7 and IT8212 controller IDE disks on xen 3.0.2 and unstable. This issue makes Xen rather unusable for me and is not reproducable without xen either ... Stefan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 12/05/2006, at 3:09 AM, Stefan Kaltenbrunner wrote:> Itai Tavor wrote: >> >> On 11/05/2006, at 5:22 PM, Rick van der LInde wrote: >> >>> Itai Tavor wrote: >>> >>>> Hi, >>>> >>>> Now that I''ve got that 3-day-long, let''s-waste-some-time-with-a- >>>> broken-dom0 episode, I can get back to the real problems that are >>>> preventing me from getting any use out of my Xen-based server. >>>> >>>> What is happening, basically, is that as soon as some guest >>>> domains >>>> get busy, the hard disks go south for the winter. >>>> >>>> I know this is not really a Xen issue (at least not directly) but >>>> as the problem only happens in a Xen setup, I thought I''d try to >>>> ask about it here. >>>> >>>> The server has four SATA hard disks. It has worked with the same >>>> hardware setup for many months, with processor- and disk- intensive >>>> applications, with no problems. Now, I can start dom0 and guest >>>> domains with no problem, and they run fine as long as they don''t >>>> try to do a lot. But as soon as they get busy - say, compiling >>>> something - the console shows an endless stream of: >>>> >>>> ata1: command 0x35 timeout, stat 0x50 host_stat 0x64 >>>> ata2: command 0x25 timeout, stat 0x50 host_stat 0x64 >>>> ata4: command 0x35 timeout, stat 0x50 host_stat 0x4 >>>> >>>> And the system stops responding, until I reset the server or throw >>>> it out the window (well, not yet, but it''s coming). >>>> >>>> I''ve seen posts saying that these errors are related to APIC, so I >>>> tried adding "noapic acpi=off" to the dom0 kernel command line. No >>>> change. >>>> >>>> If anyone''s wondering why ata3 isn''t in the list... it''s a >>>> different drive - a 250GB WD while 1, 2 and 4 are 400GB Seagates. >>>> Interesting? >>>> >>>> Any ideas/suggestions? >>> > > FWIW - I''m seeing the same issue (DMA timeouts under moderate IO > loads - > like building a kernel) here on both ICH7 and IT8212 controller IDE > disks on xen 3.0.2 and unstable. > This issue makes Xen rather unusable for me and is not reproducable > without xen either ... > > StefanThis can only be described as good news of the bad kind. Anyway... I''m upgrading the server mobo and CPU today, I''ll know soon if the problem carries over to the new setup. If I''m lucky, we''ll be able to forget about it until the next unlucky user hits it. Itai _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
For what it''s worth, Im running a Celeron 2.6 on an ECS 865PE (http://www.newegg.com/product/product.asp?item=N82E16813135152) . It''s an ICH5 865PE chipset and I have had the same issue when I ran two PATA drives on the same IDE channel and tried to RAID across them. When things got really heavy, I would get DMA timeouts and DMA would turn off on one or both of the drives. Seperating the drives onto the Master and Slave controller took care of it for me. -- Jason The place where you made your stand never mattered, only that you were there... and still on your feet On Wed, 17 May 2006, Itai Tavor wrote:> On 12/05/2006, at 3:09 AM, Stefan Kaltenbrunner wrote: > >> Itai Tavor wrote: >>> >>> On 11/05/2006, at 5:22 PM, Rick van der LInde wrote: >>> >>>> Itai Tavor wrote: >>>> >>>>> Hi, >>>>> >>>>> Now that I''ve got that 3-day-long, let''s-waste-some-time-with-a- >>>>> broken-dom0 episode, I can get back to the real problems that are >>>>> preventing me from getting any use out of my Xen-based server. >>>>> >>>>> What is happening, basically, is that as soon as some guest domains >>>>> get busy, the hard disks go south for the winter. >>>>> >>>>> I know this is not really a Xen issue (at least not directly) but >>>>> as the problem only happens in a Xen setup, I thought I''d try to >>>>> ask about it here. >>>>> >>>>> The server has four SATA hard disks. It has worked with the same >>>>> hardware setup for many months, with processor- and disk- intensive >>>>> applications, with no problems. Now, I can start dom0 and guest >>>>> domains with no problem, and they run fine as long as they don''t >>>>> try to do a lot. But as soon as they get busy - say, compiling >>>>> something - the console shows an endless stream of: >>>>> >>>>> ata1: command 0x35 timeout, stat 0x50 host_stat 0x64 >>>>> ata2: command 0x25 timeout, stat 0x50 host_stat 0x64 >>>>> ata4: command 0x35 timeout, stat 0x50 host_stat 0x4 >>>>> >>>>> And the system stops responding, until I reset the server or throw >>>>> it out the window (well, not yet, but it''s coming). >>>>> >>>>> I''ve seen posts saying that these errors are related to APIC, so I >>>>> tried adding "noapic acpi=off" to the dom0 kernel command line. No >>>>> change. >>>>> >>>>> If anyone''s wondering why ata3 isn''t in the list... it''s a >>>>> different drive - a 250GB WD while 1, 2 and 4 are 400GB Seagates. >>>>> Interesting? >>>>> >>>>> Any ideas/suggestions? >>>> >> >> FWIW - I''m seeing the same issue (DMA timeouts under moderate IO loads - >> like building a kernel) here on both ICH7 and IT8212 controller IDE >> disks on xen 3.0.2 and unstable. >> This issue makes Xen rather unusable for me and is not reproducable >> without xen either ... >> >> Stefan > > This can only be described as good news of the bad kind. > > Anyway... I''m upgrading the server mobo and CPU today, I''ll know soon if the > problem carries over to the new setup. If I''m lucky, we''ll be able to forget > about it until the next unlucky user hits it. > > Itai > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users