Xen users - Jun 2007 - Cannot start domains after FC6->F7 upgrade

I just upgrade a server from FC6->F7 using ''yum update''. 
Everything
went perfectly.  But now I cannot start any Xen domain.
xm create /etc/xen/xen0 -c
I am getting the error:
Error: Unable to connect to xend: No such file or directory.  Is xend 
running?

And of course I check xend with:
service xend status and it says [OK].  So then I restart xend but still 
I cannot start domains.

How do I get my domains to start?


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Gerry Reno wrote:
> I just upgrade a server from FC6->F7 using ''yum
update''.  Everything
> went perfectly.  But now I cannot start any Xen domain.
> xm create /etc/xen/xen0 -c
> I am getting the error:
> Error: Unable to connect to xend: No such file or directory.  Is xend 
> running?
>
> And of course I check xend with:
> service xend status and it says [OK].  So then I restart xend but 
> still I cannot start domains.
>
> How do I get my domains to start?
>
Did you reboot with the new Xen kernel? Or did you pull your kernel or 
assemble it form elsewhere?

My local mirror does not have FC7 yet, so I haven''t been able to play 
with it.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Gerry Reno wrote:
>> I just upgrade a server from FC6->F7 using ''yum
update''.  Everything
>> went perfectly.  But now I cannot start any Xen domain.
>> xm create /etc/xen/xen0 -c
>> I am getting the error:
>> Error: Unable to connect to xend: No such file or directory.  Is xend 
>> running?
>>
>> And of course I check xend with:
>> service xend status and it says [OK].  So then I restart xend but 
>> still I cannot start domains.
>>
>> How do I get my domains to start?
>>
> Did you reboot with the new Xen kernel? Or did you pull your kernel or 
> assemble it form elsewhere?
>
Yes.  Standard xen kernel.  But had to use the FC6 xen kernel because 
the F7 xen kernel crashes on our hardware.
> My local mirror does not have FC7 yet, so I haven''t been able to
play
> with it.
>
I''ve opened a Fedora bug:  
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=246169




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Friday 29 June 2007, Nico Kadel-Garcia wrote:
> Gerry Reno wrote:
> > Nico Kadel-Garcia wrote:
> >> Gerry Reno wrote:
> >>> I just upgrade a server from FC6->F7 using ''yum
update''.  Everything
> >>> went perfectly.  But now I cannot start any Xen domain.
> >>> xm create /etc/xen/xen0 -c
> >>> I am getting the error:
> >>> Error: Unable to connect to xend: No such file or directory. 
Is
> >>> xend running?
> >>>
> >>> And of course I check xend with:
> >>> service xend status and it says [OK].  So then I restart xend
but
> >>> still I cannot start domains.
> >>>
> >>> How do I get my domains to start?
> >>
> >> Did you reboot with the new Xen kernel? Or did you pull your
kernel
> >> or assemble it form elsewhere?
> >
> > Yes.  Standard xen kernel.  But had to use the FC6 xen kernel because
> > the F7 xen kernel crashes on our hardware.
>
> Huh. Running a more recent Xen release with an older kernel on Fom0 is
> kind of begging for incompatibility, I''d suspect.
>
> What is in /var/log/xen/*? And what does "yum list extras" say,
in case
> you pulled non-FC6 or non-FC7 packages while working with things? And
> have you tried building any new images and looking for differences in
> the config files? FC6 was xen 3.0.3, FC7 is Xen 3.1.0.
It''s breaking because of the version mismatch you suspected.

Looks like this problem is being mostly discussed at the Fedora bug, btw.
> > I''ve opened a Fedora bug:
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=246169
Cheers
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Gerry Reno wrote:
> Nico Kadel-Garcia wrote:
>> Gerry Reno wrote:
>>> I just upgrade a server from FC6->F7 using ''yum
update''.  Everything
>>> went perfectly.  But now I cannot start any Xen domain.
>>> xm create /etc/xen/xen0 -c
>>> I am getting the error:
>>> Error: Unable to connect to xend: No such file or directory.  Is 
>>> xend running?
>>>
>>> And of course I check xend with:
>>> service xend status and it says [OK].  So then I restart xend but 
>>> still I cannot start domains.
>>>
>>> How do I get my domains to start?
>>>
>> Did you reboot with the new Xen kernel? Or did you pull your kernel 
>> or assemble it form elsewhere?
>>
> Yes.  Standard xen kernel.  But had to use the FC6 xen kernel because 
> the F7 xen kernel crashes on our hardware.
Huh. Running a more recent Xen release with an older kernel on Fom0 is 
kind of begging for incompatibility, I''d suspect.

What is in /var/log/xen/*? And what does "yum list extras" say, in
case
you pulled non-FC6 or non-FC7 packages while working with things? And 
have you tried building any new images and looking for differences in 
the config files? FC6 was xen 3.0.3, FC7 is Xen 3.1.0.
>> My local mirror does not have FC7 yet, so I haven''t been able
to play
>> with it.
>>
> I''ve opened a Fedora bug:  
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=246169
>
>
>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Did you reboot with the new Xen kernel? Or did you pull your kernel or
> > assemble it form elsewhere?
>
> Yes.  Standard xen kernel.  But had to use the FC6 xen kernel because
> the F7 xen kernel crashes on our hardware.
>
> > My local mirror does not have FC7 yet, so I haven''t been able
to play
> > with it.
>
> I''ve opened a Fedora bug:
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=246169
Chiming in here; I don''t have a RH bugzilla account.

The matching rules for kernel / tools / xen versions are a bit complex.  The 
userspace tools and Xen itself must *always* be matched.

Since around 3.0.4 (I think?) it''s been possible to mix and match dom0
kernel
versions.  i.e. any dom0 kernel since 3.0.4 should run on any Xen from 3.0.4.  
Before that release the dom0 and Xen interfaces were tied together and you 
couldn''t really mix and match /at all/.  Although you couldn''t
strictly mix
and match, you could sometimes get dom0 to boot on Xen, but things
didn''t
work properly - this may be what you''re seeing, I guess.

Could you rebuild your kernel to not use libata?

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Huh. Running a more recent Xen release with an older kernel on Fom0 is 
> kind of begging for incompatibility, I''d suspect.
>
> What is in /var/log/xen/*? And what does "yum list extras" say,
in
> case you pulled non-FC6 or non-FC7 packages while working with things? 
> And have you tried building any new images and looking for differences 
> in the config files? FC6 was xen 3.0.3, FC7 is Xen 3.1.0.
>
>
Nico,
  I''m steering away from trying to get cross version working - too many
dependencies.  What has to happen in our case is that Fedora has to fix 
the libata problems so that we could use an F7 kernel.  I may try just 
getting VMWare server working in the interim until libata gets fixed and 
we can go back to using Xen.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Mark Williamson wrote:
>>> Did you reboot with the new Xen kernel? Or did you pull your kernel
or
>>> assemble it form elsewhere?
>>>       
>> Yes.  Standard xen kernel.  But had to use the FC6 xen kernel because
>> the F7 xen kernel crashes on our hardware.
>>
>>     
>>> My local mirror does not have FC7 yet, so I haven''t been
able to play
>>> with it.
>>>       
>> I''ve opened a Fedora bug:
>> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=246169
>>     
>
> Chiming in here; I don''t have a RH bugzilla account.
>
> The matching rules for kernel / tools / xen versions are a bit complex. 
The
> userspace tools and Xen itself must *always* be matched.
>
> Since around 3.0.4 (I think?) it''s been possible to mix and match
dom0 kernel
> versions.  i.e. any dom0 kernel since 3.0.4 should run on any Xen from
3.0.4.
> Before that release the dom0 and Xen interfaces were tied together and you 
> couldn''t really mix and match /at all/.  Although you
couldn''t strictly mix
> and match, you could sometimes get dom0 to boot on Xen, but things
didn''t
> work properly - this may be what you''re seeing, I guess.
>
> Could you rebuild your kernel to not use libata?
>   
Hmm. Doing a "yum update" OS update is prone to interesting issues as 
kernel versions switch: I''ve had SATA controllers switch the name of
the
necessary driver, and even had those ghods-awful Promise controllers 
switch from calling drives hda to calling them sda. (THAT caused serious 
swearing, because upgrading kernels meant re-writing /etc/fstab, and so 
did rolling the kernel back.

Gerry, can you do a CD or DVD based update operation and rebuild your 
grub.conf and corresponding modprobe.conf settings?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Gerry Reno wrote:
> Nico Kadel-Garcia wrote:
>> Huh. Running a more recent Xen release with an older kernel on Fom0 
>> is kind of begging for incompatibility, I''d suspect.
>>
>> What is in /var/log/xen/*? And what does "yum list extras"
say, in
>> case you pulled non-FC6 or non-FC7 packages while working with 
>> things? And have you tried building any new images and looking for 
>> differences in the config files? FC6 was xen 3.0.3, FC7 is Xen 3.1.0.
>>
>>
>
> Nico,
>  I''m steering away from trying to get cross version working - too
many
> dependencies.  What has to happen in our case is that Fedora has to 
> fix the libata problems so that we could use an F7 kernel.  I may try 
> just getting VMWare server working in the interim until libata gets 
> fixed and we can go back to using Xen.
By doing a yum upgrade rather than a clean install, or booting with an 
FC7 medium, you''ve skipped out on the kernel compatibility testing that
goes with that step. I sympathize: I used to get paid good money to 
resolve exactly that sort of update and compatibility fun. So what you 
have is not really a Xen problem: it''s an "FC7 kernel-xen
won''t boot my
hardware" problem, right? Will the non-Xen FC7 kernel boot your hardware?

Can you boot this hardware with an FC7 installation CD or DVD?



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Jun 29, 2007 at 01:30:01AM +0100, Mark Williamson
wrote:
> > > Did you reboot with the new Xen kernel? Or did you pull your
kernel or
> > > assemble it form elsewhere?
> >
> > Yes.  Standard xen kernel.  But had to use the FC6 xen kernel because
> > the F7 xen kernel crashes on our hardware.
> >
> > > My local mirror does not have FC7 yet, so I haven''t been
able to play
> > > with it.
> >
> > I''ve opened a Fedora bug:
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=246169
> 
> Chiming in here; I don''t have a RH bugzilla account.
> 
> The matching rules for kernel / tools / xen versions are a bit complex. 
The
> userspace tools and Xen itself must *always* be matched.
> 
> Since around 3.0.4 (I think?) it''s been possible to mix and match
dom0 kernel
> versions.  i.e. any dom0 kernel since 3.0.4 should run on any Xen from
3.0.4.
> Before that release the dom0 and Xen interfaces were tied together and you 
> couldn''t really mix and match /at all/.  Although you
couldn''t strictly mix
> and match, you could sometimes get dom0 to boot on Xen, but things
didn''t
> work properly - this may be what you''re seeing, I guess.
Since we distribute the hypervisor as part of kernel-xen, this is kinda
academic. For Fedora we guarentee that we will never update the Xen 
hypervisor version within the course of a major release, so as a general
rule users don''t have to worry about incompatabilities between
''xen'' and
''kernel-xen'' if applying updates to the distro.  The problem
in this case
was that there was a mix of packages from two different distros - F7 and 
FC6 between xen & kernel-xen RPMs which is not guarenteed to work because
of lack of a stable ABI in Xen. 

This whole issue could be avoided if just a tiny bit more care were taken
in Xen hypervisor development to maintain a back-compat ABI.  You don''t
expect your GLibC to break when you update to a new kernel - but that''s
exactly the situation we''re in with Xen where updating a HV breaks your
userspace :-(

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel P. Berrange wrote:
> On Fri, Jun 29, 2007 at 01:30:01AM +0100, Mark Williamson wrote:
>   
>>>> Did you reboot with the new Xen kernel? Or did you pull your
kernel or
>>>> assemble it form elsewhere?
>>>>         
>>> Yes.  Standard xen kernel.  But had to use the FC6 xen kernel
because
>>> the F7 xen kernel crashes on our hardware.
>>>
>>>       
>>>> My local mirror does not have FC7 yet, so I haven''t
been able to play
>>>> with it.
>>>>         
>>> I''ve opened a Fedora bug:
>>> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=246169
>>>       
>> Chiming in here; I don''t have a RH bugzilla account.
>>
>> The matching rules for kernel / tools / xen versions are a bit complex.
The
>> userspace tools and Xen itself must *always* be matched.
>>
>> Since around 3.0.4 (I think?) it''s been possible to mix and
match dom0 kernel
>> versions.  i.e. any dom0 kernel since 3.0.4 should run on any Xen from
3.0.4.
>> Before that release the dom0 and Xen interfaces were tied together and
you
>> couldn''t really mix and match /at all/.  Although you
couldn''t strictly mix
>> and match, you could sometimes get dom0 to boot on Xen, but things
didn''t
>> work properly - this may be what you''re seeing, I guess.
>>     
>
> Since we distribute the hypervisor as part of kernel-xen, this is kinda
> academic. For Fedora we guarentee that we will never update the Xen 
> hypervisor version within the course of a major release, so as a general
> rule users don''t have to worry about incompatabilities between
''xen'' and
> ''kernel-xen'' if applying updates to the distro.  The
problem in this case
> was that there was a mix of packages from two different distros - F7 and 
> FC6 between xen & kernel-xen RPMs which is not guarenteed to work
because
> of lack of a stable ABI in Xen. 
>
> This whole issue could be avoided if just a tiny bit more care were taken
> in Xen hypervisor development to maintain a back-compat ABI.  You
don''t
> expect your GLibC to break when you update to a new kernel - but
that''s
> exactly the situation we''re in with Xen where updating a HV breaks
your
> userspace :-(
>
> Dan.
>   
Yes, the situation for us is that I upgraded our servers by d/l the 
fedora-release* rpms, installing them and then doing a ''yum -y
upgrade''.
And guess what? Worked great. Except I was not aware that the new libata 
drivers did not properly support our old highpoint ATA controllers. So 
at first boot, instant crash. So play around with this for a while and 
then start opening bugs on the F7 kernels. I had had some problems 
getting the 2.6.20 series kernels booting on the highpoint controllers 
but that was with the old IDE drivers and some parameters in the kernel 
had changed and changed the tolerance w/regard to our bus timing. So we 
were getting ''unknown bus timing'' error there. I worked with
Sergei and
Chuck and we were able to solve that issue on FC6 with a BIOS tweak on 
our hardware. Actually overclocking the bus did the trick. But with F7 
(which I was expecting to just fix all of this), things got even worse. 
The new libata drivers failed badly w/regard to the highpoint 
controllers. So the only option was to boot F7 using the old FC6 
kernels. This worked, or so I thought, until I began restoring the 
system to operational status by bringing up all the Xen guests. Kaboom! 
No way could I get any of the domains to start. So now we are really 
stuck. I''m assuming that Alan and Sergei are probably trying to get 
libata fixed but Alan didn''t seem too optimistic that this would happen
soon. I really do not want to try a bare metal restore of the server 
back to FC6. So now I''m trying to figure out what other options might
be
in the picture. I''m thinking along the lines of maybe seeing if VMWare 
could run the xen images. I already tried some things using qemu, but 
the networking is way too slow. Like 5x to 10x slower than Xen. So maybe 
I need to go the other way and look at openvz and just toss some 
processes into separate ve''s and do that until things get straightened 
out with libata.

Gerry



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Gerry Reno wrote:
> Yes, the situation for us is that I upgraded our servers by d/l the 
> fedora-release* rpms, installing them and then doing a ''yum -y 
> upgrade''.  And guess what?  Worked great.  Except I was not aware
that
> the new libata drivers did not properly support our old highpoint ATA 
> controllers.  So at first boot, instant crash.  So play around with 
> this for a while and then start opening bugs on the F7 kernels.  I had 
> had some problems getting the 2.6.20 series kernels booting on the 
> highpoint controllers but that was with the old IDE drivers and some 
> parameters in the kernel had changed and changed the tolerance 
> w/regard to our bus timing.  So we were getting ''unknown bus
timing''
> error there.  I worked with Sergei and Chuck and we were able to solve 
> that issue on FC6 with a BIOS tweak on our hardware.  Actually 
> overclocking the bus did the trick.  But with F7 (which I was 
> expecting to just fix all of this), things got even worse.  The new 
> libata drivers failed badly w/regard to the highpoint controllers.  So 
> the only option was to boot F7 using the old FC6 kernels.  This 
> worked, or so I thought, until I began restoring the system to 
> operational status by bringing up all the Xen guests.  Kaboom!  No way 
> could I get any of the domains to start.  So now we are really stuck.  
> I''m assuming that Alan and Sergei are probably trying to get
libata
> fixed but Alan didn''t seem too optimistic that this would happen 
> soon.  I really do not want to try a bare metal restore of the server 
> back to FC6.  So now I''m trying to figure out what other options
might
> be in the picture.  I''m thinking along the lines of maybe seeing
if
> VMWare could run the xen images.  I already tried some things using 
> qemu, but the networking is way too slow.  Like 5x to 10x slower than 
> Xen.  So maybe I need to go the other way and look at openvz and just 
> toss some processes into separate ve''s and do that until things
get
> straightened out with libata.
Ouch. Dude, you''ve just spent at least a whole work day of your
valuable
time with this, right? Is your time worth switching to a more robustly 
supported ATA card, like a 3Ware or Adaptec? I know the Highpoint''s are
very inexpensive, but their "Linux support" really hasn''t
been reliable
enough to use them for boot drives.

I sympathize, I really do: I''ve had similar issues happen with a 
thousand servers where the "kernel team" had so customized their
kernel
that we couldn''t reliably update to a new OS and new kernel, and I had 
to babysit it into operating with new hardware. (Hint: any kernel 
developer who can''t give you a diff between the distributed kernel they
started with and their new kernel, and says "we can backport anything we 
need from the new kernels!", should be fired immediately.)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Gerry Reno wrote:
>> Yes, the situation for us is that I upgraded our servers by d/l the 
>> fedora-release* rpms, installing them and then doing a ''yum -y
>> upgrade''.  And guess what?  Worked great.  Except I was not
aware
>> that the new libata drivers did not properly support our old 
>> highpoint ATA controllers.  So at first boot, instant crash.  So play 
>> around with this for a while and then start opening bugs on the F7 
>> kernels.  I had had some problems getting the 2.6.20 series kernels 
>> booting on the highpoint controllers but that was with the old IDE 
>> drivers and some parameters in the kernel had changed and changed the 
>> tolerance w/regard to our bus timing.  So we were getting
''unknown
>> bus timing'' error there.  I worked with Sergei and Chuck and
we were
>> able to solve that issue on FC6 with a BIOS tweak on our hardware.  
>> Actually overclocking the bus did the trick.  But with F7 (which I 
>> was expecting to just fix all of this), things got even worse.  The 
>> new libata drivers failed badly w/regard to the highpoint 
>> controllers.  So the only option was to boot F7 using the old FC6 
>> kernels.  This worked, or so I thought, until I began restoring the 
>> system to operational status by bringing up all the Xen guests.  
>> Kaboom!  No way could I get any of the domains to start.  So now we 
>> are really stuck.  I''m assuming that Alan and Sergei are
probably
>> trying to get libata fixed but Alan didn''t seem too optimistic
that
>> this would happen soon.  I really do not want to try a bare metal 
>> restore of the server back to FC6.  So now I''m trying to
figure out
>> what other options might be in the picture.  I''m thinking
along the
>> lines of maybe seeing if VMWare could run the xen images.  I already 
>> tried some things using qemu, but the networking is way too slow.  
>> Like 5x to 10x slower than Xen.  So maybe I need to go the other way 
>> and look at openvz and just toss some processes into separate
ve''s
>> and do that until things get straightened out with libata.
> Ouch. Dude, you''ve just spent at least a whole work day of your 
> valuable time with this, right? 
Nico, I wish it was only "a whole work day".  It''s more like
several days.
> Is your time worth switching to a more robustly supported ATA card, 
> like a 3Ware or Adaptec? I know the Highpoint''s are very
inexpensive,
> but their "Linux support" really hasn''t been reliable
enough to use
> them for boot drives.
That''s not true.  We have been using the highpoint controllers for at 
least five years onboard on various M/B''s as well as pci cards and they
have been very stable.  At least until Fedora''s latest kernel changes 
starting with the 2.6.20 series.  I''ve had dozens of kernels running 
over the years on these affected servers without any problems until lately.
>
> I sympathize, I really do: I''ve had similar issues happen with a 
> thousand servers where the "kernel team" had so customized their 
> kernel that we couldn''t reliably update to a new OS and new
kernel,
> and I had to babysit it into operating with new hardware. (Hint: any 
> kernel developer who can''t give you a diff between the distributed
> kernel they started with and their new kernel, and says "we can 
> backport anything we need from the new kernels!", should be fired 
> immediately.)
For us, Fedora has created the ''perfect storm''.  We read
release notes
but didn''t see any mention of lack of support for highpoint so we 
prepared all our filesystems with LABELS and then did the upgrade.  Once 
we couldn''t boot the F7 kernels and had to drop back to F6 kernel then 
we run into the problem of the ABI incompatible changes between Xen 
3.0.x and 3.1.x.  Both of these problems at the same time just killed 
us.  As far as libata I think the kernel team should have left the old 
IDE drivers in the kernel along with libata and provided a command line 
switch that would let the user switch back to the old drivers if they 
had any major problems with the new drivers.  That would have provided 
many of the non-working libata cases a temporary workaround until the 
kernel team could solve these issues.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Gerry Reno wrote:
> Nico Kadel-Garcia wrote:
>> Is your time worth switching to a more robustly supported ATA card, 
>> like a 3Ware or Adaptec? I know the Highpoint''s are very
inexpensive,
>> but their "Linux support" really hasn''t been
reliable enough to use
>> them for boot drives.
> That''s not true.  We have been using the highpoint controllers for
at
> least five years onboard on various M/B''s as well as pci cards and
> they have been very stable.  At least until Fedora''s latest kernel
> changes starting with the 2.6.20 series.  I''ve had dozens of
kernels
> running over the years on these affected servers without any problems 
> until lately.
Have you been through this sort of "install drivers by hand" with each
OS release? That can get very expensive in manpower and downtime and 
frustration, very fast.
>>
>> I sympathize, I really do: I''ve had similar issues happen with
a
>> thousand servers where the "kernel team" had so customized
their
>> kernel that we couldn''t reliably update to a new OS and new
kernel,
>> and I had to babysit it into operating with new hardware. (Hint: any 
>> kernel developer who can''t give you a diff between the
distributed
>> kernel they started with and their new kernel, and says "we can 
>> backport anything we need from the new kernels!", should be fired 
>> immediately.)
>
> For us, Fedora has created the ''perfect storm''.  We read
release notes
> but didn''t see any mention of lack of support for highpoint so we 
> prepared all our filesystems with LABELS and then did the upgrade.  
> Once we couldn''t boot the F7 kernels and had to drop back to F6
kernel
> then we run into the problem of the ABI incompatible changes between 
> Xen 3.0.x and 3.1.x.  Both of these problems at the same time just 
> killed us.  As far as libata I think the kernel team should have left 
> the old IDE drivers in the kernel along with libata and provided a 
> command line switch that would let the user switch back to the old 
> drivers if they had any major problems with the new drivers.  That 
> would have provided many of the non-working libata cases a temporary 
> workaround until the kernel team could solve these issues.
Well, that gets into fascinating kernel support issues. I admit I''ve
not
poked around the FC7 kernels, but supporting multiple drivers for the 
same hardware is awkward and tough to test: I can completely understand 
eventually dropping old drivers. Are the old drivers in the latest 
kernel source? You could build a tweaked SRPM and RPM with the old 
drivers activated instead.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Gerry Reno wrote:
>> That''s not true.  We have been using the highpoint controllers
for at
>> least five years onboard on various M/B''s as well as pci cards
and
>> they have been very stable.  At least until Fedora''s latest
kernel
>> changes starting with the 2.6.20 series.  I''ve had dozens of
kernels
>> running over the years on these affected servers without any problems 
>> until lately.
> Have you been through this sort of "install drivers by hand" with
each
> OS release? That can get very expensive in manpower and downtime and 
> frustration, very fast.
Not sure why you are saying "install drivers by hand".  We do none of 
this.  Support for highpoint controllers has been in the kernels for a 
long time.  Only back around 2000, 2001 did we ever have to manually 
load any drivers for highpoint.
>
>> For us, Fedora has created the ''perfect storm''.  We
read release
>> notes but didn''t see any mention of lack of support for
highpoint so
>> we prepared all our filesystems with LABELS and then did the 
>> upgrade.  Once we couldn''t boot the F7 kernels and had to drop
back
>> to F6 kernel then we run into the problem of the ABI incompatible 
>> changes between Xen 3.0.x and 3.1.x.  Both of these problems at the 
>> same time just killed us.  As far as libata I think the kernel team 
>> should have left the old IDE drivers in the kernel along with libata 
>> and provided a command line switch that would let the user switch 
>> back to the old drivers if they had any major problems with the new 
>> drivers.  That would have provided many of the non-working libata 
>> cases a temporary workaround until the kernel team could solve these 
>> issues.
> Well, that gets into fascinating kernel support issues. I admit
I''ve
> not poked around the FC7 kernels, but supporting multiple drivers for 
> the same hardware is awkward and tough to test: I can completely 
> understand eventually dropping old drivers. Are the old drivers in the 
> latest kernel source? You could build a tweaked SRPM and RPM with the 
> old drivers activated instead.
>
No I''m not a kernel builder and I really don''t want to start
that now.
I want to use a distro-supported kernel.  So for now Xen is out.  I''m 
looking at either qemu or vmware server to bail us out of this situation.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Gerry Reno wrote:
> Nico Kadel-Garcia wrote:
>> Gerry Reno wrote:
>>> That''s not true.  We have been using the highpoint
controllers for
>>> at least five years onboard on various M/B''s as well as
pci cards
>>> and they have been very stable.  At least until Fedora''s
latest
>>> kernel changes starting with the 2.6.20 series.  I''ve had
dozens of
>>> kernels running over the years on these affected servers without
any
>>> problems until lately.
>> Have you been through this sort of "install drivers by hand"
with
>> each OS release? That can get very expensive in manpower and downtime 
>> and frustration, very fast.
>
> Not sure why you are saying "install drivers by hand".  We do
none of
> this.  Support for highpoint controllers has been in the kernels for a 
> long time.  Only back around 2000, 2001 did we ever have to manually 
> load any drivers for highpoint.
Ahh. *Really*. Hmm. My last direct work with Highpoint cards was.... 2 
years ago, with PCI based PATA cards to supplement a Proliant system 
whose on-board controllers mis-handled any drivers over 128 Gig. I found 
it painful to deal with, and returned the card for an Adaptec card that 
behaved beautifully.

>>> For us, Fedora has created the ''perfect storm''. 
We read release
>>> notes but didn''t see any mention of lack of support for
highpoint so
>>> we prepared all our filesystems with LABELS and then did the 
>>> upgrade.  Once we couldn''t boot the F7 kernels and had to
drop back
>>> to F6 kernel then we run into the problem of the ABI incompatible 
>>> changes between Xen 3.0.x and 3.1.x.  Both of these problems at the
>>> same time just killed us.  As far as libata I think the kernel team
>>> should have left the old IDE drivers in the kernel along with
libata
>>> and provided a command line switch that would let the user switch 
>>> back to the old drivers if they had any major problems with the new
>>> drivers.  That would have provided many of the non-working libata 
>>> cases a temporary workaround until the kernel team could solve
these
>>> issues.
>> Well, that gets into fascinating kernel support issues. I admit
I''ve
>> not poked around the FC7 kernels, but supporting multiple drivers for 
>> the same hardware is awkward and tough to test: I can completely 
>> understand eventually dropping old drivers. Are the old drivers in 
>> the latest kernel source? You could build a tweaked SRPM and RPM with 
>> the old drivers activated instead.
>>
>
> No I''m not a kernel builder and I really don''t want to
start that
> now.  I want to use a distro-supported kernel.  So for now Xen is 
> out.  I''m looking at either qemu or vmware server to bail us out
of
> this situation.
Fair enough. If I may ask, why are you using FC7? That''s leading edge
if
not bleeding edge. If you need stable servers, why not use RHEL (with 
direct RedHat support), or CentOS, or stay a version behind with your 
Fedora Core to let some other poor beggar hammer out the hard bits (such 
as those you just encountered)?

I mean, does even the FC7 kernel-smp kernel work on your hardware?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Fair enough. If I may ask, why are you using FC7? That''s leading
edge
> if not bleeding edge. If you need stable servers, why not use RHEL 
> (with direct RedHat support), or CentOS, or stay a version behind with 
> your Fedora Core to let some other poor beggar hammer out the hard 
> bits (such as those you just encountered)?
Again, with every new release we go through a small amount of pain 
(operative word: small) with Fedora.  But then we also have a great deal 
of annoying bugs fixed and support for all new hardware.  So there is a 
tradeoff.  F7 really surprised us with its serious problems.  That 
release was not ready for public consumption.  Many, many problems.  A 
blizzard of bugs have been opened so I think Fedora will probably do a 
little more testing next time and pay more attention to installation, 
libata, xen ABI, and kernel issues.  It was probably a good lesson. 
I''m
sure that once people heard about the problems that there was a big drop 
in downloads compared to FC6.  We''ll see what happens with F8 in
Oct/Nov.
>
> I mean, does even the FC7 kernel-smp kernel work on your hardware?
Well, FC{1,2,3,4,5,6} have all worked just fine on this hardware.  So 
once they fix libata I''m sure everything else will be fine.  libata 
seems to be the major problem.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users