Xen users - Mar 2010 - 2.6.18.8 xen.org dom0 kernel w/RHEL 5 PV on HVM guests

I''ve been beating my head against the wall trying to get stub domains
working with PV on HVM guests running RHEL 5. Up to now I''ve been using
the RHEL 5 dom0 kernel, but that doesn''t work with stub domains because
gntdev isn''t built in (and can''t be built as a module either
-- I
tried). So I built the 2.6.18.8 dom0 kernel from xen-unstable.hg. That
got stub domains working with RHEL 5 HVM guests, but PV on HVM guests
fail to find the root filesystem when initrd gets ready to pivot to the
real root filesystem. PV on HVM without stub domain has the same problem.

--- begin domU console output ---
Creating block device nodes.
Loading xen-platform-pci.ko module
GSI 16 sharing vector 0xA9 and IRQ 16
ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 28 (level, low) -> IRQ 169
Xen version 3.4.
Hypercall area is 1 pages.
Grant table initialized
Loading xen-vbd.ko module
Loading xen-balloon.ko module
xen_mem: Initialising balloon driver.
Loading xen-vnif.ko module
netfront: Initialising virtual ethernet driver.
Loading jbd.ko module
netfront: device eth0 has copying receive path.
Loading ext3.ko module
Loading xen-vbd.ko module
insmod: error inserting ''/lib/xen-vbd.ko'': -1 File exists
Loading xen-vbd.ko module
insmod: error inserting ''/lib/xen-vbd.ko'': -1 File exists
Trying to resume from /dev/xvda2
Unable to access resume device (/dev/xvda2)
Creating root device.
Registering block device major 202
 xvda:Mounting root filesystem.
 xvda1 xvda2 xvda3
mount: could not find filesystem ''/dev/root''
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
--- end domU console output ---

Notice that the PV disks and their partitions are detected.

Kernel command line is root=LABEL=ROOT, but root=/dev/xvda3 doesn''t
work
either.

This same VM boots fine if I reboot the hypervisor with the RHEL 5 dom0
kernel.

/var/log/xen/* show nothing out of the ordinary. The only hint of a
problem are these lines from xm dmesg:

(XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
(XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
(XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22

If I boot this same VM without disabling the IDE drivers, it mounts
filesystems using /dev/hd* and starts up successfully. Now here''s the
baffling part: I can then mount / and /home using /dev/xvd*! So the
paravirt disk driver *is* working once the domU boots off /dev/hd*, but
for some reason it doesn''t work when I try to boot from /dev/xvd*
directly.

So my questions are:

1. Why can''t the initrd mount /dev/xvd* to / ?

2. Why does this work with the RHEL 5 dom0 kernel but not with a
similarly-configured xen-unstable 2.6.18.8 dom0 kernel?

xen-unstable 2.6.18.8 dom0 kernel config attached. I''ve gotten the
Xen-related stuff as close to RHEL 5 as I can, but available CONFIG_*
defines vary somewhat between the two kernel source trees.

Thanks for any help anyone can provide.

-Andrew



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Mar 18, 2010 at 12:46:11PM -0700, Andrew Evans
wrote:
> I''ve been beating my head against the wall trying to get stub
domains
> working with PV on HVM guests running RHEL 5. Up to now I''ve been
using
> the RHEL 5 dom0 kernel, but that doesn''t work with stub domains
because
> gntdev isn''t built in (and can''t be built as a module
either -- I
> tried). So I built the 2.6.18.8 dom0 kernel from xen-unstable.hg. That
> got stub domains working with RHEL 5 HVM guests, but PV on HVM guests
> fail to find the root filesystem when initrd gets ready to pivot to the
> real root filesystem. PV on HVM without stub domain has the same problem.
> 
> --- begin domU console output ---
> Creating block device nodes.
> Loading xen-platform-pci.ko module
> GSI 16 sharing vector 0xA9 and IRQ 16
> ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 28 (level, low) -> IRQ 169
> Xen version 3.4.
> Hypercall area is 1 pages.
> Grant table initialized
> Loading xen-vbd.ko module
> Loading xen-balloon.ko module
> xen_mem: Initialising balloon driver.
> Loading xen-vnif.ko module
> netfront: Initialising virtual ethernet driver.
> Loading jbd.ko module
> netfront: device eth0 has copying receive path.
> Loading ext3.ko module
> Loading xen-vbd.ko module
> insmod: error inserting ''/lib/xen-vbd.ko'': -1 File exists
> Loading xen-vbd.ko module
> insmod: error inserting ''/lib/xen-vbd.ko'': -1 File exists
> Trying to resume from /dev/xvda2
> Unable to access resume device (/dev/xvda2)

Unable to access... hmm.

> Creating root device.
> Registering block device major 202
>  xvda:Mounting root filesystem.
>  xvda1 xvda2 xvda3
> mount: could not find filesystem ''/dev/root''
I guess it''s the same problem here..

> Setting up other filesystems.
> Setting up new root fs
> setuproot: moving /dev failed: No such file or directory
> no fstab.sys, mounting internal defaults
> setuproot: error mounting /proc: No such file or directory
> setuproot: error mounting /sys: No such file or directory
> Switching to new root and running init.
> unmounting old /dev
> unmounting old /proc
> unmounting old /sys
> switchroot: mount failed: No such file or directory
> Kernel panic - not syncing: Attempted to kill init!
> --- end domU console output ---
> 
> Notice that the PV disks and their partitions are detected.
>
Yeah, but they don''t work for some reason..
 
> Kernel command line is root=LABEL=ROOT, but root=/dev/xvda3
doesn''t work
> either.
> 
> This same VM boots fine if I reboot the hypervisor with the RHEL 5 dom0
> kernel.
> 
> /var/log/xen/* show nothing out of the ordinary. The only hint of a
> problem are these lines from xm dmesg:
> 
> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
> 
These sound related..
> If I boot this same VM without disabling the IDE drivers, it mounts
> filesystems using /dev/hd* and starts up successfully. Now here''s
the
> baffling part: I can then mount / and /home using /dev/xvd*! So the
> paravirt disk driver *is* working once the domU boots off /dev/hd*, but
> for some reason it doesn''t work when I try to boot from /dev/xvd*
directly.
> 
> So my questions are:
> 
> 1. Why can''t the initrd mount /dev/xvd* to / ?
> 
Because the devices don''t work for some reason..
> 2. Why does this work with the RHEL 5 dom0 kernel but not with a
> similarly-configured xen-unstable 2.6.18.8 dom0 kernel?
> 
That''s a good question.
> xen-unstable 2.6.18.8 dom0 kernel config attached. I''ve gotten the
> Xen-related stuff as close to RHEL 5 as I can, but available CONFIG_*
> defines vary somewhat between the two kernel source trees.
> 
> Thanks for any help anyone can provide.
> 
So the dom0 kernel was the only thing you changed? 
You used the same Xen hypervisor for both dom0 kernels? which hypervisor
version?

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 03/18/10 14:38, Pasi Kärkkäinen wrote:
> On Thu, Mar 18, 2010 at 12:46:11PM -0700, Andrew Evans wrote:
>   
>> I''ve been beating my head against the wall trying to get stub
domains
>> working with PV on HVM guests running RHEL 5. Up to now I''ve
been using
>> the RHEL 5 dom0 kernel, but that doesn''t work with stub
domains because
>> gntdev isn''t built in (and can''t be built as a module
either -- I
>> tried). So I built the 2.6.18.8 dom0 kernel from xen-unstable.hg. That
>> got stub domains working with RHEL 5 HVM guests, but PV on HVM guests
>> fail to find the root filesystem when initrd gets ready to pivot to the
>> real root filesystem. PV on HVM without stub domain has the same
problem.
>>
>> --- begin domU console output ---
>> Creating block device nodes.
>> Loading xen-platform-pci.ko module
>> GSI 16 sharing vector 0xA9 and IRQ 16
>> ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 28 (level, low) -> IRQ
169
>> Xen version 3.4.
>> Hypercall area is 1 pages.
>> Grant table initialized
>> Loading xen-vbd.ko module
>> Loading xen-balloon.ko module
>> xen_mem: Initialising balloon driver.
>> Loading xen-vnif.ko module
>> netfront: Initialising virtual ethernet driver.
>> Loading jbd.ko module
>> netfront: device eth0 has copying receive path.
>> Loading ext3.ko module
>> Loading xen-vbd.ko module
>> insmod: error inserting ''/lib/xen-vbd.ko'': -1 File
exists
>> Loading xen-vbd.ko module
>> insmod: error inserting ''/lib/xen-vbd.ko'': -1 File
exists
>> Trying to resume from /dev/xvda2
>> Unable to access resume device (/dev/xvda2)
>>     
>
> Unable to access... hmm.
>
>
>   
>> Creating root device.
>> Registering block device major 202
>>  xvda:Mounting root filesystem.
>>  xvda1 xvda2 xvda3
>> mount: could not find filesystem ''/dev/root''
>>     
> I guess it''s the same problem here..
>
>
>   
>> Setting up other filesystems.
>> Setting up new root fs
>> setuproot: moving /dev failed: No such file or directory
>> no fstab.sys, mounting internal defaults
>> setuproot: error mounting /proc: No such file or directory
>> setuproot: error mounting /sys: No such file or directory
>> Switching to new root and running init.
>> unmounting old /dev
>> unmounting old /proc
>> unmounting old /sys
>> switchroot: mount failed: No such file or directory
>> Kernel panic - not syncing: Attempted to kill init!
>> --- end domU console output ---
>>
>> Notice that the PV disks and their partitions are detected.
>>
>>     
> Yeah, but they don''t work for some reason..
>   
...until the system finishes booting (if I remove ''hda=noprobe
ide0=noprobe'' from domU kernel command line, allowing initrd/nash to
mount disks via /dev/hd*)
>> Kernel command line is root=LABEL=ROOT, but root=/dev/xvda3
doesn''t work
>> either.
>>
>> This same VM boots fine if I reboot the hypervisor with the RHEL 5 dom0
>> kernel.
>>
>> /var/log/xen/* show nothing out of the ordinary. The only hint of a
>> problem are these lines from xm dmesg:
>>
>> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
>> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
>> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
>>
>>     
> These sound related..
>   
Yup. I can''t believe event channels are generally broken though, as the
PV network driver works and the PV disk driver works once the system
comes up.
>> If I boot this same VM without disabling the IDE drivers, it mounts
>> filesystems using /dev/hd* and starts up successfully. Now
here''s the
>> baffling part: I can then mount / and /home using /dev/xvd*! So the
>> paravirt disk driver *is* working once the domU boots off /dev/hd*, but
>> for some reason it doesn''t work when I try to boot from
/dev/xvd* directly.
>>
>> So my questions are:
>>
>> 1. Why can''t the initrd mount /dev/xvd* to / ?
>>
>>     
> Because the devices don''t work for some reason..
>   
Well, yeah, I had worked that much out on my own. :) I can''t imagine
what miracle is occurring between the time that initrd/nash tries to
mount /dev/xvda3 to / and when I do it manually from the command line
once I''m able to to log in, but such a miracle is in fact occurring!
>> 2. Why does this work with the RHEL 5 dom0 kernel but not with a
>> similarly-configured xen-unstable 2.6.18.8 dom0 kernel?
>>
>>     
> That''s a good question.
>
>   
>> xen-unstable 2.6.18.8 dom0 kernel config attached. I''ve gotten
the
>> Xen-related stuff as close to RHEL 5 as I can, but available CONFIG_*
>> defines vary somewhat between the two kernel source trees.
>>
>> Thanks for any help anyone can provide.
>>
>>     
> So the dom0 kernel was the only thing you changed?
>   
Yes.
> You used the same Xen hypervisor for both dom0 kernels? which hypervisor
version?
>   
Xen 3.4.2, rebuilt from gitco source RPM w/oprofile active domains patch
applied:

$ sudo xm info
host                   : <snip>
release                : 2.6.18.8-xen
version                : #7 SMP Wed Mar 17 18:36:46 UTC 2010
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2500
hw_caps                :
bfebfbff:20100800:00000000:00000140:000ce3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 8190
free_memory            : 7266
node_to_cpu            : node0:0-7
node_to_memory         : node0:7266
xen_major              : 3
xen_minor              : 4
xen_extra              : .2
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
cc_compile_by          : root
cc_compile_domain      : <snip>
cc_compile_date        : Wed Jan 27 00:36:30 UTC 2010
xend_config_format     : 4

thanks,

-Andrew


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Mar 18, 2010 at 03:20:45PM -0700, Andrew Evans
wrote:
> >>
> >> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error
-22
> >> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error
-22
> >> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error
-22
> >>
> >>     
> > These sound related..
> >   
> 
> Yup. I can''t believe event channels are generally broken though,
as the
> PV network driver works and the PV disk driver works once the system
> comes up.
>
Ok. I wonder if there''s some timeout related change in the RHEL5 dom0
kernel,
and the RHEL5 pv-on-hvm drivers depends on it? 
 
> >> If I boot this same VM without disabling the IDE drivers, it
mounts
> >> filesystems using /dev/hd* and starts up successfully. Now
here''s the
> >> baffling part: I can then mount / and /home using /dev/xvd*! So
the
> >> paravirt disk driver *is* working once the domU boots off
/dev/hd*, but
> >> for some reason it doesn''t work when I try to boot from
/dev/xvd* directly.
> >>
> >> So my questions are:
> >>
> >> 1. Why can''t the initrd mount /dev/xvd* to / ?
> >>
> >>     
> > Because the devices don''t work for some reason..
> >   
> 
> Well, yeah, I had worked that much out on my own. :) I can''t
imagine
> what miracle is occurring between the time that initrd/nash tries to
> mount /dev/xvda3 to / and when I do it manually from the command line
> once I''m able to to log in, but such a miracle is in fact
occurring!
> 
Maybe it''s just some timeout.. 
> >> 2. Why does this work with the RHEL 5 dom0 kernel but not with a
> >> similarly-configured xen-unstable 2.6.18.8 dom0 kernel?
> >>
> >>     
> > That''s a good question.
> >
> >   
> >> xen-unstable 2.6.18.8 dom0 kernel config attached. I''ve
gotten the
> >> Xen-related stuff as close to RHEL 5 as I can, but available
CONFIG_*
> >> defines vary somewhat between the two kernel source trees.
> >>
> >> Thanks for any help anyone can provide.
> >>
> >>     
> > So the dom0 kernel was the only thing you changed?
> >   
> 
> Yes.
> 
> > You used the same Xen hypervisor for both dom0 kernels? which
hypervisor version?
> >   
> 
> Xen 3.4.2, rebuilt from gitco source RPM w/oprofile active domains patch
> applied:
> 
Hmm.. can you try with the RHEL5 default hypervisor (3.1.2) ? 
(and the 2.6.18-xen dom0 kernel) 

Would be good to know if it makes any difference.

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 03/18/10 15:32, Pasi Kärkkäinen wrote:
> Hmm.. can you try with the RHEL5 default hypervisor (3.1.2) ?
> (and the 2.6.18-xen dom0 kernel) 
>
> Would be good to know if it makes any difference.
>   
I installed the xen-3.0.3 RPM (xm dmesg/xm info says 3.1.2 -- what the
heck is up with Red Hat''s version numbering?) but am getting the
dreaded
"Hotplug scripts not working" errors. Looks like a TUN/TAP problem:

$ cat qemu-dm.4910.log
domid: 2
qemu: the number of cpus is 7
Change xvda to look like hda
Change xvdb to look like hdb
Change xvdc to look like hdc
Watching /local/domain/2/logdirty/next-active
Watching /local/domain/0/device-model/2/command
/etc/xen/qemu-ifup: could not launch network script
Could not initialize device ''tap''

$ cat xend-debug.log.1
Xend started at Thu Mar 18 23:25:38 2010.
Nothing to flush.
Nothing to flush.
can''t add tap0 to bridge eth0: Operation not supported
can''t add tap0 to bridge eth0: Operation not supported

$ cat /etc/xen/qemu-ifup
#!/bin/sh

#. /etc/rc.d/init.d/functions
#ulimit -c unlimited

echo ''config qemu network with xen bridge for '' $*

ifconfig $1 0.0.0.0 up
brctl addif $2 $1

strace of `brctl addif eth0 tap0` shows:

ioctl(4, 0x89a2, 0x7fff724c1e50)        = -1 EOPNOTSUPP (Operation not
supported)
ioctl(4, SIOCDEVPRIVATE, 0x7fff724c1e50) = -1 EOPNOTSUPP (Operation not
supported)
write(2, "can''t add tap0 to bridge eth0: O"...,
55can''t add tap0 to
bridge eth0: Operation not supported
) = 55
exit_group(1)                           = ?

I think the RHEL 5 2.6.18 dom0 kernel must support this ioctl, while the
xen.org 2.6.18.8 dom0 kernel must not. So RHEL 5''s Xen 3.1.2 package on
xen.org 2.6.18.8 dom0 kernel looks like a dead end.

I''ll go back to Xen 3.4.2 with xen.org 2.6.18.8 dom0. PV disk on HVM is
actually not very important to me at the moment.

-Andrew


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Mar 22, 2010 at 11:13:43AM -0700, Andrew Evans
wrote:
> On 03/18/10 15:32, Pasi Kärkkäinen wrote:
> > Hmm.. can you try with the RHEL5 default hypervisor (3.1.2) ?
> > (and the 2.6.18-xen dom0 kernel) 
> >
> > Would be good to know if it makes any difference.
> >   
> 
> I installed the xen-3.0.3 RPM (xm dmesg/xm info says 3.1.2 -- what the
> heck is up with Red Hat''s version numbering?)
>
Xen hypervisor in RHEL5 comes in the kernel rpm together with the dom0 kernel.

RHEL5 currently has Xen 3.1.2 (+patches) hypervisor, but the userland
tools (xen rpm) are based on 3.0.3. 

That is because RHEL5 is an enterprise distro, and they need to keep
the ABI stable.. RHEL 5.0 first shipped with Xen 3.0.3 so they cannot
change versions and break things just like that. 

> but am getting the dreaded
> "Hotplug scripts not working" errors. Looks like a TUN/TAP
problem:
> 
> $ cat qemu-dm.4910.log
> domid: 2
> qemu: the number of cpus is 7
> Change xvda to look like hda
> Change xvdb to look like hdb
> Change xvdc to look like hdc
> Watching /local/domain/2/logdirty/next-active
> Watching /local/domain/0/device-model/2/command
> /etc/xen/qemu-ifup: could not launch network script
> Could not initialize device ''tap''
> 
> $ cat xend-debug.log.1
> Xend started at Thu Mar 18 23:25:38 2010.
> Nothing to flush.
> Nothing to flush.
> can''t add tap0 to bridge eth0: Operation not supported
> can''t add tap0 to bridge eth0: Operation not supported
> 
> $ cat /etc/xen/qemu-ifup
> #!/bin/sh
> 
> #. /etc/rc.d/init.d/functions
> #ulimit -c unlimited
> 
> echo ''config qemu network with xen bridge for '' $*
> 
> ifconfig $1 0.0.0.0 up
> brctl addif $2 $1
> 
> strace of `brctl addif eth0 tap0` shows:
> 
> ioctl(4, 0x89a2, 0x7fff724c1e50)        = -1 EOPNOTSUPP (Operation not
> supported)
> ioctl(4, SIOCDEVPRIVATE, 0x7fff724c1e50) = -1 EOPNOTSUPP (Operation not
> supported)
> write(2, "can''t add tap0 to bridge eth0: O"...,
55can''t add tap0 to
> bridge eth0: Operation not supported
> ) = 55
> exit_group(1)                           = ?
> 
> I think the RHEL 5 2.6.18 dom0 kernel must support this ioctl, while the
> xen.org 2.6.18.8 dom0 kernel must not. So RHEL 5''s Xen 3.1.2
package on
> xen.org 2.6.18.8 dom0 kernel looks like a dead end.
> 
> I''ll go back to Xen 3.4.2 with xen.org 2.6.18.8 dom0. PV disk on
HVM is
> actually not very important to me at the moment.
> 
Do you have a bridge called ''eth0'' for real? Try "brctl
show" to verify.
RHEL5 as a default creates ''xenbr0'', not
''eth0''.

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 03/22/10 11:22, Pasi Kärkkäinen wrote:
> On Mon, Mar 22, 2010 at 11:13:43AM -0700, Andrew Evans wrote:
>   
>> On 03/18/10 15:32, Pasi Kärkkäinen wrote:
>>     
>>> Hmm.. can you try with the RHEL5 default hypervisor (3.1.2) ?
>>> (and the 2.6.18-xen dom0 kernel) 
>>>
>>> Would be good to know if it makes any difference.
>>>   
>>>       
>> I installed the xen-3.0.3 RPM (xm dmesg/xm info says 3.1.2 -- what the
>> heck is up with Red Hat''s version numbering?)
>>
>>     
> Xen hypervisor in RHEL5 comes in the kernel rpm together with the dom0
kernel.
>
> RHEL5 currently has Xen 3.1.2 (+patches) hypervisor, but the userland
> tools (xen rpm) are based on 3.0.3. 
>
> That is because RHEL5 is an enterprise distro, and they need to keep
> the ABI stable.. RHEL 5.0 first shipped with Xen 3.0.3 so they cannot
> change versions and break things just like that.
>   
Ah, that makes sense. I realize RHEL makes ABI backward compatibility
guarantees, but I hadn''t guessed that was the issue here.
>> but am getting the dreaded
>> "Hotplug scripts not working" errors. Looks like a TUN/TAP
problem:
>>
>> $ cat qemu-dm.4910.log
>> domid: 2
>> qemu: the number of cpus is 7
>> Change xvda to look like hda
>> Change xvdb to look like hdb
>> Change xvdc to look like hdc
>> Watching /local/domain/2/logdirty/next-active
>> Watching /local/domain/0/device-model/2/command
>> /etc/xen/qemu-ifup: could not launch network script
>> Could not initialize device ''tap''
>>
>> $ cat xend-debug.log.1
>> Xend started at Thu Mar 18 23:25:38 2010.
>> Nothing to flush.
>> Nothing to flush.
>> can''t add tap0 to bridge eth0: Operation not supported
>> can''t add tap0 to bridge eth0: Operation not supported
>>
>> $ cat /etc/xen/qemu-ifup
>> #!/bin/sh
>>
>> #. /etc/rc.d/init.d/functions
>> #ulimit -c unlimited
>>
>> echo ''config qemu network with xen bridge for '' $*
>>
>> ifconfig $1 0.0.0.0 up
>> brctl addif $2 $1
>>
>> strace of `brctl addif eth0 tap0` shows:
>>
>> ioctl(4, 0x89a2, 0x7fff724c1e50)        = -1 EOPNOTSUPP (Operation not
>> supported)
>> ioctl(4, SIOCDEVPRIVATE, 0x7fff724c1e50) = -1 EOPNOTSUPP (Operation not
>> supported)
>> write(2, "can''t add tap0 to bridge eth0: O"...,
55can''t add tap0 to
>> bridge eth0: Operation not supported
>> ) = 55
>> exit_group(1)                           = ?
>>
>> I think the RHEL 5 2.6.18 dom0 kernel must support this ioctl, while
the
>> xen.org 2.6.18.8 dom0 kernel must not. So RHEL 5''s Xen 3.1.2
package on
>> xen.org 2.6.18.8 dom0 kernel looks like a dead end.
>>     
> Do you have a bridge called ''eth0'' for real? Try
"brctl show" to verify.
> RHEL5 as a default creates ''xenbr0'', not
''eth0''.
>   
Well spotted! I don''t have such a bridge, and I can''t create
one because
eth0 is already an existing network interface name. I''ve changed my
libvirt domU config to use xenbr0 instead of eth0. Not sure why this
works with Xen 3.4.2 though.

My domU *still* isn''t starting, though. xend-debug.log shows this
traceback:

Traceback (most recent call last):
  File "/usr/lib64/python2.4/SocketServer.py", line 463, in
process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib64/python2.4/SocketServer.py", line 254, in
finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python2.4/SocketServer.py", line 521, in __init__
    self.handle()
  File "/usr/lib64/python2.4/BaseHTTPServer.py", line 316, in handle
    self.handle_one_request()
  File "/usr/lib64/python2.4/BaseHTTPServer.py", line 310, in
handle_one_request
    method()
  File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py",
line
66, in do_POST
    self.send_response(200)
  File "/usr/lib64/python2.4/BaseHTTPServer.py", line 367, in
send_response
    self.wfile.write("%s %d %s\r\n" %
  File "/usr/lib64/python2.4/socket.py", line 256, in write
    self.flush()
  File "/usr/lib64/python2.4/socket.py", line 243, in flush
    self._sock.sendall(buffer)
error: (32, ''Broken pipe'')

No idea if it''s relevant or not. Certainly doesn''t look good
though.

xend.log contains everyone''s favorite, "Hotplug scripts not
working":

[2010-03-22 19:28:38 xend 4863] ERROR (SrvBase:88) Request
wait_for_devices failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/web/SrvBase.py", line
85,
in perform
    return op_method(op, req)
  File
"/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomain.py",
line
72, in op_wait_for_devices
    return self.dom.waitForDevices()
  File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 2330, in waitForDevices
    self.waitForDevices_(c)
  File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 1422, in waitForDevices_
    return self.getDeviceController(deviceClass).waitForDevices()
  File
"/usr/lib64/python2.4/site-packages/xen/xend/server/DevController.py",
line 160, in waitForDevices
    return map(self.waitForDevice, self.deviceIDs())
  File
"/usr/lib64/python2.4/site-packages/xen/xend/server/DevController.py",
line 170, in waitForDevice
    raise VmError("Device %s (%s) could not be connected. "
VmError: Device 0 (vif) could not be connected. Hotplug scripts not working.

Device model log:

$ cat qemu-dm.16544.log
domid: 4
qemu: the number of cpus is 7
Change xvda to look like hda
Change xvdb to look like hdb
Change xvdc to look like hdc
Watching /local/domain/4/logdirty/next-active
Watching /local/domain/0/device-model/4/command
xs_read(): vncpasswd get error.
/vm/51711672-9512-4bd8-91ea-a1d542bbda92/vncpasswd.
char device redirected to /dev/pts/1
qemu_map_cache_init nr_buckets = 10000
shared page at pfn effff
buffered io page at pfn efffd
xs_read(/vm/51711672-9512-4bd8-91ea-a1d542bbda92/rtc/timeoffset): read error
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
Triggered log-dirty buffer switch
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0

Current domU config (libvirt XML):

<domain type=''xen''>
  <name>foo</name>
  <memory>7340032</memory>
  <currentMemory>7340032</currentMemory>
  <vcpu>7</vcpu>
  <os>
    <type arch=''x86_64''
machine=''xenfv''>hvm</type>
    <loader>/usr/lib/xen/boot/hvmloader</loader>
    <boot dev=''hd''/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset=''utc''/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
    <disk type=''block'' device=''disk''>
      <driver name=''phy''/>
      <source dev=''/dev/sys/lv0''/>
      <target dev=''xvda'' bus=''xen''/>
    </disk>
    <disk type=''block'' device=''disk''>
      <driver name=''phy''/>
      <source dev=''/dev/sys/lv1''/>
      <target dev=''xvdb'' bus=''xen''/>
    </disk>
    <disk type=''block'' device=''disk''>
      <driver name=''phy''/>
      <source dev=''/dev/sys/lv2''/>
      <target dev=''xvdc'' bus=''xen''/>
    </disk>
    <interface type=''bridge''>
      <mac address=''00:50:56:0f:dd:02''/>
      <source bridge=''xenbr0''/>
      <script path=''vif-bridge''/>
    </interface>
    <serial type=''pty''>
      <target port=''0''/>
    </serial>
    <console type=''pty''>
      <target port=''0''/>
    </console>
    <input type=''mouse'' bus=''ps2''/>
    <graphics type=''vnc'' port=''-1''
autoport=''yes''/>
  </devices>
</domain>

thanks,

-Andrew


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Mar 22, 2010 at 12:41:14PM -0700, Andrew Evans
wrote:
> >>     
> > Do you have a bridge called ''eth0'' for real? Try
"brctl show" to verify.
> > RHEL5 as a default creates ''xenbr0'', not
''eth0''.
> >   
> 
> Well spotted! I don''t have such a bridge, and I can''t
create one because
> eth0 is already an existing network interface name. I''ve changed
my
> libvirt domU config to use xenbr0 instead of eth0. Not sure why this
> works with Xen 3.4.2 though.
>
Xen 3.4.x with the default configuration creates a bridge called
''eth0'',
and renames the physical network adapter to peth0.

Xen 3.0.x and 3.1.x with the default configuration creates a bridge called
''xenbr0'',
and renames the physical network adapter to peth0, and also creates a separate
dom0 management only virtual interface called ''eth0''.

So it''s just different defaults.
 
> My domU *still* isn''t starting, though. xend-debug.log shows this
traceback:
> 
Are you sure you got the earlier xen 3.4.x stuff completely removed? 

Did you also make sure virt-manager, libvirt, virsh etc were downgraded 
to the RHEL5 default versions?

Dunno if the xml configuration created with newer libvirt is still compatible 
with the older libvirt/xen..

-- Pasi
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/SocketServer.py", line 463, in
> process_request_thread
>     self.finish_request(request, client_address)
>   File "/usr/lib64/python2.4/SocketServer.py", line 254, in
finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib64/python2.4/SocketServer.py", line 521, in
__init__
>     self.handle()
>   File "/usr/lib64/python2.4/BaseHTTPServer.py", line 316, in
handle
>     self.handle_one_request()
>   File "/usr/lib64/python2.4/BaseHTTPServer.py", line 310, in
> handle_one_request
>     method()
>   File
"/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line
> 66, in do_POST
>     self.send_response(200)
>   File "/usr/lib64/python2.4/BaseHTTPServer.py", line 367, in
send_response
>     self.wfile.write("%s %d %s\r\n" %
>   File "/usr/lib64/python2.4/socket.py", line 256, in write
>     self.flush()
>   File "/usr/lib64/python2.4/socket.py", line 243, in flush
>     self._sock.sendall(buffer)
> error: (32, ''Broken pipe'')
> 
> No idea if it''s relevant or not. Certainly doesn''t look
good though.
> 
> xend.log contains everyone''s favorite, "Hotplug scripts not
working":
> 
> [2010-03-22 19:28:38 xend 4863] ERROR (SrvBase:88) Request
> wait_for_devices failed.
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/xen/web/SrvBase.py",
line 85,
> in perform
>     return op_method(op, req)
>   File
>
"/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomain.py",
line
> 72, in op_wait_for_devices
>     return self.dom.waitForDevices()
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
> line 2330, in waitForDevices
>     self.waitForDevices_(c)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
> line 1422, in waitForDevices_
>     return self.getDeviceController(deviceClass).waitForDevices()
>   File
>
"/usr/lib64/python2.4/site-packages/xen/xend/server/DevController.py",
> line 160, in waitForDevices
>     return map(self.waitForDevice, self.deviceIDs())
>   File
>
"/usr/lib64/python2.4/site-packages/xen/xend/server/DevController.py",
> line 170, in waitForDevice
>     raise VmError("Device %s (%s) could not be connected. "
> VmError: Device 0 (vif) could not be connected. Hotplug scripts not
working.
> 
> Device model log:
> 
> $ cat qemu-dm.16544.log
> domid: 4
> qemu: the number of cpus is 7
> Change xvda to look like hda
> Change xvdb to look like hdb
> Change xvdc to look like hdc
> Watching /local/domain/4/logdirty/next-active
> Watching /local/domain/0/device-model/4/command
> xs_read(): vncpasswd get error.
> /vm/51711672-9512-4bd8-91ea-a1d542bbda92/vncpasswd.
> char device redirected to /dev/pts/1
> qemu_map_cache_init nr_buckets = 10000
> shared page at pfn effff
> buffered io page at pfn efffd
> xs_read(/vm/51711672-9512-4bd8-91ea-a1d542bbda92/rtc/timeoffset): read
error
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> Triggered log-dirty buffer switch
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> 
> Current domU config (libvirt XML):
> 
> <domain type=''xen''>
>   <name>foo</name>
>   <memory>7340032</memory>
>   <currentMemory>7340032</currentMemory>
>   <vcpu>7</vcpu>
>   <os>
>     <type arch=''x86_64''
machine=''xenfv''>hvm</type>
>     <loader>/usr/lib/xen/boot/hvmloader</loader>
>     <boot dev=''hd''/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <pae/>
>   </features>
>   <clock offset=''utc''/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
>     <disk type=''block''
device=''disk''>
>       <driver name=''phy''/>
>       <source dev=''/dev/sys/lv0''/>
>       <target dev=''xvda''
bus=''xen''/>
>     </disk>
>     <disk type=''block''
device=''disk''>
>       <driver name=''phy''/>
>       <source dev=''/dev/sys/lv1''/>
>       <target dev=''xvdb''
bus=''xen''/>
>     </disk>
>     <disk type=''block''
device=''disk''>
>       <driver name=''phy''/>
>       <source dev=''/dev/sys/lv2''/>
>       <target dev=''xvdc''
bus=''xen''/>
>     </disk>
>     <interface type=''bridge''>
>       <mac address=''00:50:56:0f:dd:02''/>
>       <source bridge=''xenbr0''/>
>       <script path=''vif-bridge''/>
>     </interface>
>     <serial type=''pty''>
>       <target port=''0''/>
>     </serial>
>     <console type=''pty''>
>       <target port=''0''/>
>     </console>
>     <input type=''mouse''
bus=''ps2''/>
>     <graphics type=''vnc'' port=''-1''
autoport=''yes''/>
>   </devices>
> </domain>
> 
> thanks,
> 
> -Andrew
> 
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 03/22/10 12:54, Pasi Kärkkäinen wrote:
> Are you sure you got the earlier xen 3.4.x stuff completely removed?
> Did you also make sure virt-manager, libvirt, virsh etc were downgraded 
> to the RHEL5 default versions?
>   
You were right again. libvirt-0.7.2 built against Xen 3.3.1 was
preventing the RHEL 5 Xen 3.1.2 xend from working.
> Dunno if the xml configuration created with newer libvirt is still
compatible
> with the older libvirt/xen..
>   
That was also a problem. I had to add a <uuid> tag as libvirt
wouldn''t
generate one by default, and a <console> tag to get access to the serial
console again.

So...after all that, same problem: the domU fails to find the root
filesystem, just as with the gitco xen-3.4.2 RPMs. If I omit
''hda=noprobe'' from the kernel command line, the domU boots,
using
/dev/hd* via the ide-disk driver. I can then mount the virtual disks via
/dev/xvd*, so at that point xen-vbd is working.

I''m going to put this aside for now. I''ll hopefully be trying
a pv_ops
dom0 kernel soon. Hopefully I can get a RHEL 5 domU to boot from
/dev/xvd* with that setup.

Thanks for all your help.

-Andrew


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Mar 23, 2010 at 04:24:43PM -0700, Andrew Evans
wrote:
> On 03/22/10 12:54, Pasi Kärkkäinen wrote:
> > Are you sure you got the earlier xen 3.4.x stuff completely removed?
> > Did you also make sure virt-manager, libvirt, virsh etc were
downgraded
> > to the RHEL5 default versions?
> >   
> 
> You were right again. libvirt-0.7.2 built against Xen 3.3.1 was
> preventing the RHEL 5 Xen 3.1.2 xend from working.
> 
Ok.
> > Dunno if the xml configuration created with newer libvirt is still
compatible
> > with the older libvirt/xen..
> >   
> 
> That was also a problem. I had to add a <uuid> tag as libvirt
wouldn''t
> generate one by default, and a <console> tag to get access to the
serial
> console again.
> 
> So...after all that, same problem: the domU fails to find the root
> filesystem, just as with the gitco xen-3.4.2 RPMs. If I omit
> ''hda=noprobe'' from the kernel command line, the domU
boots, using
> /dev/hd* via the ide-disk driver. I can then mount the virtual disks via
> /dev/xvd*, so at that point xen-vbd is working.
> 
When it also happens with the vanilla EL5 stuff, it might be worth opening a bug
in redhat''s bugzilla, and put all the details there.

-- Pasi
> I''m going to put this aside for now. I''ll hopefully be
trying a pv_ops
> dom0 kernel soon. Hopefully I can get a RHEL 5 domU to boot from
> /dev/xvd* with that setup.
> 
> Thanks for all your help.
> 
> -Andrew
> 
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users