Xen devel - Dec 2011 - Xen 4.x / Linux 3.x (dom0 and HVM domU) and NIC handling

Moving to public discussion...

This was found with Xen hypervisor version supporting device unplugging and the
domU kernel having net-/blkfront and pci platform built-in (or as module).

The block device is defined as hda and the NIC type=ioemu (so theoretically
guests without pv support would work, too).

Since both drivers are present, the kernel tries to unplug the emulated devices
and succeeds. The blkfront driver detects the xvda device available in parallel
and is working ok.

However the network interface does not work. There are entries present under
sysfs for the xenbus but trying to bring it up fails with errors. And also there
seems to be no mac address set (all zeros in sysfs).
When the type=ioemu is removed in the configuration, this works.

I have not much more debugging information beyond that, yet. But it sounds a bit
like NICs should behave the same as block devices. So if there is an emulated
device defined there will be an alternate paravirt interface for it and after
unplugging the emulated ones we end up with the pv ones.
Is that something that can be seen with newer Xen versions, too (I am using
4.1.1)?

-Stefan

On Thu, 2011-12-01 at 15:09 +0000, Stefan Bader wrote:
> Moving to public discussion...
> 
> This was found with Xen hypervisor version supporting device unplugging and
the
> domU kernel having net-/blkfront and pci platform built-in (or as module).
> 
> The block device is defined as hda and the NIC type=ioemu (so theoretically
> guests without pv support would work, too).
> 
> Since both drivers are present, the kernel tries to unplug the emulated
devices
> and succeeds. The blkfront driver detects the xvda device available in
parallel
> and is working ok.
> 
> However the network interface does not work. There are entries present
under
> sysfs for the xenbus but trying to bring it up fails with errors. And also
there
> seems to be no mac address set (all zeros in sysfs).
> When the type=ioemu is removed in the configuration, this works.
Which toolstack are you using?

The weird thing is that, at least with xl, type=ioemu is the default for
an HVM guest.

What vif related entries do you get in xenstore, both front and backend?

Also what does your qemu-dm command line end up looking like?
> I have not much more debugging information beyond that, yet. But it sounds
a bit
> like NICs should behave the same as block devices. So if there is an
emulated
> device defined there will be an alternate paravirt interface for it and
after
> unplugging the emulated ones we end up with the pv ones.
That is certainly the expectation.
> Is that something that can be seen with newer Xen versions, too (I am using
4.1.1)?
I appear to have some other problem with xen-unstable at the moment.
I''ve never noticed a problem in that past, although I don''t
habitually
use type=XXX at all in my vif configuration.

Ian.
> -Stefan

On 01.12.2011 17:03, Ian Campbell wrote:
> On Thu, 2011-12-01 at 15:09 +0000, Stefan Bader wrote:
>> Moving to public discussion...
>>
>> This was found with Xen hypervisor version supporting device unplugging
and the
>> domU kernel having net-/blkfront and pci platform built-in (or as
module).
>>
>> The block device is defined as hda and the NIC type=ioemu (so
theoretically
>> guests without pv support would work, too).
>>
>> Since both drivers are present, the kernel tries to unplug the emulated
devices
>> and succeeds. The blkfront driver detects the xvda device available in
parallel
>> and is working ok.
>>
>> However the network interface does not work. There are entries present
under
>> sysfs for the xenbus but trying to bring it up fails with errors. And
also there
>> seems to be no mac address set (all zeros in sysfs).
>> When the type=ioemu is removed in the configuration, this works.
> 
> Which toolstack are you using?
> 
xm (xl with the same config seems to work)
> The weird thing is that, at least with xl, type=ioemu is the default for
> an HVM guest.
> 
> What vif related entries do you get in xenstore, both front and backend?
> 
output of xenstore-ls attached (hopefully contains all info)
> Also what does your qemu-dm command line end up looking like?
> 
also in the attached file.
>> I have not much more debugging information beyond that, yet. But it
sounds a bit
>> like NICs should behave the same as block devices. So if there is an
emulated
>> device defined there will be an alternate paravirt interface for it and
after
>> unplugging the emulated ones we end up with the pv ones.
> 
> That is certainly the expectation.
> 
>> Is that something that can be seen with newer Xen versions, too (I am
using 4.1.1)?
> 
> I appear to have some other problem with xen-unstable at the moment.
> I''ve never noticed a problem in that past, although I
don''t habitually
> use type=XXX at all in my vif configuration.
> 
> Ian.
> 
>> -Stefan
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Dec 01, 2011 at 04:09:38PM +0100, Stefan Bader
wrote:
> Moving to public discussion...
> 
> This was found with Xen hypervisor version supporting device unplugging and
the
> domU kernel having net-/blkfront and pci platform built-in (or as module).
> 
> The block device is defined as hda and the NIC type=ioemu (so theoretically
> guests without pv support would work, too).
> 
> Since both drivers are present, the kernel tries to unplug the emulated
devices
> and succeeds. The blkfront driver detects the xvda device available in
parallel
> and is working ok.
> 
> However the network interface does not work. There are entries present
under
> sysfs for the xenbus but trying to bring it up fails with errors. And also
there
> seems to be no mac address set (all zeros in sysfs).
> When the type=ioemu is removed in the configuration, this works.
> 
> I have not much more debugging information beyond that, yet. But it sounds
a bit
> like NICs should behave the same as block devices. So if there is an
emulated
> device defined there will be an alternate paravirt interface for it and
after
> unplugging the emulated ones we end up with the pv ones.
> Is that something that can be seen with newer Xen versions, too (I am using
4.1.1)?
> 
Hey,

Have you seen?: 
http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers

Especially the following note:
"NOTE! If you have "type=ioemu" specified for the
"vif"-line, PVHVM drivers WILL NOT work! Don''t specify
"type" parameter for the vif. (with type=ioemu the pvhvm nic in the VM
will have mac address full of zeroes - and thus won''t work!)."

"type=ioemu" is not needed, at least with xm/xend toolstack both HVM
and PVHVM guests work OK without it.

-- Pasi

On Thu, 2011-12-01 at 18:09 +0000, Pasi Kärkkäinen
wrote:
> On Thu, Dec 01, 2011 at 04:09:38PM +0100, Stefan Bader wrote:
> > Moving to public discussion...
> > 
> > This was found with Xen hypervisor version supporting device
unplugging and the
> > domU kernel having net-/blkfront and pci platform built-in (or as
module).
> > 
> > The block device is defined as hda and the NIC type=ioemu (so
theoretically
> > guests without pv support would work, too).
> > 
> > Since both drivers are present, the kernel tries to unplug the
emulated devices
> > and succeeds. The blkfront driver detects the xvda device available in
parallel
> > and is working ok.
> > 
> > However the network interface does not work. There are entries present
under
> > sysfs for the xenbus but trying to bring it up fails with errors. And
also there
> > seems to be no mac address set (all zeros in sysfs).
> > When the type=ioemu is removed in the configuration, this works.
> > 
> > I have not much more debugging information beyond that, yet. But it
sounds a bit
> > like NICs should behave the same as block devices. So if there is an
emulated
> > device defined there will be an alternate paravirt interface for it
and after
> > unplugging the emulated ones we end up with the pv ones.
> > Is that something that can be seen with newer Xen versions, too (I am
using 4.1.1)?
> > 
> 
> Hey,
> 
> Have you seen?: 
> http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers
I was about to say "Please use the new wiki URLs where possible" but
the
section you refer to seems to be missing in the new wiki!

http://wiki.xen.org/wiki/XenLinuxPVonHVMdrivers

It is there in
http://wiki.xen.org/old-wiki-converted/XenLinuxPVonHVMdrivers.txt but
not in the first revision in the new wiki. Lars, do you know what
happened here?
> 
> Especially the following note:
> "NOTE! If you have "type=ioemu" specified for the
"vif"-line, PVHVM drivers WILL NOT work! Don't specify
"type" parameter for the vif. (with type=ioemu the pvhvm nic in the VM
will have mac address full of zeroes - and thus won't work!)."
> 
> "type=ioemu" is not needed, at least with xm/xend toolstack both
HVM and PVHVM guests work OK without it.
> 
> -- Pasi
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 01.12.2011 19:09, Pasi Kärkkäinen wrote:
> On Thu, Dec 01, 2011 at 04:09:38PM +0100, Stefan Bader wrote:
>> Moving to public discussion...
>>
>> This was found with Xen hypervisor version supporting device unplugging
and the
>> domU kernel having net-/blkfront and pci platform built-in (or as
module).
>>
>> The block device is defined as hda and the NIC type=ioemu (so
theoretically
>> guests without pv support would work, too).
>>
>> Since both drivers are present, the kernel tries to unplug the emulated
devices
>> and succeeds. The blkfront driver detects the xvda device available in
parallel
>> and is working ok.
>>
>> However the network interface does not work. There are entries present
under
>> sysfs for the xenbus but trying to bring it up fails with errors. And
also there
>> seems to be no mac address set (all zeros in sysfs).
>> When the type=ioemu is removed in the configuration, this works.
>>
>> I have not much more debugging information beyond that, yet. But it
sounds a bit
>> like NICs should behave the same as block devices. So if there is an
emulated
>> device defined there will be an alternate paravirt interface for it and
after
>> unplugging the emulated ones we end up with the pv ones.
>> Is that something that can be seen with newer Xen versions, too (I am
using 4.1.1)?
>>
> 
> Hey,
> 
> Have you seen?: 
> http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers
> 
> Especially the following note:
> "NOTE! If you have "type=ioemu" specified for the
"vif"-line, PVHVM drivers WILL NOT work! Don''t specify
"type" parameter for the vif. (with type=ioemu the pvhvm nic in the VM
will have mac address full of zeroes - and thus won''t work!)."
> 
> "type=ioemu" is not needed, at least with xm/xend toolstack both
HVM and PVHVM guests work OK without it.
> 
> -- Pasi
> 
Thanks Pasi,

hmm, so it is documented actually and thus sort of expected. Still it is
confusing. For one driver it does not make a difference to use the form of an
emulated device in the config, for the other it does. The xl stack works, the xm
stack does not.
And then, ok this is probably a quite naive approach, it seemed to make sense to
go through the pain of always having potentially both interfaces available
(emulated and pv) so in theory the same guest config can accommodate a guest os
supporting one or the other (or easily switch from one to the other). Otherwise
I would expect an emulated device only when I have hd? in the config and a pv
device when I write xvd?.

-Stefan

On Thu, 2011-12-01 at 17:47 +0000, Stefan Bader wrote:
> On 01.12.2011 17:03, Ian Campbell wrote:
> > On Thu, 2011-12-01 at 15:09 +0000, Stefan Bader wrote:
> >> Moving to public discussion...
> >>
> >> This was found with Xen hypervisor version supporting device
unplugging and the
> >> domU kernel having net-/blkfront and pci platform built-in (or as
module).
> >>
> >> The block device is defined as hda and the NIC type=ioemu (so
theoretically
> >> guests without pv support would work, too).
> >>
> >> Since both drivers are present, the kernel tries to unplug the
emulated devices
> >> and succeeds. The blkfront driver detects the xvda device
available in parallel
> >> and is working ok.
> >>
> >> However the network interface does not work. There are entries
present under
> >> sysfs for the xenbus but trying to bring it up fails with errors.
And also there
> >> seems to be no mac address set (all zeros in sysfs).
> >> When the type=ioemu is removed in the configuration, this works.
> > 
> > Which toolstack are you using?
> > 
> xm (xl with the same config seems to work)
That''s good. xm is deprecated and this seems to be a known issue with
xend. There is a workaround which is to not specify type=.
> 
> > The weird thing is that, at least with xl, type=ioemu is the default
for
> > an HVM guest.
> > 
> > What vif related entries do you get in xenstore, both front and
backend?
> > 
> output of xenstore-ls attached (hopefully contains all info)
FWIW the xend frontend dir looks more empty than I would have expected.
However given the above I don''t think we want to spend time figuring
out
why.

Ian.
> 
> > Also what does your qemu-dm command line end up looking like?
> > 
> also in the attached file.
> 
> >> I have not much more debugging information beyond that, yet. But
it sounds a bit
> >> like NICs should behave the same as block devices. So if there is
an emulated
> >> device defined there will be an alternate paravirt interface for
it and after
> >> unplugging the emulated ones we end up with the pv ones.
> > 
> > That is certainly the expectation.
> > 
> >> Is that something that can be seen with newer Xen versions, too (I
am using 4.1.1)?
> > 
> > I appear to have some other problem with xen-unstable at the moment.
> > I''ve never noticed a problem in that past, although I
don''t habitually
> > use type=XXX at all in my vif configuration.
> > 
> > Ian.
> > 
> >> -Stefan
> > 
> > 
>

On Fri, Dec 02, 2011 at 11:11:38AM +0100, Stefan Bader
wrote:
> On 01.12.2011 19:09, Pasi Kärkkäinen wrote:
> > On Thu, Dec 01, 2011 at 04:09:38PM +0100, Stefan Bader wrote:
> >> Moving to public discussion...
> >>
> >> This was found with Xen hypervisor version supporting device
unplugging and the
> >> domU kernel having net-/blkfront and pci platform built-in (or as
module).
> >>
> >> The block device is defined as hda and the NIC type=ioemu (so
theoretically
> >> guests without pv support would work, too).
> >>
> >> Since both drivers are present, the kernel tries to unplug the
emulated devices
> >> and succeeds. The blkfront driver detects the xvda device
available in parallel
> >> and is working ok.
> >>
> >> However the network interface does not work. There are entries
present under
> >> sysfs for the xenbus but trying to bring it up fails with errors.
And also there
> >> seems to be no mac address set (all zeros in sysfs).
> >> When the type=ioemu is removed in the configuration, this works.
> >>
> >> I have not much more debugging information beyond that, yet. But
it sounds a bit
> >> like NICs should behave the same as block devices. So if there is
an emulated
> >> device defined there will be an alternate paravirt interface for
it and after
> >> unplugging the emulated ones we end up with the pv ones.
> >> Is that something that can be seen with newer Xen versions, too (I
am using 4.1.1)?
> >>
> > 
> > Hey,
> > 
> > Have you seen?: 
> > http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers
> > 
> > Especially the following note:
> > "NOTE! If you have "type=ioemu" specified for the
"vif"-line, PVHVM drivers WILL NOT work! Don''t specify
"type" parameter for the vif. (with type=ioemu the pvhvm nic in the VM
will have mac address full of zeroes - and thus won''t work!)."
> > 
> > "type=ioemu" is not needed, at least with xm/xend toolstack
both HVM and PVHVM guests work OK without it.
> > 
> > -- Pasi
> > 
> Thanks Pasi,
> 
> hmm, so it is documented actually and thus sort of expected. Still it is
> confusing. For one driver it does not make a difference to use the form of
an
> emulated device in the config, for the other it does. The xl stack works,
the xm
> stack does not.
> And then, ok this is probably a quite naive approach, it seemed to make
sense to
> go through the pain of always having potentially both interfaces available
> (emulated and pv) so in theory the same guest config can accommodate a
guest os
> supporting one or the other (or easily switch from one to the other).
Otherwise
> I would expect an emulated device only when I have hd? in the config and a
pv
> device when I write xvd?.
> 
This works for both HVM and PVHVM (also mentioned on the wiki page):
vif = [ ''mac=00:16:5e:02:07:45, bridge=xenbr0, model=e1000'' ]

So there''s no need for "type=ioemu" option with xm/xend.

You can switch between HVM and PVHVM with:
xen_platform_pci=0|1

-- Pasi

--On 2 December 2011 12:41:10 +0200 Pasi Kärkkäinen <pasik@iki.fi> wrote:
>> And then, ok this is probably a quite naive approach, it seemed to make
>> sense to go through the pain of always having potentially both
>> interfaces available (emulated and pv) so in theory the same guest
>> config can accommodate a guest os supporting one or the other (or
easily
>> switch from one to the other). Otherwise I would expect an emulated
>> device only when I have hd? in the config and a pv device when I write
>> xvd?.
>>
>
> This works for both HVM and PVHVM (also mentioned on the wiki page):
> vif = [ 'mac=00:16:5e:02:07:45, bridge=xenbr0, model=e1000' ]
>
> So there's no need for "type=ioemu" option with xm/xend.
>
> You can switch between HVM and PVHVM with:
> xen_platform_pci=0|1
AFAIK changing xen_platform_pci=0|1 will switch rather more than just
the NIC. It will switch your disk too, instantly causing your previously
happily booting OS to fail to boot as the root device name changes.

-- 
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 2011-12-02 at 16:36 +0000, Alex Bligh wrote:
> 
> --On 2 December 2011 12:41:10 +0200 Pasi Kärkkäinen <pasik@iki.fi>
wrote:
> 
> >> And then, ok this is probably a quite naive approach, it seemed to
make
> >> sense to go through the pain of always having potentially both
> >> interfaces available (emulated and pv) so in theory the same guest
> >> config can accommodate a guest os supporting one or the other (or
easily
> >> switch from one to the other). Otherwise I would expect an
emulated
> >> device only when I have hd? in the config and a pv device when I
write
> >> xvd?.
> >>
> >
> > This works for both HVM and PVHVM (also mentioned on the wiki page):
> > vif = [ 'mac=00:16:5e:02:07:45, bridge=xenbr0, model=e1000' ]
> >
> > So there's no need for "type=ioemu" option with xm/xend.
> >
> > You can switch between HVM and PVHVM with:
> > xen_platform_pci=0|1
> 
> AFAIK changing xen_platform_pci=0|1 will switch rather more than just
> the NIC. It will switch your disk too, instantly causing your previously
> happily booting OS to fail to boot as the root device name changes.
We recommend you use "root=LABEL=foo" rather than
"root=/dev/blah" for
this reason. Fortunately most distros use that scheme by default these
days.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

--On 2 December 2011 16:40:48 +0000 Ian Campbell <Ian.Campbell@citrix.com>
wrote:
>> AFAIK changing xen_platform_pci=0|1 will switch rather more than just
>> the NIC. It will switch your disk too, instantly causing your
previously
>> happily booting OS to fail to boot as the root device name changes.
>
> We recommend you use "root=LABEL=foo" rather than
"root=/dev/blah" for
> this reason. Fortunately most distros use that scheme by default these
> days.
Yes; and /etc/fstab. UUID= works too.

FWIW my experience is that various built-for-cloud type distros don''t
use
that scheme, mainly because they use grub1 which IIRC does not support
this, and building images in a non-root environment that have grub1
in is rather easier than grub2. So, for instance, all the vm-builder
stuff in debian/ubuntu used grub1 and did not work this way.

However, my point was that xen_platform_pci does not only change
whether your net driver is emulated or PVHVM, but also whether your
disk, and indeed everything else is emulated or PVHVM.

-- 
Alex Bligh

On Fri, 2011-12-02 at 17:16 +0000, Alex Bligh wrote:
> 
> --On 2 December 2011 16:40:48 +0000 Ian Campbell
<Ian.Campbell@citrix.com>
> wrote:
> 
> >> AFAIK changing xen_platform_pci=0|1 will switch rather more than
just
> >> the NIC. It will switch your disk too, instantly causing your
previously
> >> happily booting OS to fail to boot as the root device name
changes.
> >
> > We recommend you use "root=LABEL=foo" rather than
"root=/dev/blah" for
> > this reason. Fortunately most distros use that scheme by default these
> > days.
> 
> Yes; and /etc/fstab. UUID= works too.
Right.
> FWIW my experience is that various built-for-cloud type distros
don''t use
> that scheme, mainly because they use grub1 which IIRC does not support
> this, and building images in a non-root environment that have grub1
> in is rather easier than grub2.
UUID= and LABEL= are functions of your initrd (actually udev) and not
the bootloader. They work with both grub1 and grub2.
> So, for instance, all the vm-builder stuff in debian/ubuntu used grub1
> and did not work this way.
Which ones are these?

The Debian installer uses UUID where it can AFAIK. Ubuntu''s installer
is
derived from Debian''s so I''d expect it to be the same.
> However, my point was that xen_platform_pci does not only change
> whether your net driver is emulated or PVHVM, but also whether your
> disk, and indeed everything else is emulated or PVHVM.
That is indeed worth highlighting.

Ian.

Ian,

--On 2 December 2011 17:42:35 +0000 Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> Right.
>
>> FWIW my experience is that various built-for-cloud type distros
don''t use
>> that scheme, mainly because they use grub1 which IIRC does not support
>> this, and building images in a non-root environment that have grub1
>> in is rather easier than grub2.
>
> UUID= and LABEL= are functions of your initrd (actually udev) and not
> the bootloader. They work with both grub1 and grub2.
I /think/ we might be slightly at cross purposes.

At least when I was busy making images for Xen for PVHVM a couple of
years ago, the problem is roughly as follows:

The boot loader needs to know what device to load the kernel and initrd 
from. To do this (roughly speaking) it needs to know what BIOS device to 
use. Of course it does not matter whether the boot loader uses the PV 
device or the emulated device, save that this requires the emulated device 
be present (at least whilst the boot loader doesn''t support drivers for
the
pv device). The problem is that the device the boot loader accesses is in 
general specified in the boot loader configuration file not as a bios 
device number, but as a device name. Equally, it needs to know the device 
so that it can write the boot sector in order to reinstall the boot loader, 
set options etc. The problem we ran into here was that this needs to be set 
to xvda in order to to write the boot sector etc., because the sda device 
is unplugged. However, it only recognised a BIOS mapping for sda. So 
neither worked, without fiddling with mappings, but that wasn''t
possible on
a straight install. For some reason, grub1 was far more forgiving.
>> So, for instance, all the vm-builder stuff in debian/ubuntu used grub1
>> and did not work this way.
>
> Which ones are these?
EG:
 http://packages.ubuntu.com/search?keywords=ubuntu-vm-builder
 http://wiki.debian.org/VMBuilder
> The Debian installer uses UUID where it can AFAIK. Ubuntu''s
installer is
> derived from Debian''s so I''d expect it to be the same.
There is more than one method of generating debian/ubuntu images
(debootstrap, multistrap, vmbuilder to name but 3). Some of these
run an installer, some just generate a working image for a particular
environment (and it''s the latter which are problematic).

FWIW my understanding is that though Ubuntu and Debian''s installers
both
use the same underlying d-i stuff (or used to), they are now reasonably 
different (not that this has much bearing on the argument, as the main 
difference between the two is the kernel which has led to rather different 
results between the two); certainly which boot loader they use is
independent of the install architecture, their partitioning schemes
have been substantially different, and I would expect use of UUID/LABEL
not necessarily to correspond.

-- 
Alex Bligh

On Fri, Dec 02, 2011 at 04:40:48PM +0000, Ian Campbell
wrote:
> On Fri, 2011-12-02 at 16:36 +0000, Alex Bligh wrote:
> > 
> > --On 2 December 2011 12:41:10 +0200 Pasi Kärkkäinen
<pasik@iki.fi> wrote:
> > 
> > >> And then, ok this is probably a quite naive approach, it
seemed to make
> > >> sense to go through the pain of always having potentially
both
> > >> interfaces available (emulated and pv) so in theory the same
guest
> > >> config can accommodate a guest os supporting one or the other
(or easily
> > >> switch from one to the other). Otherwise I would expect an
emulated
> > >> device only when I have hd? in the config and a pv device
when I write
> > >> xvd?.
> > >>
> > >
> > > This works for both HVM and PVHVM (also mentioned on the wiki
page):
> > > vif = [ ''mac=00:16:5e:02:07:45, bridge=xenbr0,
model=e1000'' ]
> > >
> > > So there''s no need for "type=ioemu" option
with xm/xend.
> > >
> > > You can switch between HVM and PVHVM with:
> > > xen_platform_pci=0|1
> > 
> > AFAIK changing xen_platform_pci=0|1 will switch rather more than just
> > the NIC. It will switch your disk too, instantly causing your
previously
> > happily booting OS to fail to boot as the root device name changes.
> 
> We recommend you use "root=LABEL=foo" rather than
"root=/dev/blah" for
> this reason. Fortunately most distros use that scheme by default these
> days.
> 
Yep, for example Fedora works out-of-the-box with both HVM and PVHVM..

-- Pasi

On Fri, 2011-12-02 at 18:32 +0000, Alex Bligh wrote:
> Ian,
> 
> --On 2 December 2011 17:42:35 +0000 Ian Campbell
<Ian.Campbell@citrix.com>
> wrote:
> 
> > Right.
> >
> >> FWIW my experience is that various built-for-cloud type distros
don''t use
> >> that scheme, mainly because they use grub1 which IIRC does not
support
> >> this, and building images in a non-root environment that have
grub1
> >> in is rather easier than grub2.
> >
> > UUID= and LABEL= are functions of your initrd (actually udev) and not
> > the bootloader. They work with both grub1 and grub2.
> 
> I /think/ we might be slightly at cross purposes.
> 
> At least when I was busy making images for Xen for PVHVM a couple of
> years ago, the problem is roughly as follows:
> 
> The boot loader needs to know what device to load the kernel and initrd 
> from. To do this (roughly speaking) it needs to know what BIOS device to 
> use. Of course it does not matter whether the boot loader uses the PV 
> device or the emulated device, save that this requires the emulated device 
> be present (at least whilst the boot loader doesn''t support
drivers for the
> pv device). The problem is that the device the boot loader accesses is in 
> general specified in the boot loader configuration file not as a bios 
> device number, but as a device name. Equally, it needs to know the device 
> so that it can write the boot sector in order to reinstall the boot loader,
> set options etc. The problem we ran into here was that this needs to be set
> to xvda in order to to write the boot sector etc., because the sda device 
> is unplugged. However, it only recognised a BIOS mapping for sda. So 
> neither worked, without fiddling with mappings, but that wasn''t
possible on
> a straight install. For some reason, grub1 was far more forgiving.
grub-install /dev/xvdX is supposed to work just as well as
grub-install /dev/sdX depending on which is currently active. If it does
not then you have found a bug and this should be reported against the
grub package in the distro you are using.

This was fixed in grub 1 in Debian Lenny (#456776) and grub2 in Debian
Squeeze (#456777). The grub2 fixes are upstream however grub1 doesn''t
have an upstream so other distros may be missing this fix. If you find
this please report it to them. Likewise if you find that this support
has regressed in Debian (or Ubuntu) then please report those bugs to
them.

Please don''t just assume that because something is broken for you that
this is the way it must be. Report bugs to the appropriate place and
there is some chance that they will get fixed. Likewise if something was
broken for you "years ago" please don''t assume that it has
remained so.
> >> So, for instance, all the vm-builder stuff in debian/ubuntu used
grub1
> >> and did not work this way.
> >
> > Which ones are these?
> 
> EG:
>  http://packages.ubuntu.com/search?keywords=ubuntu-vm-builder
>  http://wiki.debian.org/VMBuilder
> 
> > The Debian installer uses UUID where it can AFAIK. Ubuntu''s
installer is
> > derived from Debian''s so I''d expect it to be the
same.
> 
> There is more than one method of generating debian/ubuntu images
> (debootstrap, multistrap, vmbuilder to name but 3). Some of these
> run an installer, some just generate a working image for a particular
> environment (and it''s the latter which are problematic).
If a tools such as these are not correctly generating a PVHVM capable
configuration when possible then IMHO that is a bug in those tools.
Please report it to the tool authors as such.

If you cannot flip between PVHVM and emulated HVM easily then IMHO this
is also a bug (although perhaps less serious) and should be reported as
such, perhaps as a wishlist bug.
> FWIW my understanding is that though Ubuntu and Debian''s
installers both
> use the same underlying d-i stuff (or used to), they are now reasonably 
> different (not that this has much bearing on the argument, as the main 
> difference between the two is the kernel which has led to rather different 
> results between the two); certainly which boot loader they use is
> independent of the install architecture, their partitioning schemes
> have been substantially different, and I would expect use of UUID/LABEL
> not necessarily to correspond.
The Ubuntu installer folks are closely involved in Debian installer and
in particular the folks who deal with bootloader type things on both
sides are the same people.

Ian.

On 02.12.2011 11:41, Pasi Kärkkäinen wrote:
> On Fri, Dec 02, 2011 at 11:11:38AM +0100, Stefan Bader wrote:
>> On 01.12.2011 19:09, Pasi Kärkkäinen wrote:
>>> On Thu, Dec 01, 2011 at 04:09:38PM +0100, Stefan Bader wrote:
>>>> Moving to public discussion...
>>>>
>>>> This was found with Xen hypervisor version supporting device
unplugging and the
>>>> domU kernel having net-/blkfront and pci platform built-in (or
as module).
>>>>
>>>> The block device is defined as hda and the NIC type=ioemu (so
theoretically
>>>> guests without pv support would work, too).
>>>>
>>>> Since both drivers are present, the kernel tries to unplug the
emulated devices
>>>> and succeeds. The blkfront driver detects the xvda device
available in parallel
>>>> and is working ok.
>>>>
>>>> However the network interface does not work. There are entries
present under
>>>> sysfs for the xenbus but trying to bring it up fails with
errors. And also there
>>>> seems to be no mac address set (all zeros in sysfs).
>>>> When the type=ioemu is removed in the configuration, this
works.
>>>>
>>>> I have not much more debugging information beyond that, yet.
But it sounds a bit
>>>> like NICs should behave the same as block devices. So if there
is an emulated
>>>> device defined there will be an alternate paravirt interface
for it and after
>>>> unplugging the emulated ones we end up with the pv ones.
>>>> Is that something that can be seen with newer Xen versions, too
(I am using 4.1.1)?
>>>>
>>>
>>> Hey,
>>>
>>> Have you seen?: 
>>> http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers
>>>
>>> Especially the following note:
>>> "NOTE! If you have "type=ioemu" specified for the
"vif"-line, PVHVM drivers WILL NOT work! Don''t specify
"type" parameter for the vif. (with type=ioemu the pvhvm nic in the VM
will have mac address full of zeroes - and thus won''t work!)."
>>>
>>> "type=ioemu" is not needed, at least with xm/xend
toolstack both HVM and PVHVM guests work OK without it.
>>>
>>> -- Pasi
>>>
>> Thanks Pasi,
>>
>> hmm, so it is documented actually and thus sort of expected. Still it
is
>> confusing. For one driver it does not make a difference to use the form
of an
>> emulated device in the config, for the other it does. The xl stack
works, the xm
>> stack does not.
>> And then, ok this is probably a quite naive approach, it seemed to make
sense to
>> go through the pain of always having potentially both interfaces
available
>> (emulated and pv) so in theory the same guest config can accommodate a
guest os
>> supporting one or the other (or easily switch from one to the other).
Otherwise
>> I would expect an emulated device only when I have hd? in the config
and a pv
>> device when I write xvd?.
>>
> 
> This works for both HVM and PVHVM (also mentioned on the wiki page):
> vif = [ ''mac=00:16:5e:02:07:45, bridge=xenbr0,
model=e1000'' ]
> 
> So there''s no need for "type=ioemu" option with xm/xend.
> 
> You can switch between HVM and PVHVM with:
> xen_platform_pci=0|1
> 
> -- Pasi
> 
It seems that this works (without the model, too) for xm and xl. So even without
explicitly saying type=ioemu, there is an emulated device created. And if the
platform device is enabled, the emulated devices are unplugged by default.
So it sounds like "type=ioemu" is not only not needed but hurts in the
one case
of xm. Would it make sense to completely remove it from the example configs?

-Stefan

On Mon, 5 Dec 2011, Stefan Bader wrote:
> It seems that this works (without the model, too) for xm and xl. So even
without
> explicitly saying type=ioemu, there is an emulated device created. And if
the
> platform device is enabled, the emulated devices are unplugged by default.
> So it sounds like "type=ioemu" is not only not needed but hurts
in the one case
> of xm. Would it make sense to completely remove it from the example
configs?
I think it would

On 02.12.2011 18:16, Alex Bligh wrote:
> 
> 
> --On 2 December 2011 16:40:48 +0000 Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> 
>>> AFAIK changing xen_platform_pci=0|1 will switch rather more than
just
>>> the NIC. It will switch your disk too, instantly causing your
previously
>>> happily booting OS to fail to boot as the root device name changes.
>>
>> We recommend you use "root=LABEL=foo" rather than
"root=/dev/blah" for
>> this reason. Fortunately most distros use that scheme by default these
>> days.
> 
> Yes; and /etc/fstab. UUID= works too.
> 
> FWIW my experience is that various built-for-cloud type distros
don''t use
> that scheme, mainly because they use grub1 which IIRC does not support
> this, and building images in a non-root environment that have grub1
> in is rather easier than grub2. So, for instance, all the vm-builder
> stuff in debian/ubuntu used grub1 and did not work this way.
> 
> However, my point was that xen_platform_pci does not only change
> whether your net driver is emulated or PVHVM, but also whether your
> disk, and indeed everything else is emulated or PVHVM.
> 
I can understand a policy of using the pv devices whenever it is possible. The
change of the device name is there but as pointed out most distro installations
try to avoid those anyway since there is a similar problem with usb keys or
drives potentially moving around. Same for network devices that get mapped based
on mac address.
Not sure how real the need for a mixed setup is. If, then I can see that it gets
a bit weird. While you can use xen_emul_unplug with other keywords to prevent
unplugging disks or nics. But that would not remove the related pv devices. And
I am not sure whether this would be a desired behavior or actually be feasible
in a clean layered way. Personally I would think nobody could want both
interfaces at the same time but I would not assume I know all of the use cases.

-Stefan

On 03.12.2011 11:44, Ian Campbell wrote:
> On Fri, 2011-12-02 at 18:32 +0000, Alex Bligh wrote:
>> Ian,
>>
>> --On 2 December 2011 17:42:35 +0000 Ian Campbell
<Ian.Campbell@citrix.com>
>> wrote:
>>
>>> Right.
>>>
>>>> FWIW my experience is that various built-for-cloud type distros
don''t use
>>>> that scheme, mainly because they use grub1 which IIRC does not
support
>>>> this, and building images in a non-root environment that have
grub1
>>>> in is rather easier than grub2.
>>>
>>> UUID= and LABEL= are functions of your initrd (actually udev) and
not
>>> the bootloader. They work with both grub1 and grub2.
>>
>> I /think/ we might be slightly at cross purposes.
>>
>> At least when I was busy making images for Xen for PVHVM a couple of
>> years ago, the problem is roughly as follows:
>>
>> The boot loader needs to know what device to load the kernel and initrd
>> from. To do this (roughly speaking) it needs to know what BIOS device
to
>> use. Of course it does not matter whether the boot loader uses the PV 
>> device or the emulated device, save that this requires the emulated
device
>> be present (at least whilst the boot loader doesn''t support
drivers for the
>> pv device). The problem is that the device the boot loader accesses is
in
>> general specified in the boot loader configuration file not as a bios 
>> device number, but as a device name. Equally, it needs to know the
device
>> so that it can write the boot sector in order to reinstall the boot
loader,
>> set options etc. The problem we ran into here was that this needs to be
set
>> to xvda in order to to write the boot sector etc., because the sda
device
>> is unplugged. However, it only recognised a BIOS mapping for sda. So 
>> neither worked, without fiddling with mappings, but that
wasn''t possible on
>> a straight install. For some reason, grub1 was far more forgiving.
> 
> grub-install /dev/xvdX is supposed to work just as well as
> grub-install /dev/sdX depending on which is currently active. If it does
> not then you have found a bug and this should be reported against the
> grub package in the distro you are using.
> 
> This was fixed in grub 1 in Debian Lenny (#456776) and grub2 in Debian
> Squeeze (#456777). The grub2 fixes are upstream however grub1
doesn''t
> have an upstream so other distros may be missing this fix. If you find
> this please report it to them. Likewise if you find that this support
> has regressed in Debian (or Ubuntu) then please report those bugs to
> them.
> 
> Please don''t just assume that because something is broken for you
that
> this is the way it must be. Report bugs to the appropriate place and
> there is some chance that they will get fixed. Likewise if something was
> broken for you "years ago" please don''t assume that it
has remained so.
> 
>>>> So, for instance, all the vm-builder stuff in debian/ubuntu
used grub1
>>>> and did not work this way.
>>>
>>> Which ones are these?
>>
>> EG:
>>  http://packages.ubuntu.com/search?keywords=ubuntu-vm-builder
>>  http://wiki.debian.org/VMBuilder
>>
>>> The Debian installer uses UUID where it can AFAIK.
Ubuntu''s installer is
>>> derived from Debian''s so I''d expect it to be the
same.
>>
>> There is more than one method of generating debian/ubuntu images
>> (debootstrap, multistrap, vmbuilder to name but 3). Some of these
>> run an installer, some just generate a working image for a particular
>> environment (and it''s the latter which are problematic).
> 
> If a tools such as these are not correctly generating a PVHVM capable
> configuration when possible then IMHO that is a bug in those tools.
> Please report it to the tool authors as such.
> 
> If you cannot flip between PVHVM and emulated HVM easily then IMHO this
> is also a bug (although perhaps less serious) and should be reported as
> such, perhaps as a wishlist bug.
> 
>> FWIW my understanding is that though Ubuntu and Debian''s
installers both
>> use the same underlying d-i stuff (or used to), they are now reasonably
>> different (not that this has much bearing on the argument, as the main 
>> difference between the two is the kernel which has led to rather
different
>> results between the two); certainly which boot loader they use is
>> independent of the install architecture, their partitioning schemes
>> have been substantially different, and I would expect use of UUID/LABEL
>> not necessarily to correspond.
> 
> The Ubuntu installer folks are closely involved in Debian installer and
> in particular the folks who deal with bootloader type things on both
> sides are the same people.
> 
At least in my limited experiments, this was not any issue. Though any newer
installation uses grub2 and uuids. The grub stage was working most of the time
(beside initial interrupt related problems). It was more the missing pv modules
and the default to unplug emulated devices, that caused later stages to go not
so smoothly. And finding that there is a way to configure NICs that seems to
work in any combination of pci platform and tool stack sounds like a good way
forward. It just seems that sometimes there are a bit too many knobs to frob and
twiddle. ;)

-Stefan
> Ian.
> 
> 
>

On Mon, 2011-12-05 at 14:55 +0000, Stefan Bader wrote:
> On 02.12.2011 18:16, Alex Bligh wrote:
> > 
> > 
> > --On 2 December 2011 16:40:48 +0000 Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > 
> >>> AFAIK changing xen_platform_pci=0|1 will switch rather more
than just
> >>> the NIC. It will switch your disk too, instantly causing your
previously
> >>> happily booting OS to fail to boot as the root device name
changes.
> >>
> >> We recommend you use "root=LABEL=foo" rather than
"root=/dev/blah" for
> >> this reason. Fortunately most distros use that scheme by default
these
> >> days.
> > 
> > Yes; and /etc/fstab. UUID= works too.
> > 
> > FWIW my experience is that various built-for-cloud type distros
don''t use
> > that scheme, mainly because they use grub1 which IIRC does not support
> > this, and building images in a non-root environment that have grub1
> > in is rather easier than grub2. So, for instance, all the vm-builder
> > stuff in debian/ubuntu used grub1 and did not work this way.
> > 
> > However, my point was that xen_platform_pci does not only change
> > whether your net driver is emulated or PVHVM, but also whether your
> > disk, and indeed everything else is emulated or PVHVM.
> > 
> I can understand a policy of using the pv devices whenever it is possible.
The
> change of the device name is there but as pointed out most distro
installations
> try to avoid those anyway since there is a similar problem with usb keys or
> drives potentially moving around. Same for network devices that get mapped
based
> on mac address.
> Not sure how real the need for a mixed setup is.
Basically non-existent IMHO. The option to not use pv devices is more
for debugging or recovering in the case that you can''t boot from the pv
device for some reason (e.g. because you aren''t using LABEL/UUID= ;-)).
> If, then I can see that it gets
> a bit weird. While you can use xen_emul_unplug with other keywords to
prevent
> unplugging disks or nics. But that would not remove the related pv devices.
It does (at least in upstream Linux) prevent the PV driver from
initialising, which isn''t quite the same but serves the same end
purpose.

Ian.
>  And
> I am not sure whether this would be a desired behavior or actually be
feasible
> in a clean layered way. Personally I would think nobody could want both
> interfaces at the same time but I would not assume I know all of the use
cases.
> 
> -Stefan

Stefan,

--On 5 December 2011 15:55:49 +0100 Stefan Bader 
<stefan.bader@canonical.com> wrote:
> Personally I would think nobody could want both
> interfaces at the same time but I would not assume I know all of the use
> cases.
There are no sensible use cases I know of for having both at the
same time, but there are use cases where either a P2V migration
is done (where the P installation has hardcoded root=/dev/sda) or where
a kernel upgrade is done to a kernel supporting PV drivers and
unplug, where the current behaviour of unplugging stuff is not
the policy of least surprise.

OTOH we''ve got Xen4 cloud images booting with Precise with unplug
better than kvm does, precisely because kvm leaves both around
(as you have probably gathered from a bug that I filed in another
place...)

-- 
Alex Bligh

--On 5 December 2011 16:03:30 +0100 Stefan Bader 
<stefan.bader@canonical.com> wrote:
> At least in my limited experiments, this was not any issue. Though any
> newer installation uses grub2 and uuids.
Well, my experience here was mostly Lucid (as I think I said), and
vm-builder, which I believe is abandonware.

Ubuntu''s cloud image stuff is now far better (having played with
it for a couple of days now we have Xen4 working). I am not sure
what Debian are doing.

-- 
Alex Bligh

Ian,

--On 5 December 2011 16:10:54 +0000 Ian Campbell <Ian.Campbell@citrix.com>
wrote:
>> Not sure how real the need for a mixed setup is.
>
> Basically non-existent IMHO. The option to not use pv devices is more
> for debugging or recovering in the case that you can''t boot from
the pv
> device for some reason (e.g. because you aren''t using LABEL/UUID=
;-)).
I think the question was ''mixed setup'' which I mostly agree
on.

There is plenty of reason to use emulated only setup, including that
the kernel on the image does not support pv drivers at all. There''s
an awful lot of crud that runs on 2.6.18.

-- 
Alex Bligh

 ------------------------------------------------------------------
The information transmitted is intended only for the person or entity to which
it is addressed and may contain confidential and/or privileged material. Any
review, retransmission, dissemination or other use of, or taking of any action
in reliance upon, this information by persons or entities other than the
intended recipient is prohibited. If you received this in error, please contact
the sender and delete the material from any computer.