Xen devel - Apr 2013 - non-dom0 disk backend still not working after recent patches

Although some patches were submitted recently to allow a disk backend
domain to be
specified, it appears there is still some code that presumes dom0 is
the backend.  Has
this actually been tested for creating an HVM?  I am hoping someone a
bit more familiar
with the process of creating an HVM can lend a hand.

As an example, I have a FreeBSD-based domU (9.1 HVM w/ PV drivers,
domid=1, name=freebsd)
on which I created a small 40MB test image at /pool1/media/betest.img,
to see if I could
get another domU to access the image.

After applying the patch I sent out a little earlier today and the
following xl.cfg line
(the below config for hda is my best guess - should this be specified
differently?):

disk = [ ''backend=freebsd, format=raw, vdev=hda,
target=/pool1/media/betest.img, access=rw'',
        
''/mnt/bootimgs/install-amd64-minimal-20130110.iso,,hdc:cdrom,r''
]

I attempt to start up an HVM, but qemu terminates early.  The HVM
starts up fine if the
hda info is removed.  In the below excerpt from "xl -vvv create ...",
it looks like qemu
is not informed that hda is on domain 1, not dom0.  I assume the
backend domid should be
specified on the command line (unless qemu pulls disk info from
xenstore, in which case
why bother passing any disk parameters via the command line?)

. . .
libxl: debug: libxl_dm.c:1211:libxl__spawn_local_dm: Spawning
device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:
/usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   4
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-4,server,nowait
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:
chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   gentoo
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   192.168.1.10:1,to=99
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -global
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   isa-fdc.driveAlibxl:
debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -vga
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   cirrus
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -global
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   vga.vram_size_mb=8
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   order=d
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -smp
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   2,maxcpus=2
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:
e1000,id=nic0,netdev=net0,mac=00:16:3e:4c:ff:31
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -netdev
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:
type=tap,id=net0,ifname=vif4.0-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -M
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   xenfv
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   2040
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:
file=/pool1/media/betest.img,if=ide,index=0,media=disk,format=raw,cache=writeback
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:
file=/mnt/bootimgs/install-amd64-minimal-20130110.iso,if=ide,index=2,media=cdrom,format=raw,cache=writeback,id=ide-5632
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
w=0xac5240 wpath=/local/domain/0/device-model/4/state token=3/0:
register slotnum=3
libxl: debug: libxl_create.c:1259:do_domain_create: ao 0xac3ba0:
inprogress: poller=0xac4ca0, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xac5240
wpath=/local/domain/0/device-model/4/state token=3/0: event
epath=/local/domain/0/device-model/4/state
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch
w=0xac5240 wpath=/local/domain/0/device-model/4/state token=3/0:
deregister slotnum=3
libxl: error: libxl_dm.c:1280:device_model_spawn_outcome: domain 4
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1091:domcreate_devmodel_started: device
model did not start: -3
libxl: error: libxl_dm.c:1311:libxl__destroy_device_model: Device
Model already exited
libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0xac3ba0:
complete, rc=-3
libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0xac3ba0: destroy

On Tue, 2013-04-30 at 16:40 +0100, Eric Shelton wrote:
> Although some patches were submitted recently to allow a disk backend
> domain to be
> specified, it appears there is still some code that presumes dom0 is
> the backend.  Has
> this actually been tested for creating an HVM?  I am hoping someone a
> bit more familiar
> with the process of creating an HVM can lend a hand.
> 
> As an example, I have a FreeBSD-based domU (9.1 HVM w/ PV drivers,
> domid=1, name=freebsd)
> on which I created a small 40MB test image at /pool1/media/betest.img,
> to see if I could
> get another domU to access the image.
> 
> After applying the patch I sent out a little earlier today and the
> following xl.cfg line
> (the below config for hda is my best guess - should this be specified
> differently?):
It look approximately correct to me.
> disk = [ ''backend=freebsd, format=raw, vdev=hda,
> target=/pool1/media/betest.img, access=rw'',
>         
''/mnt/bootimgs/install-amd64-minimal-20130110.iso,,hdc:cdrom,r''
]
> 
> I attempt to start up an HVM, but qemu terminates early.  The HVM
> starts up fine if the
> hda info is removed.  In the below excerpt from "xl -vvv create
...",
> it looks like qemu
> is not informed that hda is on domain 1, not dom0.  I assume the
> backend domid should be
> specified on the command line (unless qemu pulls disk info from
> xenstore, in which case
> why bother passing any disk parameters via the command line?)
These are the emulated (not PV) disks.

Unfortunately I think we currently have a requirement that the qemu for
an HVM guest runs in the same domain as the disk backend, since it needs
to be able to access the disk (or image) directly (qemu doesn''t speak
the PV disk protocol).

In principal we could setup a dom0 vbd for the qemu process to be able
to access this disk, but currently we do not.

Running with a qemu stubdomain would work around this because the
stubdomain would speak the PV disk protocol to the driver domain.

Ian.

On Apr 30, 2013, at 12:44 PM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> On Tue, 2013-04-30 at 16:40 +0100, Eric Shelton wrote:
>> Although some patches were submitted recently to allow a disk backend
>> domain to be
>> specified, it appears there is still some code that presumes dom0 is
>> the backend.  Has
>> this actually been tested for creating an HVM?  I am hoping someone a
>> bit more familiar
>> with the process of creating an HVM can lend a hand.
>>
>> As an example, I have a FreeBSD-based domU (9.1 HVM w/ PV drivers,
>> domid=1, name=freebsd)
>> on which I created a small 40MB test image at /pool1/media/betest.img,
>> to see if I could
>> get another domU to access the image.
>>
>> After applying the patch I sent out a little earlier today and the
>> following xl.cfg line
>> (the below config for hda is my best guess - should this be specified
>> differently?):
>
> It look approximately correct to me.
>
>> disk = [ ''backend=freebsd, format=raw, vdev=hda,
>> target=/pool1/media/betest.img, access=rw'',
>>        
''/mnt/bootimgs/install-amd64-minimal-20130110.iso,,hdc:cdrom,r''
]
>>
>> I attempt to start up an HVM, but qemu terminates early.  The HVM
>> starts up fine if the
>> hda info is removed.  In the below excerpt from "xl -vvv create
...",
>> it looks like qemu
>> is not informed that hda is on domain 1, not dom0.  I assume the
>> backend domid should be
>> specified on the command line (unless qemu pulls disk info from
>> xenstore, in which case
>> why bother passing any disk parameters via the command line?)
>
> These are the emulated (not PV) disks.
>
> Unfortunately I think we currently have a requirement that the qemu for
> an HVM guest runs in the same domain as the disk backend, since it needs
> to be able to access the disk (or image) directly (qemu doesn''t
speak
> the PV disk protocol).
>
> In principal we could setup a dom0 vbd for the qemu process to be able
> to access this disk, but currently we do not.
>
> Running with a qemu stubdomain would work around this because the
> stubdomain would speak the PV disk protocol to the driver domain.
>
> Ian.
>
Thank you- I will have to try using a stubdomain.

Two related questions:

Will/should xl block-attach make the domU backend disk available to dom0?

Would you expect this to work directly for a PV guest, or am I going
to have to turn to pygrub (assuming it similarly facilitates storage
access) or some other mechanism?

- Eric

On Tue, 2013-04-30 at 18:04 +0100, Eric Shelton wrote:
> Will/should xl block-attach make the domU backend disk available to dom0?
I think so, it might depend a bit on your dom0 kernel. Although whether
you would then be allowed to use it as a guest disk (which I guess is
why you are asking) I''m not sure.
> Would you expect this to work directly for a PV guest,
Yes.
> or am I going
> to have to turn to pygrub (assuming it similarly facilitates storage
> access) or some other mechanism?
Actually, pygrub is one thing I would expect not to work (precisely
because it needs access to the domain disks in dom0). I''d expect either
explicitly providing the kernel or using pvgrub would work.

Ian.

On May 1, 2013, at 4:13 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2013-04-30 at 18:04 +0100, Eric Shelton wrote:
>> Will/should xl block-attach make the domU backend disk available to
dom0?
>
> I think so, it might depend a bit on your dom0 kernel. Although whether
> you would then be allowed to use it as a guest disk (which I guess is
> why you are asking) I''m not sure.
>
>> Would you expect this to work directly for a PV guest,
>
> Yes.
>
>> or am I going
>> to have to turn to pygrub (assuming it similarly facilitates storage
>> access) or some other mechanism?
>
> Actually, pygrub is one thing I would expect not to work (precisely
> because it needs access to the domain disks in dom0). I''d expect
either
> explicitly providing the kernel or using pvgrub would work.
>
> Ian.
Thanks again.  I got a chance to look into pvgrub a bit more, and I
see it essentially has the same issue as non-stubdom hvm.

Problems remain in the stubdom code path.  Now it dies in
xenstore_get_backend_path() in qemu-xen-traditional because bpath
corresponds to the correct domU path, but the expected path is under
the dom0 path.  This is because domid_backend is initialized to 0, and
is never changed.  There is a comment where it is initialized that
essentially identifies this as a todo.

My guess is that the backend domain number should be indicated
somewhere in the command line for qemu-dm (likely the -disk
parameters), and then stored in domid-backend.  Does anyone have any
comments on the appropriate implementation?

I also will note that this will require a change to
qemu-xen-traditional.  On the other hand, I expect the change to be
small, and probably reasonable to roll into 4.3.  However, as
qemu-cen-traditional appears to be quasi-depricated, I would like to
know if such a change would be accepted.  A similar change, if not
already implemented, would also be needed in qemu-xen for
linux-stubdoms.

-Eric

On Wed, May 1, 2013 at 4:13 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> On Tue, 2013-04-30 at 18:04 +0100, Eric Shelton wrote:
>> Will/should xl block-attach make the domU backend disk available to
dom0?
>
> I think so, it might depend a bit on your dom0 kernel. Although whether
> you would then be allowed to use it as a guest disk (which I guess is
> why you are asking) I''m not sure.
>
>> Would you expect this to work directly for a PV guest,
>
> Yes.
>
>> or am I going
>> to have to turn to pygrub (assuming it similarly facilitates storage
>> access) or some other mechanism?
>
> Actually, pygrub is one thing I would expect not to work (precisely
> because it needs access to the domain disks in dom0). I''d expect
either
> explicitly providing the kernel or using pvgrub would work.
>
> Ian.
>
[my apologies if this ends up as a duplicate message]

Thanks again.  I got a chance to look into pvgrub a bit more, and I
saw it essentially has the same issue as non-stubdom hvm.

Problems remain in the stubdom code path.  Now it dies in
xenstore_get_backend_path() in qemu-xen-traditional because bpath
corresponds to the correct domU path, but the expected path is under
the dom0 path.  This is because domid_backend is initialized to 0, and
is never changed.  There is a comment where it is initialized that
essentially identifies this as a todo.

My initial guess is that the backend domain number should be indicated
somewhere in the command line for qemu-dm (likely the -disk
parameters), and then stored in domid-backend.  Does anyone have any
comments on the appropriate implementation?

I note that this will require a change to qemu-xen-traditional.  On
the other hand, I expect the change to be small, and it would seem
reasonable to roll into 4.3.  However, as qemu-cen-traditional appears
to be quasi-depricated, it would be helpful to know if such a change
would be accepted.  A similar change, if not already implemented,
would also be needed in qemu-xen for linux-stubdoms.

- Eric

On Wed, 2013-05-01 at 14:27 +0100, Eric Shelton wrote:
> On May 1, 2013, at 4:13 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> 
> > On Tue, 2013-04-30 at 18:04 +0100, Eric Shelton wrote:
> >> Will/should xl block-attach make the domU backend disk available
to dom0?
> >
> > I think so, it might depend a bit on your dom0 kernel. Although
whether
> > you would then be allowed to use it as a guest disk (which I guess is
> > why you are asking) I''m not sure.
> >
> >> Would you expect this to work directly for a PV guest,
> >
> > Yes.
> >
> >> or am I going
> >> to have to turn to pygrub (assuming it similarly facilitates
storage
> >> access) or some other mechanism?
> >
> > Actually, pygrub is one thing I would expect not to work (precisely
> > because it needs access to the domain disks in dom0). I''d
expect either
> > explicitly providing the kernel or using pvgrub would work.
> >
> > Ian.
> 
> Thanks again.  I got a chance to look into pvgrub a bit more, and I
> see it essentially has the same issue as non-stubdom hvm.
Do you mean pvgrub or pygrub? They are very different beasts.
http://wiki.xen.org/wiki/PvGrub vs http://wiki.xen.org/wiki/PyGrub
> Problems remain in the stubdom code path.  Now it dies in
> xenstore_get_backend_path() in qemu-xen-traditional because bpath
> corresponds to the correct domU path, but the expected path is under
> the dom0 path.  This is because domid_backend is initialized to 0, and
> is never changed.  There is a comment where it is initialized that
> essentially identifies this as a todo.
It''s also a bug that this is global since the backend is per device.
> My guess is that the backend domain number should be indicated
> somewhere in the command line for qemu-dm (likely the -disk
> parameters), and then stored in domid-backend.  Does anyone have any
> comments on the appropriate implementation?
For better or worse I think frontends typically parse the backend path
to extract the b.e. domid from it.
> I also will note that this will require a change to
> qemu-xen-traditional.  On the other hand, I expect the change to be
> small, and probably reasonable to roll into 4.3.  However, as
> qemu-cen-traditional appears to be quasi-depricated, I would like to
> know if such a change would be accepted.  A similar change, if not
> already implemented, would also be needed in qemu-xen for
> linux-stubdoms.
I can''t say to be honest, as a prerequisite we would normally like to
see the fix posted for qemu-upstream before considering something for
qemu-xen-trad.

Ian.

On Wed, May 1, 2013 at 11:27 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> On Wed, 2013-05-01 at 14:27 +0100, Eric Shelton wrote:
>> On May 1, 2013, at 4:13 AM, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
>>
>> > On Tue, 2013-04-30 at 18:04 +0100, Eric Shelton wrote:
>> >> Will/should xl block-attach make the domU backend disk
available to dom0?
>> >
>> > I think so, it might depend a bit on your dom0 kernel. Although
whether
>> > you would then be allowed to use it as a guest disk (which I guess
is
>> > why you are asking) I''m not sure.
>> >
>> >> Would you expect this to work directly for a PV guest,
>> >
>> > Yes.
>> >
>> >> or am I going
>> >> to have to turn to pygrub (assuming it similarly facilitates
storage
>> >> access) or some other mechanism?
>> >
>> > Actually, pygrub is one thing I would expect not to work
(precisely
>> > because it needs access to the domain disks in dom0). I''d
expect either
>> > explicitly providing the kernel or using pvgrub would work.
>> >
>> > Ian.
>>
>> Thanks again.  I got a chance to look into pvgrub a bit more, and I
>> see it essentially has the same issue as non-stubdom hvm.
>
> Do you mean pvgrub or pygrub? They are very different beasts.
> http://wiki.xen.org/wiki/PvGrub vs http://wiki.xen.org/wiki/PyGrub
In the end, pvgrub is what I would use.  However, my understanding as
to how pbgrub obtains blkback access was incorrect.
>> Problems remain in the stubdom code path.  Now it dies in
>> xenstore_get_backend_path() in qemu-xen-traditional because bpath
>> corresponds to the correct domU path, but the expected path is under
>> the dom0 path.  This is because domid_backend is initialized to 0, and
>> is never changed.  There is a comment where it is initialized that
>> essentially identifies this as a todo.
>
> It''s also a bug that this is global since the backend is per
device.
That is a very good point.  The variable should go, although the
reliance of get_device_variable_path() complicates its removal a
little.
>> My guess is that the backend domain number should be indicated
>> somewhere in the command line for qemu-dm (likely the -disk
>> parameters), and then stored in domid-backend.  Does anyone have any
>> comments on the appropriate implementation?
>
> For better or worse I think frontends typically parse the backend path
> to extract the b.e. domid from it.
Now that you mention it, it would seem that xenstore is where that
information should be found.  The change should be easy, as the
current code takes care of most of it already.
>> I also will note that this will require a change to
>> qemu-xen-traditional.  On the other hand, I expect the change to be
>> small, and probably reasonable to roll into 4.3.  However, as
>> qemu-cen-traditional appears to be quasi-depricated, I would like to
>> know if such a change would be accepted.  A similar change, if not
>> already implemented, would also be needed in qemu-xen for
>> linux-stubdoms.
>
> I can''t say to be honest, as a prerequisite we would normally like
to
> see the fix posted for qemu-upstream before considering something for
> qemu-xen-trad.
qemu-upstream has different code for interacting with this aspect of
xenstore.  I haven''t tried using it, so it may even already be
correct.  Either way, that prereq may not be of much use for this
issue.

- Eric