I want to implement booting from VHD images using pygrub for blktap3. I''m trying to use the existing code for qemu: in libxl__device_disk_local_initiate_attach I tell libxl to do exactly the same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the switch statement (similar for libxl__device_disk_local_initiate_detach). I run into the following problem: blkfront goes directly from state 1 to state 3, the back-end follows by jumping to state 4, and finally the front-end goes to state 4 and everything works fine (this is what is done for domU guests using blktap3 without pygrub). However, it seems that libxl expects the backend to step through each state (specifically it times out waiting for the back-end to go to state 2 but the back-end has already gone to state 4). If I correctly understand the protocol specification in xen/include/public/io/blkif.h, libxl shouldn''t be doing that. Here''s the output of libxl: libxl_create.c:1244:do_domain_create: ao 0xe79160: create: how=(nil) callback=(nil) poller=0xe77090 libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3 libxl_create.c:691:initiate_domain_create: running bootloader libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap3 libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3 libxl_event.c:559:libxl__ev_xswatch_register: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: register slotnum=3 libxl_create.c:1257:do_domain_create: ao 0xe79160: inprogress: poller=0xe77090, flags=i libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 1 libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 4 libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 4 libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 4 libxl_event.c:661:devstate_timeout: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 timed out libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: deregister slotnum=3 libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0xe78130: deregister unregistered libxl_device.c:882:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd3/0/51712 libxl.c:2687:local_device_attach_cb: unable to add vbd with id 51712: Resource temporarily unavailable libxl_bootloader.c:405:bootloader_disk_attached_cb: failed to attach local disk for bootloader execution If I modify the back-end to go to the same state as the front-end one but making sure that this is done by writing all the intermediate state transitions in XenStore, libxl again complains because it cannot see the back-end state passing from state 2. Here''s the output of libxl: libxl_create.c:1244:do_domain_create: ao 0x1394160: create: how=(nil) callback=(nil) poller=0x1392090 libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3 libxl_create.c:691:initiate_domain_create: running bootloader libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap3 libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3 libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: register slotnum=3 libxl_create.c:1257:do_domain_create: ao 0x1394160: inprogress: poller=0x1392090, flags=i libxl_event.c:503:watchfd_callback: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 1 libxl_event.c:503:watchfd_callback: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 3 libxl_event.c:503:watchfd_callback: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 3 libxl_event.c:661:devstate_timeout: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 timed out libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: deregister slotnum=3 libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x1393130: deregister unregistered libxl_device.c:882:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd3/0/51712 libxl.c:2687:local_device_attach_cb: unable to add vbd with id 51712: Resource temporarily unavailable libxl_bootloader.c:405:bootloader_disk_attached_cb: failed to attach local disk for bootloader execution If I modify the back-end to only step into the next state from its current one in trying to reach the front-end''s state (e.g. if the back-end state is 1 and the front-end state is 3, the back-end will go to state 2), the back-end gets stuck in state 2, which does make libxl happy but the Xenbus protocol isn''t advancing because there isn''t any other event in XenStore such that the back-end can continue increasing its state and utterly match the front-end''s one. Here''s the output of libxl: libxl_create.c:1244:do_domain_create: ao 0x194e160: create: how=(nil) callback=(nil) poller=0x194c090 libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3 libxl_create.c:691:initiate_domain_create: running bootloader libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap3 libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3 libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: register slotnum=3 libxl_create.c:1257:do_domain_create: ao 0x194e160: inprogress: poller=0x194c090, flags=i libxl_event.c:503:watchfd_callback: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 1 libxl_event.c:503:watchfd_callback: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 ok libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: deregister slotnum=3 libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x194d130: deregister unregistered libxl.c:2692:local_device_attach_cb: locally attaching qdisk /dev/xvda libxl_device.c:1108:libxl__wait_for_backend: Backend /local/domain/0/backend/vbd3/0/51712 not ready libxl_bootloader.c:405:bootloader_disk_attached_cb: failed to attach local disk for bootloader execution Any help appreciated. -- Thanos Makatos
FYI modifying the "if (got == ds->wanted)" to "if (got >= ds->wanted)" in devstate_watch_callback (libxl_event.c) resolves the problem. Is this a correct solution or will it break stuff? (There are some checks missing).> -----Original Message----- > From: Thanos Makatos > Sent: 14 May 2013 14:20 > To: ''xen-devel@lists.xen.org'' > Subject: libxl: locally attaching disk > > I want to implement booting from VHD images using pygrub for blktap3. > I''m trying to use the existing code for qemu: in > libxl__device_disk_local_initiate_attach I tell libxl to do exactly the > same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the switch > statement (similar for libxl__device_disk_local_initiate_detach). > > I run into the following problem: blkfront goes directly from state 1 > to state 3, the back-end follows by jumping to state 4, and finally the > front-end goes to state 4 and everything works fine (this is what is > done for domU guests using blktap3 without pygrub). However, it seems > that libxl expects the backend to step through each state (specifically > it times out waiting for the back-end to go to state 2 but the back-end > has already gone to state 4). If I correctly understand the protocol > specification in xen/include/public/io/blkif.h, libxl shouldn''t be
On Tue, 2013-05-14 at 14:20 +0100, Thanos Makatos wrote:> I want to implement booting from VHD images using pygrub for blktap3. > I''m trying to use the existing code for qemu: in > libxl__device_disk_local_initiate_attach I tell libxl to do exactly > the same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the > switch statement (similar for > libxl__device_disk_local_initiate_detach). > > I run into the following problem: blkfront goes directly from state 1 > to state 3, the back-end follows by jumping to state 4, and finally > the front-end goes to state 4 and everything works fine (this is what > is done for domU guests using blktap3 without pygrub). However, it > seems that libxl expects the backend to step through each state > (specifically it times out waiting for the back-end to go to state 2 > but the back-end has already gone to state 4). If I correctly > understand the protocol specification in > xen/include/public/io/blkif.h, libxl shouldn''t be doing that.Indeed, not just because the backend might skip a state but because even if it did go through state 2 it might have gone to state 4 by the time libxl processes the watch and reads the state. Roger, do you have any ideas how to fix this? Usually xenstore interactions wait for state >= N not just == N to handle this case. Ian.
On 14/05/13 15:20, Thanos Makatos wrote:> I want to implement booting from VHD images using pygrub for blktap3. I''m trying to use the existing code for qemu: in libxl__device_disk_local_initiate_attach I tell libxl to do exactly the same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the switch statement (similar for libxl__device_disk_local_initiate_detach). > > I run into the following problem: blkfront goes directly from state 1 to state 3, the back-end follows by jumping to state 4, and finally the front-end goes to state 4 and everything works fine (this is what is done for domU guests using blktap3 without pygrub). However, it seems that libxl expects the backend to step through each state (specifically it times out waiting for the back-end to go to state 2 but the back-end has already gone to state 4). If I correctly understand the protocol specification in xen/include/public/io/blkif.h, libxl shouldn''t be doing that. Here''s the output of libxl:Hello, In libxl we wait for the backend to reach state 2 because for blkback backends we have to execute hotplug scripts. If you take a look at libxl__wait_device_connection in libx_device.c you will see that if the backend is Qemu, we skip the waiting and instead jump to device_hotplug directly. Since blktap backend don''t use hotplug scripts, and hence don''t wait in state 2 for device hotplug execution you should implement something similar for blktap backends.
On 14/05/13 15:42, Thanos Makatos wrote:> FYI modifying the "if (got == ds->wanted)" to "if (got >= ds->wanted)" in devstate_watch_callback (libxl_event.c) resolves the problem. Is this a correct solution or will it break stuff? (There are some checks missing).I don''t think this is the right solution, see my reply to the other thread.
> > I run into the following problem: blkfront goes directly from state 1 > to state 3, the back-end follows by jumping to state 4, and finally the > front-end goes to state 4 and everything works fine (this is what is > done for domU guests using blktap3 without pygrub). However, it seems > that libxl expects the backend to step through each state (specifically > it times out waiting for the back-end to go to state 2 but the back-end > has already gone to state 4). If I correctly understand the protocol > specification in xen/include/public/io/blkif.h, libxl shouldn''t be > doing that. Here''s the output of libxl: > > Hello, > > In libxl we wait for the backend to reach state 2 because for blkback > backends we have to execute hotplug scripts. If you take a look at > libxl__wait_device_connection in libx_device.c you will see that if the > backend is Qemu, we skip the waiting and instead jump to device_hotplug > directly. Since blktap backend don''t use hotplug scripts, and hence > don''t wait in state 2 for device hotplug execution you should implement > something similar for blktap backends.So if I get this straight, we shouldn''t execute device_hotplug for blktap3 because we don''t use hotplug scripts. Also, in the same function, we do need to execute libxl__ev_devstate_wait but setting the XenbusState to XenbusStateConnected, because lixbl must wait for blkfront to connect to tapdisk in order access the block device. Is this correct?
On 14/05/13 18:17, Thanos Makatos wrote:>>> I run into the following problem: blkfront goes directly from state 1 >> to state 3, the back-end follows by jumping to state 4, and finally the >> front-end goes to state 4 and everything works fine (this is what is >> done for domU guests using blktap3 without pygrub). However, it seems >> that libxl expects the backend to step through each state (specifically >> it times out waiting for the back-end to go to state 2 but the back-end >> has already gone to state 4). If I correctly understand the protocol >> specification in xen/include/public/io/blkif.h, libxl shouldn''t be >> doing that. Here''s the output of libxl: >> >> Hello, >> >> In libxl we wait for the backend to reach state 2 because for blkback >> backends we have to execute hotplug scripts. If you take a look at >> libxl__wait_device_connection in libx_device.c you will see that if the >> backend is Qemu, we skip the waiting and instead jump to device_hotplug >> directly. Since blktap backend don''t use hotplug scripts, and hence >> don''t wait in state 2 for device hotplug execution you should implement >> something similar for blktap backends. > > So if I get this straight, we shouldn''t execute device_hotplug for blktap3 because we don''t use hotplug scripts. Also, in the same function, we do need to execute libxl__ev_devstate_wait but setting the XenbusState to XenbusStateConnected, because lixbl must wait for blkfront to connect to tapdisk in order access the block device. Is this correct?local_device_attach_cb (in libxl.c) already does wait for the device to reach state 4 (connected), in libxl__wait_device_connection you should just skip the wait for state 2.