Hi Stefano, quite a while back in time, you and Konrad had a discussion about some HVM setup problems via libvirt. One part was graphics and the problem seemed to be that when creating a new instance through xend for HVM, the use of vfb was wrong. It mostly does work but then also defines a vkbd which takes a long time in the xenbus setup to finally fail. Because this was not a really fatal problem it did take a long time to actually get back to it. But now I had a look and found that libvirt indeed does use the vfb form for both the xen-xm and xen-sxpr formats (the latter being used to create guests). The decision is made based on the xend version number in the HVM case. Which would be wrong if I did understand your reply correctly. I have been testing a patch to libvirt, which would not use a vfb definition whenever HVM is used (regardless of xend version). And it does seem to work (xm list -l however has a vfb device definition, but the same happens when creating the instance with a xm style config file that definitely has no vfb section in it). But I am testing based on our 12.04 release which uses Xen 4.1.2. So I want to make sure the solution for libvirt is correct for even the current Xen version. So in short, is this always correct? if (HVM or (PVM when xend is from xen < 3.0.4 / xend version < 3)) do not define a vfb device else /* PVM and xend version >= 3 */ define a vfb device Thanks, Stefan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, 4 Apr 2012, Stefan Bader wrote:> Hi Stefano, > > quite a while back in time, you and Konrad had a discussion about some HVM setup > problems via libvirt. One part was graphics and the problem seemed to be that > when creating a new instance through xend for HVM, the use of vfb was wrong. It > mostly does work but then also defines a vkbd which takes a long time in the > xenbus setup to finally fail. > > Because this was not a really fatal problem it did take a long time to actually > get back to it. But now I had a look and found that libvirt indeed does use the > vfb form for both the xen-xm and xen-sxpr formats (the latter being used to > create guests). The decision is made based on the xend version number in the HVM > case. Which would be wrong if I did understand your reply correctly. > > I have been testing a patch to libvirt, which would not use a vfb definition > whenever HVM is used (regardless of xend version). And it does seem to work (xm > list -l however has a vfb device definition, but the same happens when creating > the instance with a xm style config file that definitely has no vfb section in > it). But I am testing based on our 12.04 release which uses Xen 4.1.2. So I want > to make sure the solution for libvirt is correct for even the current Xen version. > > So in short, is this always correct? > > if (HVM or (PVM when xend is from xen < 3.0.4 / xend version < 3)) > do not define a vfb device > else /* PVM and xend version >= 3 */ > define a vfb devicevkbd and vfb can be considered optimizations for PV on HVM guests. No PV on HVM guest that I know should be able to use vfb right now, but Linux should be able to use vkbd since: commit 5f098ecd4288333d87e2293239fab1c13ec90dae Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> Date: Mon Jul 4 19:22:00 2011 -0700 Input: xen-kbdfront - enable driver for HVM guests Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Signed-off-by: Dmitry Torokhov <dtor@mail.ru> XL in xen-unstable enables vkbd for HVM guests so that you can have keyboard and mouse without usb emulation (that eats significant resources in dom0). That said, vkbd is just a (small) optimization, it is far more important to get rid of the very annoying wait time at bootup. Rather than messing with libvirt and xend I would fix it from the Linux side, see the following thread: http://marc.info/?l=xen-devel&m=133238564132683&w=2 I think that the right thing to do would be removing the additional wait time for vfb and vkbd devices in the PV on HVM case. We don''t want to completely remove vfb and vkbd support for PV on HVM guests though. Konrad, do you agree with the last reply I sent? Do you think you can come up with a patch? Maybe separating non-essential from essential devices to have two wait loops is not feasible? In any case, given the current state of affairs, the first patch in the thread from Konrad is still better than nothing.
On Thu, Apr 05, 2012 at 12:02:45PM +0100, Stefano Stabellini wrote:> On Wed, 4 Apr 2012, Stefan Bader wrote: > > Hi Stefano, > > > > quite a while back in time, you and Konrad had a discussion about some HVM setup > > problems via libvirt. One part was graphics and the problem seemed to be that > > when creating a new instance through xend for HVM, the use of vfb was wrong. It > > mostly does work but then also defines a vkbd which takes a long time in the > > xenbus setup to finally fail. > > > > Because this was not a really fatal problem it did take a long time to actually > > get back to it. But now I had a look and found that libvirt indeed does use the > > vfb form for both the xen-xm and xen-sxpr formats (the latter being used to > > create guests). The decision is made based on the xend version number in the HVM > > case. Which would be wrong if I did understand your reply correctly. > > > > I have been testing a patch to libvirt, which would not use a vfb definition > > whenever HVM is used (regardless of xend version). And it does seem to work (xm > > list -l however has a vfb device definition, but the same happens when creating > > the instance with a xm style config file that definitely has no vfb section in > > it). But I am testing based on our 12.04 release which uses Xen 4.1.2. So I want > > to make sure the solution for libvirt is correct for even the current Xen version. > > > > So in short, is this always correct? > > > > if (HVM or (PVM when xend is from xen < 3.0.4 / xend version < 3)) > > do not define a vfb device > > else /* PVM and xend version >= 3 */ > > define a vfb device > > vkbd and vfb can be considered optimizations for PV on HVM guests. > No PV on HVM guest that I know should be able to use vfb right now, but > Linux should be able to use vkbd since: > > commit 5f098ecd4288333d87e2293239fab1c13ec90dae > Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > Date: Mon Jul 4 19:22:00 2011 -0700 > > Input: xen-kbdfront - enable driver for HVM guests > > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > Signed-off-by: Dmitry Torokhov <dtor@mail.ru> > > XL in xen-unstable enables vkbd for HVM guests so that you can have > keyboard and mouse without usb emulation (that eats significant > resources in dom0). > > That said, vkbd is just a (small) optimization, it is far more > important to get rid of the very annoying wait time at bootup. > Rather than messing with libvirt and xend I would fix it from the Linux > side, see the following thread: > > http://marc.info/?l=xen-devel&m=133238564132683&w=2 > > I think that the right thing to do would be removing the additional wait > time for vfb and vkbd devices in the PV on HVM case. We don''t want to > completely remove vfb and vkbd support for PV on HVM guests though. > > Konrad, do you agree with the last reply I sent? Do you think you canYes.> come up with a patch? Maybe separating non-essential from essentialYes. I was going to after wresting with the dom0_mem=X patches.> devices to have two wait loops is not feasible?It should be no trouble.> In any case, given the current state of affairs, the first patch in the > thread from Konrad is still better than nothing.
On 05.04.2012 13:02, Stefano Stabellini wrote:> On Wed, 4 Apr 2012, Stefan Bader wrote: >> Hi Stefano, >> >> quite a while back in time, you and Konrad had a discussion about some HVM setup >> problems via libvirt. One part was graphics and the problem seemed to be that >> when creating a new instance through xend for HVM, the use of vfb was wrong. It >> mostly does work but then also defines a vkbd which takes a long time in the >> xenbus setup to finally fail. >> >> Because this was not a really fatal problem it did take a long time to actually >> get back to it. But now I had a look and found that libvirt indeed does use the >> vfb form for both the xen-xm and xen-sxpr formats (the latter being used to >> create guests). The decision is made based on the xend version number in the HVM >> case. Which would be wrong if I did understand your reply correctly. >> >> I have been testing a patch to libvirt, which would not use a vfb definition >> whenever HVM is used (regardless of xend version). And it does seem to work (xm >> list -l however has a vfb device definition, but the same happens when creating >> the instance with a xm style config file that definitely has no vfb section in >> it). But I am testing based on our 12.04 release which uses Xen 4.1.2. So I want >> to make sure the solution for libvirt is correct for even the current Xen version. >> >> So in short, is this always correct? >> >> if (HVM or (PVM when xend is from xen < 3.0.4 / xend version < 3)) >> do not define a vfb device >> else /* PVM and xend version >= 3 */ >> define a vfb device > > vkbd and vfb can be considered optimizations for PV on HVM guests. > No PV on HVM guest that I know should be able to use vfb right now, but > Linux should be able to use vkbd since: > > commit 5f098ecd4288333d87e2293239fab1c13ec90dae > Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > Date: Mon Jul 4 19:22:00 2011 -0700 > > Input: xen-kbdfront - enable driver for HVM guests > > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > Signed-off-by: Dmitry Torokhov <dtor@mail.ru> > > XL in xen-unstable enables vkbd for HVM guests so that you can have > keyboard and mouse without usb emulation (that eats significant > resources in dom0). > > That said, vkbd is just a (small) optimization, it is far more > important to get rid of the very annoying wait time at bootup. > Rather than messing with libvirt and xend I would fix it from the Linux > side, see the following thread: > > http://marc.info/?l=xen-devel&m=133238564132683&w=2That would work as well. The downside being that this modification then has to go in any kernels between 3.1 and the patch. The approach from the other side seemed to make sense so far as it changes generated output that seems targeted only at talking to xend. The xen-xm format looks to be usable for xl too. But I would suspect that whenever libvirt starts to support xen api/xl/libxen it will be using a different interface which then could make use of vfb and vkbd. Of course that does not make the change in the kernel less valuable. It prevents the wait time whenever someone manually configures things with vfb. It just seems to be useless to generate a config that has no use for an interface that does not support it. Stefan> I think that the right thing to do would be removing the additional wait > time for vfb and vkbd devices in the PV on HVM case. We don''t want to > completely remove vfb and vkbd support for PV on HVM guests though. > > Konrad, do you agree with the last reply I sent? Do you think you can > come up with a patch? Maybe separating non-essential from essential > devices to have two wait loops is not feasible? > In any case, given the current state of affairs, the first patch in the > thread from Konrad is still better than nothing._______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, 5 Apr 2012, Stefan Bader wrote:> On 05.04.2012 13:02, Stefano Stabellini wrote: > > On Wed, 4 Apr 2012, Stefan Bader wrote: > >> Hi Stefano, > >> > >> quite a while back in time, you and Konrad had a discussion about some HVM setup > >> problems via libvirt. One part was graphics and the problem seemed to be that > >> when creating a new instance through xend for HVM, the use of vfb was wrong. It > >> mostly does work but then also defines a vkbd which takes a long time in the > >> xenbus setup to finally fail. > >> > >> Because this was not a really fatal problem it did take a long time to actually > >> get back to it. But now I had a look and found that libvirt indeed does use the > >> vfb form for both the xen-xm and xen-sxpr formats (the latter being used to > >> create guests). The decision is made based on the xend version number in the HVM > >> case. Which would be wrong if I did understand your reply correctly. > >> > >> I have been testing a patch to libvirt, which would not use a vfb definition > >> whenever HVM is used (regardless of xend version). And it does seem to work (xm > >> list -l however has a vfb device definition, but the same happens when creating > >> the instance with a xm style config file that definitely has no vfb section in > >> it). But I am testing based on our 12.04 release which uses Xen 4.1.2. So I want > >> to make sure the solution for libvirt is correct for even the current Xen version. > >> > >> So in short, is this always correct? > >> > >> if (HVM or (PVM when xend is from xen < 3.0.4 / xend version < 3)) > >> do not define a vfb device > >> else /* PVM and xend version >= 3 */ > >> define a vfb device > > > > vkbd and vfb can be considered optimizations for PV on HVM guests. > > No PV on HVM guest that I know should be able to use vfb right now, but > > Linux should be able to use vkbd since: > > > > commit 5f098ecd4288333d87e2293239fab1c13ec90dae > > Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > > Date: Mon Jul 4 19:22:00 2011 -0700 > > > > Input: xen-kbdfront - enable driver for HVM guests > > > > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > > Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > > Signed-off-by: Dmitry Torokhov <dtor@mail.ru> > > > > XL in xen-unstable enables vkbd for HVM guests so that you can have > > keyboard and mouse without usb emulation (that eats significant > > resources in dom0). > > > > That said, vkbd is just a (small) optimization, it is far more > > important to get rid of the very annoying wait time at bootup. > > Rather than messing with libvirt and xend I would fix it from the Linux > > side, see the following thread: > > > > http://marc.info/?l=xen-devel&m=133238564132683&w=2 > > That would work as well. The downside being that this modification then has to > go in any kernels between 3.1 and the patch. The approach from the other side > seemed to make sense so far as it changes generated output that seems targeted > only at talking to xend. The xen-xm format looks to be usable for xl too. But I > would suspect that whenever libvirt starts to support xen api/xl/libxen it will > be using a different interface which then could make use of vfb and vkbd. > > Of course that does not make the change in the kernel less valuable. It prevents > the wait time whenever someone manually configures things with vfb. > It just seems to be useless to generate a config that has no use for an > interface that does not support it.Nobody is using vfb right now, but vkbd is already enabled in Linux. This is why it is not that clear to me what we should do on the xend side: are we going to disable vfb and keep vkbd? In any case, as I said before, if the alternatives are keeping the wait time or patching xend, I would go for patching xend. If we don''t think we can fix Linux and backport the fix in a reasonable time, patching xend might be the only option.
On 05.04.2012 16:26, Stefano Stabellini wrote:> On Thu, 5 Apr 2012, Stefan Bader wrote: >> On 05.04.2012 13:02, Stefano Stabellini wrote: >>> On Wed, 4 Apr 2012, Stefan Bader wrote: >>>> Hi Stefano, >>>> >>>> quite a while back in time, you and Konrad had a discussion about some HVM setup >>>> problems via libvirt. One part was graphics and the problem seemed to be that >>>> when creating a new instance through xend for HVM, the use of vfb was wrong. It >>>> mostly does work but then also defines a vkbd which takes a long time in the >>>> xenbus setup to finally fail. >>>> >>>> Because this was not a really fatal problem it did take a long time to actually >>>> get back to it. But now I had a look and found that libvirt indeed does use the >>>> vfb form for both the xen-xm and xen-sxpr formats (the latter being used to >>>> create guests). The decision is made based on the xend version number in the HVM >>>> case. Which would be wrong if I did understand your reply correctly. >>>> >>>> I have been testing a patch to libvirt, which would not use a vfb definition >>>> whenever HVM is used (regardless of xend version). And it does seem to work (xm >>>> list -l however has a vfb device definition, but the same happens when creating >>>> the instance with a xm style config file that definitely has no vfb section in >>>> it). But I am testing based on our 12.04 release which uses Xen 4.1.2. So I want >>>> to make sure the solution for libvirt is correct for even the current Xen version. >>>> >>>> So in short, is this always correct? >>>> >>>> if (HVM or (PVM when xend is from xen < 3.0.4 / xend version < 3)) >>>> do not define a vfb device >>>> else /* PVM and xend version >= 3 */ >>>> define a vfb device >>> >>> vkbd and vfb can be considered optimizations for PV on HVM guests. >>> No PV on HVM guest that I know should be able to use vfb right now, but >>> Linux should be able to use vkbd since: >>> >>> commit 5f098ecd4288333d87e2293239fab1c13ec90dae >>> Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> >>> Date: Mon Jul 4 19:22:00 2011 -0700 >>> >>> Input: xen-kbdfront - enable driver for HVM guests >>> >>> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> >>> Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> >>> Signed-off-by: Dmitry Torokhov <dtor@mail.ru> >>> >>> XL in xen-unstable enables vkbd for HVM guests so that you can have >>> keyboard and mouse without usb emulation (that eats significant >>> resources in dom0). >>> >>> That said, vkbd is just a (small) optimization, it is far more >>> important to get rid of the very annoying wait time at bootup. >>> Rather than messing with libvirt and xend I would fix it from the Linux >>> side, see the following thread: >>> >>> http://marc.info/?l=xen-devel&m=133238564132683&w=2 >> >> That would work as well. The downside being that this modification then has to >> go in any kernels between 3.1 and the patch. The approach from the other side >> seemed to make sense so far as it changes generated output that seems targeted >> only at talking to xend. The xen-xm format looks to be usable for xl too. But I >> would suspect that whenever libvirt starts to support xen api/xl/libxen it will >> be using a different interface which then could make use of vfb and vkbd. >> >> Of course that does not make the change in the kernel less valuable. It prevents >> the wait time whenever someone manually configures things with vfb. >> It just seems to be useless to generate a config that has no use for an >> interface that does not support it. > > Nobody is using vfb right now, but vkbd is already enabled in Linux. > This is why it is not that clear to me what we should do on the xend > side: are we going to disable vfb and keep vkbd? > > In any case, as I said before, if the alternatives are keeping the wait > time or patching xend, I would go for patching xend. > If we don''t think we can fix Linux and backport the fix in a reasonable > time, patching xend might be the only option.My impression is that you (the generic you) would not really want to modify xend too much as it and the xm stack should go away anyway. Instead I would fix libvirt''s use of xend when it is known that it is not working well (if using "vfb = [ ''vnc=1, ...'' ]" or similar for sxpr is creating a vkbd and xend/the xm stack does not support it, then just don''t use it). The question of removing the delays is not so much (well yes it is too, but not always in ones own hands) whether it can be done or how quickly. Providing the means to run guests is something rather under our control. Be it Ubuntu as dom0 or Xenserver. But which kernels are run as guests is not. So, as long as xend does not change its behavior, then changing libvirt in a way that does never use the configuration format which causes a vkbd to be created (for HVM) is ok. And if it gets picked up upstream it helps all users of libvirt the same. But if xend would change to allow using a vkbd, then it would be good if that could be synced with a xend version change which could be used inside libvirt to modify its config output (as it does now but in some way with the wrong version). The kernel change to remove delays imo is a completely separate issue. And if just as an additional "pre-caution". _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, 5 Apr 2012, Stefan Bader wrote:> > In any case, as I said before, if the alternatives are keeping the wait > > time or patching xend, I would go for patching xend. > > If we don''t think we can fix Linux and backport the fix in a reasonable > > time, patching xend might be the only option. > > My impression is that you (the generic you) would not really want to modify xend > too much as it and the xm stack should go away anyway. > Instead I would fix libvirt''s use of xend when it is known that it is not > working well (if using "vfb = [ ''vnc=1, ...'' ]" or similar for sxpr is creating > a vkbd and xend/the xm stack does not support it, then just don''t use it). > > The question of removing the delays is not so much (well yes it is too, but not > always in ones own hands) whether it can be done or how quickly. Providing the > means to run guests is something rather under our control. Be it Ubuntu as dom0 > or Xenserver. But which kernels are run as guests is not. > > So, as long as xend does not change its behavior, then changing libvirt in a way > that does never use the configuration format which causes a vkbd to be created > (for HVM) is ok. And if it gets picked up upstream it helps all users of libvirt > the same. > But if xend would change to allow using a vkbd, then it would be good if that > could be synced with a xend version change which could be used inside libvirt to > modify its config output (as it does now but in some way with the wrong version). > > The kernel change to remove delays imo is a completely separate issue. And if > just as an additional "pre-caution".So the argument is that ATM libvirt uses a vfb config line with HVM guests and that is wrong. I agree with you there, the vfb config line is for PV guests only and should be removed from any HVM guests configurations. In fact, even if we add a vfb frontend/backend pair for HVM guests, it probably won''t go through a vfb config line, because the vnc/sdl configuration would be shared between the vfb and vga devices. So you convinced me that is OK to remove it from libvirt :-)
On 05.04.2012 17:29, Stefano Stabellini wrote:> On Thu, 5 Apr 2012, Stefan Bader wrote: >>> In any case, as I said before, if the alternatives are keeping the wait >>> time or patching xend, I would go for patching xend. >>> If we don''t think we can fix Linux and backport the fix in a reasonable >>> time, patching xend might be the only option. >> >> My impression is that you (the generic you) would not really want to modify xend >> too much as it and the xm stack should go away anyway. >> Instead I would fix libvirt''s use of xend when it is known that it is not >> working well (if using "vfb = [ ''vnc=1, ...'' ]" or similar for sxpr is creating >> a vkbd and xend/the xm stack does not support it, then just don''t use it). >> >> The question of removing the delays is not so much (well yes it is too, but not >> always in ones own hands) whether it can be done or how quickly. Providing the >> means to run guests is something rather under our control. Be it Ubuntu as dom0 >> or Xenserver. But which kernels are run as guests is not. >> >> So, as long as xend does not change its behavior, then changing libvirt in a way >> that does never use the configuration format which causes a vkbd to be created >> (for HVM) is ok. And if it gets picked up upstream it helps all users of libvirt >> the same. >> But if xend would change to allow using a vkbd, then it would be good if that >> could be synced with a xend version change which could be used inside libvirt to >> modify its config output (as it does now but in some way with the wrong version). >> >> The kernel change to remove delays imo is a completely separate issue. And if >> just as an additional "pre-caution". > > So the argument is that ATM libvirt uses a vfb config line with HVM > guests and that is wrong.Yes! :)> I agree with you there, the vfb config line is for PV guests only and > should be removed from any HVM guests configurations. > In fact, even if we add a vfb frontend/backend pair for HVM guests, it > probably won''t go through a vfb config line, because the vnc/sdl > configuration would be shared between the vfb and vga devices. > > So you convinced me that is OK to remove it from libvirt :-)Ok, then I try to convince them as well. :) Actually I think we were agreeing most of the time... just not always about the same thing. ;) Which is probably due to me trying to wrap the issue into too many words... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Apr 04, 2012 at 05:08:30PM +0200, Stefan Bader wrote:> Hi Stefano, > > quite a while back in time, you and Konrad had a discussion about some HVM setup > problems via libvirt. One part was graphics and the problem seemed to be that > when creating a new instance through xend for HVM, the use of vfb was wrong. It > mostly does work but then also defines a vkbd which takes a long time in the > xenbus setup to finally fail.So, this patch http://git.kernel.org/?p=linux/kernel/git/konrad/xen.git;a=commit;h=3066616ce23aad5719c23a0f21f32676402cb44b fixes it in the kernel (and I remembered this time to stick stable@kernel.org on it). It should eleviate some of the pains by making the timeout only 30 seconds when vkbd is used, instead of the lengthy 6 minutes.> > Because this was not a really fatal problem it did take a long time to actually > get back to it. But now I had a look and found that libvirt indeed does use the > vfb form for both the xen-xm and xen-sxpr formats (the latter being used to > create guests). The decision is made based on the xend version number in the HVM > case. Which would be wrong if I did understand your reply correctly. > > I have been testing a patch to libvirt, which would not use a vfb definition > whenever HVM is used (regardless of xend version). And it does seem to work (xm > list -l however has a vfb device definition, but the same happens when creating > the instance with a xm style config file that definitely has no vfb section in > it). But I am testing based on our 12.04 release which uses Xen 4.1.2. So I want > to make sure the solution for libvirt is correct for even the current Xen version. > > So in short, is this always correct? > > if (HVM or (PVM when xend is from xen < 3.0.4 / xend version < 3)) > do not define a vfb device > else /* PVM and xend version >= 3 */ > define a vfb device > > Thanks, > Stefan >