Anthony Liguori
2005-Apr-14 01:18 UTC
[Xen-devel] Unstableness in grant table block drivers
We''ve experienced a lot of problems in the new grant table-based block drivers. It''s exposing itself in a number of ways. I''ve seen it stop working after working for a bit, sometimes XenU stops half-way through boot, and on one system init panics because it can''t find a root filesystem. Grant table block drivers are now the default config option in xen-unstable even though it''s marked as experimental. Perhaps it should not be default until it gets a bit more stable? I know the last week''s been really busy, but in the future, it would be nice if a big destabilizing change like this could get announced on xen-devel so we could be on the look out for these sort of bugs. Thanks, -- Anthony Liguori Linux Technology Center (LTC) - IBM Austin E-mail: aliguori@us.ibm.com Phone: (512) 838-1208 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Steven Hand
2005-Apr-14 01:20 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
>We''ve experienced a lot of problems in the new grant table-based block >drivers. It''s exposing itself in a number of ways. I''ve seen it stop >working after working for a bit, sometimes XenU stops half-way through >boot, and on one system init panics because it can''t find a root >filesystem.Have you made any progress in further identifying or narrowing down these bugs? Detailed bug reports or, better, patches, are always more useful than general reports of issues.>Grant table block drivers are now the default config option in >xen-unstable even though it''s marked as experimental. Perhaps it should >not be default until it gets a bit more stable?We''ve not seen any bugs ourselves but obviously if it''s biting you regularly then may be worth while disabling in your config. I''m loathe to disable it by default in the general config since this won''t help us uncover and fix whatever bugs are present.>I know the last week''s been really busy, but in the future, it would be >nice if a big destabilizing change like this could get announced on >xen-devel so we could be on the look out for these sort of bugs.Sure, good point (though as mentioned above, we hadn''t seen any problems ourselves -- we strongly appreciate the community effort in helping us ensure new changes work on a broad range of configs and usage scenarios). cheers, S. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori
2005-Apr-14 01:46 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
Steven Hand wrote:>Have you made any progress in further identifying or narrowing down >these bugs? Detailed bug reports or, better, patches, are always more >useful than general reports of issues. > >This is where having a bugzilla would really help out. At this stage, I don''t know that much but enough that I could enter something in bugzilla. As I learn more, we could add more detail until there was enough that someone (or I) could actually fix it.>We''ve not seen any bugs ourselves but obviously if it''s biting you >regularly then may be worth while disabling in your config. I''m >loathe to disable it by default in the general config since this >won''t help us uncover and fix whatever bugs are present. > >That''s why I suggested just sending a note to the list. Bugs need to be found and that''s what xen-unstable should be for. I''m happy to leave it enabled by default but wanted to let the rest of the list know that if they run into stability problems that might be a good place to start looking. Regards, Anthony Liguori _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Sure, good point (though as mentioned above, we hadn''t seen any > problems ourselves -- we strongly appreciate the community effort > in helping us ensure new changes work on a broad range of configs > and usage scenarios).I think it is safe to say that the blkback/blkfront still have some serious bugs to be fixed even in the absences of grant tables. Exporting more than a couple of block devices to VMs will eventually cause domUs to stop receiving block I/O. I hope to have time to create a scenario that works quicktly and with a sufficiently small configuration that you''ll be able to reproduce it there. To date I''ve only tried it with a fairly large number of LUNs. Developers using xen here periodically have to re-boot their machines as a result. They can''t reproduce it consistently (thankfully), but they''re typically only using 7-10 VBDs. -Kip _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori
2005-Apr-14 02:27 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
Kip Macy wrote:>I think it is safe to say that the blkback/blkfront still have some >serious bugs to be fixed even in the absences of grant tables. > >How many is a lot of VBDs? I''ve done quite a bit of testing of the devices (with Xend and with VM-Tools) and I''ve found the block drivers to be very reliable. I''ve not gone much higher than 3 or 4 VBDs for each domain though. Regards, Anthony Liguoris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
If I do 3 VMs x 40 iSCSI LUNs/VM x foreach (lun) dd if=/dev/zero of=/dev/xbd<n> bs=64k iscsi ops will stop going across the wire in 2 - 3 minutes. Access to the LUNs from DOM0 still works. I haven''t had a chance to try a smaller configuration. This is all scripted so it only takes a minute to set up, but I''m busy trying to get FreeBSD to run (and not have DOM0 lockup) on a recent -unstable. -Kip On 4/13/05, Anthony Liguori <aliguori@us.ibm.com> wrote:> Kip Macy wrote: > > >I think it is safe to say that the blkback/blkfront still have some > >serious bugs to be fixed even in the absences of grant tables. > > > > > How many is a lot of VBDs? I''ve done quite a bit of testing of the > devices (with Xend and with VM-Tools) and I''ve found the block drivers > to be very reliable. I''ve not gone much higher than 3 or 4 VBDs for > each domain though. > > Regards, > Anthony Liguoris >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ge van Geldorp
2005-Apr-14 10:17 UTC
RE: [Xen-devel] Unstableness in grant table block drivers
> From: Christopher Clark <christopher.w.clark@gmail.com> > > The machines you can''t get to start domUs, and symptom of not > finding the root filesystem is exactly what you''d see if your > dom0 and domU configs don''t match; either both or neither > need to be using grant tables for transport.Is CONFIG_XEN_BLKDEV_GRANT going to be a permanent thing or will the "! defined(CONFIG_XEN_BLKDEV_GRANT)" code be removed at some point? IMHO, having binary incompatibilities like this between dom0 and domU is going to be a pain. Ge van Geldorp. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Is CONFIG_XEN_BLKDEV_GRANT going to be a permanent thing or > will the "! > defined(CONFIG_XEN_BLKDEV_GRANT)" code be removed at some > point?The config option will probably be removed at some point. It''s just there to aid debugging in case someone reported block dev problems.> IMHO, having binary incompatibilities like this > between dom0 and domU is going to be a pain.In the *unstable* series you really shouldn''t be relying on compatibility between different versions of the tools, xen, or the kernels. You should always install a complete set of binaries. In the unstable series we reserve the right to change interfaces, and frequently do. Best, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ge van Geldorp
2005-Apr-14 10:34 UTC
RE: [Xen-devel] Unstableness in grant table block drivers
> From: Ian Pratt [mailto:m+Ian.Pratt@cl.cam.ac.uk] > > > Is CONFIG_XEN_BLKDEV_GRANT going to be a permanent thing or > > will the "! defined(CONFIG_XEN_BLKDEV_GRANT)" code be > > removed at some point? > > The config option will probably be removed at some point. > It''s just there to aid debugging in case someone reported > block dev problems.Ok, cool.> In the unstable series we reserve the right to > change interfaces, and frequently do.Yeah, I noticed <g>. That''s cool too, I was just hoping that the config option would be gone by the time -unstable turns into -testing, and it seems that will probably be the case. Ge van Geldorp. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> * Kip Macy <kip.macy@gmail.com> [2005-04-15 16:44]: > > Ouch. I just used it this morning for a tree from last night, so I > > didn''t anticipate any problems. > > Yeah, more than likely it''s something ''unique'' about my aging > hardware. > Thanks for the info, none-the-less.Try "maxcpus=1" I added it a couple of days ago. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher Clark
2005-Apr-16 01:18 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
Anyone have ideas as to what this is in both of Ryan''s Xen/dom0 output? "ssttoop_pth_ithsis__ccppu ud diissaablbele_l_olcaolca_l_AAPPIICC stsompp_s_ethnid_sst_opc pud diisasblea_lboclae_ll_AoPcaIl_CAPI C " It doesn''t look good to me. c. On 4/15/05, Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> wrote:> > * Kip Macy <kip.macy@gmail.com> [2005-04-15 16:44]: > > > Ouch. I just used it this morning for a tree from last night, so I > > > didn''t anticipate any problems. > > > > Yeah, more than likely it''s something ''unique'' about my aging > > hardware. > > Thanks for the info, none-the-less. > > Try "maxcpus=1" > > I added it a couple of days ago. > > Thanks, > Ian > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Anyone have ideas as to what this is in both of Ryan''s > Xen/dom0 output? > > "ssttoop_pth_ithsis__ccppu ud diissaablbele_l_olcaolca_l_AAPPIICC > > stsompp_s_ethnid_sst_opc pud diisasblea_lboclae_ll_AoPcaIl_CAPI > C > "Panic is the only thing that calls smp_send_stop, so I think the badness has already happened. Ryan: is this an SMP guest? Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher Clark
2005-Apr-16 01:32 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
got it. "stop_this_cpu disable_local_APIC" from smp.c c. On 4/15/05, Christopher Clark <christopher.w.clark@gmail.com> wrote:> Anyone have ideas as to what this is in both of Ryan''s Xen/dom0 output? > > "ssttoop_pth_ithsis__ccppu ud diissaablbele_l_olcaolca_l_AAPPIICC > > stsompp_s_ethnid_sst_opc pud diisasblea_lboclae_ll_AoPcaIl_CAPI > C > " >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David Hopwood
2005-Apr-16 03:03 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
Christopher Clark wrote:> got it. "stop_this_cpu disable_local_APIC" > from smp.cThe second one is also interleaved with "smp_send_stop disable_local_APIC".>>stsompp_s_ethnid_sst_opc pud diisasblea_lboclae_ll_AoPcaIl_CAPIC-- David Hopwood <david.nospam.hopwood@blueyonder.co.uk> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Apr-16 15:39 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2005-04-16 10:29]:> > Anyone have ideas as to what this is in both of Ryan''s > > Xen/dom0 output? > > > > "ssttoop_pth_ithsis__ccppu ud diissaablbele_l_olcaolca_l_AAPPIICC > > > > stsompp_s_ethnid_sst_opc pud diisasblea_lboclae_ll_AoPcaIl_CAPI > > C > > " > > Panic is the only thing that calls smp_send_stop, so I think the badness > has already happened. > > Ryan: is this an SMP guest?Yes, that only shows up when trying to boot an SMP guest. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher Clark
2005-Apr-18 16:38 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
> > Ryan: is this an SMP guest? > > Yes, that only shows up when trying to boot an SMP guest.Does the crash only occur when trying to boot an SMP guest - ie. do UP guests start correctly? Ryan and Anthony: I''ve attached a patch that I''d like you to apply to your unstable tree -- it just printk''s almost everything so I can get an idea of the code path you''re running. Alternatively, the three .c files modified by the patch are also attached - they''re taken from 2005-04-14 unstable and you should be able to just replace the files in your tree. After changing your tree and rebuilding, please send the console logs from boot onwards for xen/dom0 and domU, in a uniprocessor configuration, both with and without using grant tables for block transport. thanks Christopher _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Apr-18 16:45 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-18 11:37]:> > > Ryan: is this an SMP guest? > > > > Yes, that only shows up when trying to boot an SMP guest. > > Does the crash only occur when trying to boot an SMP guest - ie. do UP > guests start correctly?No, all domU''s, whether SMP or UP crash the same way.> Ryan and Anthony: I''ve attached a patch that I''d like you to apply to > your unstable tree -- it just printk''s almost everything so I can get > an idea of the code path you''re running. > Alternatively, the three .c files modified by the patch are also > attached - they''re taken from 2005-04-14 unstable and you should be > able to just replace the files in your tree. > > After changing your tree and rebuilding, please send the console logs > from boot onwards for xen/dom0 and domU, in a uniprocessor > configuration, both with and without using grant tables for block > transport.I''ll try this out> thanksSure. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Apr-18 20:51 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-18 11:39]:> Ryan and Anthony: I''ve attached a patch that I''d like you to apply to > your unstable tree -- it just printk''s almost everything so I can get > an idea of the code path you''re running. > Alternatively, the three .c files modified by the patch are also > attached - they''re taken from 2005-04-14 unstable and you should be > able to just replace the files in your tree. > > After changing your tree and rebuilding, please send the console logs > from boot onwards for xen/dom0 and domU, in a uniprocessor > configuration, both with and without using grant tables for block > transport.I''ve applied the patch and I have a bunch of logs. 1) domU boot output when dom0up boots domUup 2) domU boot output when dom0up boots domUsmp 3) xen console output for 1) and 2) 4) domU boot output when dom0smp boots domUup 5) domU boot output when dom0smp boots domUsmp 6) xen console output for 4) and 5) In the above cases, dom0up was simulated with maxcpu=1, as I still don''t have that UP box setup. If you can''t find anything in these logs then I''ll go get a real UP box and repeat the dumps. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher Clark
2005-Apr-20 18:52 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
Ryan Many thanks for taking the traces. I''ve located a bug in the block backend that may or may not be the required fix -- it''s a move of a single line, see the patch below. c. Signed-off-by: Christopher Clark diff -Nru a/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c b/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c --- a/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c 2005-04-20 11:31:20 -07:00 +++ b/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c 2005-04-20 11:31:20 -07:00 @@ -516,11 +516,11 @@ for ( i = 0; i < nseg; i++ ) { + fas = req->frame_and_sects[i]; #ifdef CONFIG_XEN_BLKDEV_GRANT seg[i].buf = (aop[i].u.map_grant_ref.dev_bus_addr << PAGE_SHIFT) | (blkif_first_sect(fas) << 9); #else - fas = req->frame_and_sects[i]; seg[i].buf = (fas & PAGE_MASK) | (blkif_first_sect(fas) << 9); seg[i].nsec = blkif_last_sect(fas) - blkif_first_sect(fas) + 1; if ( seg[i].nsec <= 0 ) On 4/18/05, Ryan Harper <ryanh@us.ibm.com> wrote:> * Christopher Clark <christopher.w.clark@gmail.com> [2005-04-18 11:39]: > > Ryan and Anthony: I''ve attached a patch that I''d like you to apply to > > your unstable tree -- it just printk''s almost everything so I can get > > an idea of the code path you''re running. > > Alternatively, the three .c files modified by the patch are also > > attached - they''re taken from 2005-04-14 unstable and you should be > > able to just replace the files in your tree. > > > > After changing your tree and rebuilding, please send the console logs > > from boot onwards for xen/dom0 and domU, in a uniprocessor > > configuration, both with and without using grant tables for block > > transport. > > I''ve applied the patch and I have a bunch of logs. > > 1) domU boot output when dom0up boots domUup > 2) domU boot output when dom0up boots domUsmp > 3) xen console output for 1) and 2) > 4) domU boot output when dom0smp boots domUup > 5) domU boot output when dom0smp boots domUsmp > 6) xen console output for 4) and 5) > > In the above cases, dom0up was simulated with maxcpu=1, as I still don''t > have that UP box setup. If you can''t find anything in these logs then > I''ll go get a real UP box and repeat the dumps. > > -- > Ryan Harper > Software Engineer; Linux Technology Center > IBM Corp., Austin, Tx > (512) 838-9253 T/L: 678-9253 > ryanh@us.ibm.com > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Apr-20 19:37 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-20 13:53]:> Ryan > > Many thanks for taking the traces. I''ve located a bug in the block > backend that may or may not be the required fix -- it''s a move of a > single line, see the patch below.Sure. I''ll give this patch a try and let you know. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''m trying to build xen on a dual Opteron running Debian (sid) and am encountering include file conflicts. Has anybody successfully built and run on a configuration like this or am I into new territory? thanks, ray _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Chris Wright
2005-Apr-20 20:29 UTC
Re: [Xen-devel] Building unstable on a Debian Opteron box.
* Ray Lanza (ray.lanza@hp.com) wrote:> I''m trying to build xen on a dual Opteron running Debian (sid) and am > encountering include file conflicts. Has anybody successfully built and run > on a configuration like this or am I into new territory?I heard this once before. It looked to me like a Debian header issue. Do you have specifics? (BTW, xen on dual opteron builds fine here). thanks, -chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Apr-22 18:57 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
* Ryan Harper <ryanh@us.ibm.com> [2005-04-20 14:38]:> * Christopher Clark <christopher.w.clark@gmail.com> [2005-04-20 13:53]: > > Ryan > > > > Many thanks for taking the traces. I''ve located a bug in the block > > backend that may or may not be the required fix -- it''s a move of a > > single line, see the patch below. > > Sure. I''ll give this patch a try and let you know.Sorry to be late on this. Moving that patch does it! Thanks. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori
2005-Apr-22 19:06 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
Ryan Harper wrote:>* Ryan Harper <ryanh@us.ibm.com> [2005-04-20 14:38]: > > >>* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-20 13:53]: >> >> >>>Ryan >>> >>>Many thanks for taking the traces. I''ve located a bug in the block >>>backend that may or may not be the required fix -- it''s a move of a >>>single line, see the patch below. >>> >>> >>Sure. I''ll give this patch a try and let you know. >> >> > >Sorry to be late on this. Moving that patch does it! Thanks. > >I''ve had the patch applied and the grant table code enabled for about a day and have not experienced any troubles. Previously, I was experiencing a crash every couple hours. I''ll let you know if anything changes. Thanks Chris! Regards, Anthony Liguori _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Xiaofeng Ling
2005-May-12 01:50 UTC
Re: [Xen-devel] Unstableness in grant table block drivers
I found there is bug in the 2.4 vbd grant table code ===== xen-bk/linux-2.6.11-xen-sparse/drivers/xen/blkfront/blkfront.c 1.53 vs edited ====--- 1.53/linux-2.6.11-xen-sparse/drivers/xen/blkfront/blkfront.c 2005-04-21 17:14:12 +08:00 +++ edited/xen-bk/linux-2.6.11-xen-sparse/drivers/xen/blkfront/blkfront.c 2005-05-12 09:15:38 +08:00 @@ -824,7 +824,7 @@ buffer_ma >> PAGE_SHIFT, ( operation == BLKIF_OP_WRITE ? 1 : 0 ) ); - blk_shadow[id].frame[req->nr_segments] + blk_shadow[req->id].frame[req->nr_segments] buffer_ma >> PAGE_SHIFT; req->frame_and_sects[req->nr_segments] Christopher Clark wrote:> Ryan > > Many thanks for taking the traces. I''ve located a bug in the block > backend that may or may not be the required fix -- it''s a move of a > single line, see the patch below. > > c. > > Signed-off-by: Christopher Clark > > diff -Nru a/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c > b/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c > --- a/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c 2005-04-20 > 11:31:20 -07:00 > +++ b/linux-2.6.11-xen-sparse/drivers/xen/blkback/blkback.c 2005-04-20 > 11:31:20 -07:00 > @@ -516,11 +516,11 @@_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel