Xen devel - Apr 2005 - RE: Unstableness in grant table block drivers

> We''ve experienced a lot of problems in the new grant 
> table-based block drivers.  It''s exposing itself in a number 
> of ways.  I''ve seen it stop working after working for a bit, 
> sometimes XenU stops half-way through boot, and on one system 
> init panics because it can''t find a root filesystem.
OK, that''s news to us - we haven''t seen any problems.

It''s been tested with dd''s and kernel builds, and
wasn''t expected to be
a particularly dangerous change.

How you confirmed that disabling the option fixes your problems? Can you
give a simple recipe to provoke problems?
> I know the last week''s been really busy, but in the future, 
> it would be nice if a big destabilizing change like this 
> could get announced on xen-devel so we could be on the look 
> out for these sort of bugs.
It doesn''t seem to be destablizing, at least for us. Possibly
there''s
been some feature interaction with later checkins. 

Anyone using the unstable tree should be reading the changelog
closely...

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Pratt wrote:
>OK, that''s news to us - we haven''t seen any problems.
>
>It''s been tested with dd''s and kernel builds, and
wasn''t expected to be
>a particularly dangerous change.
>
>How you confirmed that disabling the option fixes your problems? Can you
>give a simple recipe to provoke problems?
>
We''ve had three different systems lose the ability to start domUs.  
Disabling VBD grant tables fixed the problem.  The best reproducable 
example we have is default config on a domU from today''s xen-unstable 
snapshot.  /dev/console is not accessible and no valid root is found.

In my own case, it was working fairly well and then it just stopped 
working.  It would die mid-way through boot.  Again, disabling VBD 
grant-tables fixed the problem.  I''m going to look into the problem
this
evening and see if I can get you guys a more reproducable scenario.
>It doesn''t seem to be destablizing, at least for us. Possibly
there''s
>been some feature interaction with later checkins. 
>  
>
Possibly.
>Anyone using the unstable tree should be reading the changelog
>closely...
>  
>
I agree.  I do, but I don''t think a lot of people do.  I''m not
trying to
give you guys more work to do :-)  Just a quick note to xen-devel would 
suffice (if you''ve got an extra second).

Thanks Ian,

Regards,
Anthony Liguori
>Ian
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello Anthony

First off, hands up: this was my code and I just didn''t get around to
sending the mail to the list as I had enough to deal with relocating
from the UK to the US last week. The code was, however, tested under
load and working fine, and I did consider it to be ready for wider
testing in unstable.

The machines you can''t get to start domUs, and symptom of not finding
the root filesystem is exactly what you''d see if your dom0 and domU
configs don''t match; either both or neither need to be using grant
tables for transport. Your other issues will need more investigation,
and a reproducable scenario would be extremely helpful indeed.

Regards,

Christopher


On 4/13/05, Anthony Liguori <aliguori@us.ibm.com>
wrote:
> Ian Pratt wrote:
> 
> >OK, that''s news to us - we haven''t seen any problems.
> >
> >It''s been tested with dd''s and kernel builds, and
wasn''t expected to be
> >a particularly dangerous change.
> >
> >How you confirmed that disabling the option fixes your problems? Can
you
> >give a simple recipe to provoke problems?
> >
> We''ve had three different systems lose the ability to start domUs.
> Disabling VBD grant tables fixed the problem.  The best reproducable
> example we have is default config on a domU from today''s
xen-unstable
> snapshot.  /dev/console is not accessible and no valid root is found.
> 
> In my own case, it was working fairly well and then it just stopped
> working.  It would die mid-way through boot.  Again, disabling VBD
> grant-tables fixed the problem.  I''m going to look into the
problem this
> evening and see if I can get you guys a more reproducable scenario.
> 
> >It doesn''t seem to be destablizing, at least for us. Possibly
there''s
> >been some feature interaction with later checkins.
> >
> >
> Possibly.
> 
> >Anyone using the unstable tree should be reading the changelog
> >closely...
> >
> >
> I agree.  I do, but I don''t think a lot of people do. 
I''m not trying to
> give you guys more work to do :-)  Just a quick note to xen-devel would
> suffice (if you''ve got an extra second).
> 
> Thanks Ian,
> 
> Regards,
> Anthony Liguori
> 
> >Ian
> >
> >
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

So domUs have to have grant table support enabled if dom0 does?

        -Kip

On 4/13/05, Christopher Clark <christopher.w.clark@gmail.com>
wrote:
> Hello Anthony
> 
> First off, hands up: this was my code and I just didn''t get around
to
> sending the mail to the list as I had enough to deal with relocating
> from the UK to the US last week. The code was, however, tested under
> load and working fine, and I did consider it to be ready for wider
> testing in unstable.
> 
> The machines you can''t get to start domUs, and symptom of not
finding
> the root filesystem is exactly what you''d see if your dom0 and
domU
> configs don''t match; either both or neither need to be using grant
> tables for transport. Your other issues will need more investigation,
> and a reproducable scenario would be extremely helpful indeed.
> 
> Regards,
> 
> Christopher
> 
> 
> On 4/13/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
> > Ian Pratt wrote:
> >
> > >OK, that''s news to us - we haven''t seen any
problems.
> > >
> > >It''s been tested with dd''s and kernel builds,
and wasn''t expected to be
> > >a particularly dangerous change.
> > >
> > >How you confirmed that disabling the option fixes your problems?
Can you
> > >give a simple recipe to provoke problems?
> > >
> > We''ve had three different systems lose the ability to start
domUs.
> > Disabling VBD grant tables fixed the problem.  The best reproducable
> > example we have is default config on a domU from today''s
xen-unstable
> > snapshot.  /dev/console is not accessible and no valid root is found.
> >
> > In my own case, it was working fairly well and then it just stopped
> > working.  It would die mid-way through boot.  Again, disabling VBD
> > grant-tables fixed the problem.  I''m going to look into the
problem this
> > evening and see if I can get you guys a more reproducable scenario.
> >
> > >It doesn''t seem to be destablizing, at least for us.
Possibly there''s
> > >been some feature interaction with later checkins.
> > >
> > >
> > Possibly.
> >
> > >Anyone using the unstable tree should be reading the changelog
> > >closely...
> > >
> > >
> > I agree.  I do, but I don''t think a lot of people do. 
I''m not trying to
> > give you guys more work to do :-)  Just a quick note to xen-devel
would
> > suffice (if you''ve got an extra second).
> >
> > Thanks Ian,
> >
> > Regards,
> > Anthony Liguori
> >
> > >Ian
> > >
> > >
> > >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christopher Clark wrote:
>Hello Anthony
>
>First off, hands up: this was my code and I just didn''t get around
to
>sending the mail to the list as I had enough to deal with relocating
>from the UK to the US last week. The code was, however, tested under
>load and working fine, and I did consider it to be ready for wider
>testing in unstable.
>  
>
Hi Christopher,

My note was more of a suggestion for the future than a complaint.  You 
guys in Cambridge move very quickly and it''s hard for some of us
outside
to keep up :-)  You guys just keep churning out cool stuff.
>The machines you can''t get to start domUs, and symptom of not
finding
>the root filesystem is exactly what you''d see if your dom0 and domU
>configs don''t match; either both or neither need to be using grant
>tables for transport. Your other issues will need more investigation,
>and a reproducable scenario would be extremely helpful indeed.
>  
>
Actually, this seems like it might be a common thread between all of our 
scenarios.  I''m using (and I assume others) a standard dom0 config file
that has all of my hardware enabled while I just use the default config 
file for domU.

I''ll try to confirm this tomorrow with the other people that were 
experiencing problems.  Perhaps I''ll run a copy of fsstress too and let
you guys know the results.

Regards,
Anthony Liguori
>Regards,
>
>Christopher
>
>
>On 4/13/05, Anthony Liguori <aliguori@us.ibm.com> wrote:
>  
>
>>Ian Pratt wrote:
>>
>>    
>>
>>>OK, that''s news to us - we haven''t seen any
problems.
>>>
>>>It''s been tested with dd''s and kernel builds, and
wasn''t expected to be
>>>a particularly dangerous change.
>>>
>>>How you confirmed that disabling the option fixes your problems? Can
you
>>>give a simple recipe to provoke problems?
>>>
>>>      
>>>
>>We''ve had three different systems lose the ability to start
domUs.
>>Disabling VBD grant tables fixed the problem.  The best reproducable
>>example we have is default config on a domU from today''s
xen-unstable
>>snapshot.  /dev/console is not accessible and no valid root is found.
>>
>>In my own case, it was working fairly well and then it just stopped
>>working.  It would die mid-way through boot.  Again, disabling VBD
>>grant-tables fixed the problem.  I''m going to look into the
problem this
>>evening and see if I can get you guys a more reproducable scenario.
>>
>>    
>>
>>>It doesn''t seem to be destablizing, at least for us.
Possibly there''s
>>>been some feature interaction with later checkins.
>>>
>>>
>>>      
>>>
>>Possibly.
>>
>>    
>>
>>>Anyone using the unstable tree should be reading the changelog
>>>closely...
>>>
>>>
>>>      
>>>
>>I agree.  I do, but I don''t think a lot of people do. 
I''m not trying to
>>give you guys more work to do :-)  Just a quick note to xen-devel would
>>suffice (if you''ve got an extra second).
>>
>>Thanks Ian,
>>
>>Regards,
>>Anthony Liguori
>>
>>    
>>
>>>Ian
>>>
>>>
>>>
>>>      
>>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>    
>>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-13 21:04]:
Hi,
> The machines you can''t get to start domUs, and symptom of not
finding
> the root filesystem is exactly what you''d see if your dom0 and
domU
> configs don''t match; either both or neither need to be using grant
> tables for transport. Your other issues will need more investigation,
I build xen-unstable nightly and install it completely to my test box.
My configs between dom0 and domU are always identicle except for
priviledged domain.
> and a reproducable scenario would be extremely helpful indeed.
My scenario is building the default configs either as UP or with adding
in CONFIG_SMP.  The resule is always, dom0 will boot, domU will not.
domU fails with not finding /dev/console and no root device.

I use loopback device mounts exported as /dev/sda1.

As soon as I turn off CONFIG_XEN_BLKDEV_GRANT in dom0/U configs,
the domU boots properly.

I build with debug=y and I can generate any sort of dump or information
you need to track this down.  Let me know what you need.

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Kip:
Yes, domUs must use grant tables for block device access if dom0 does.
It''s configuring the format of the shared rings between the front and
back ends.

Ryan:
Are your issues with 2.4 or 2.6?
The least tested code is the 2.4 backend in dom0.
2.6 in dom0 should support both 2.4 and 2.6 front ends fine, though so
far I''ve been using partitions rather than exported loopback devices.

I''m aiming to have a working development environment by the end of the
day, and first order of business will be to reproduce these issues.

Christopher



On 4/14/05, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Christopher Clark <christopher.w.clark@gmail.com> [2005-04-13
21:04]:
> Hi,
> 
> > The machines you can''t get to start domUs, and symptom of not
finding
> > the root filesystem is exactly what you''d see if your dom0
and domU
> > configs don''t match; either both or neither need to be using
grant
> > tables for transport. Your other issues will need more investigation,
> 
> I build xen-unstable nightly and install it completely to my test box.
> My configs between dom0 and domU are always identicle except for
> priviledged domain.
> 
> > and a reproducable scenario would be extremely helpful indeed.
> 
> My scenario is building the default configs either as UP or with adding
> in CONFIG_SMP.  The resule is always, dom0 will boot, domU will not.
> domU fails with not finding /dev/console and no root device.
> 
> I use loopback device mounts exported as /dev/sda1.
> 
> As soon as I turn off CONFIG_XEN_BLKDEV_GRANT in dom0/U configs,
> the domU boots properly.
> 
> I build with debug=y and I can generate any sort of dump or information
> you need to track this down.  Let me know what you need.
> 
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-14
11:35]:
> Ryan:
> Are your issues with 2.4 or 2.6?
I''m 2.6 all the way. 

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Ryan

OK, I''ve got a system I can work on now.
Could you please modify the GRANT_DEBUG and GRANT_DEBUG_VERBOSE to be
set to 1, at the top of xen/common/grant_table.c and then make a debug
build of xen.

The verbose option will slow any successful boot down to a crawl, so
you don''t want to leave it on unnecessarily once things are working.

I''d like to see:
all the boot messages from xen and dom0
the same from your attempt to boot domU

If you have to option to test it on a uniprocessor box, please use
that as that would match what I''m using here.

thanks

Christopher


On 4/14/05, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Christopher Clark <christopher.w.clark@gmail.com> [2005-04-14
11:35]:
> > Ryan:
> > Are your issues with 2.4 or 2.6?
> 
> I''m 2.6 all the way.
> 
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-15
13:44]:
> Hi Ryan
> 
> OK, I''ve got a system I can work on now.
> Could you please modify the GRANT_DEBUG and GRANT_DEBUG_VERBOSE to be
> set to 1, at the top of xen/common/grant_table.c and then make a debug
> build of xen.
> 
> The verbose option will slow any successful boot down to a crawl, so
> you don''t want to leave it on unnecessarily once things are
working.
> 
> I''d like to see:
> all the boot messages from xen and dom0
> the same from your attempt to boot domU
Attached, both smp and non-smp.
> If you have to option to test it on a uniprocessor box, please use
> that as that would match what I''m using here.
I only have an SMP box setup right now, Ill repeat those dumps on a UP
box tomorrow.

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

pass nosmp to xen at boot and you have UP

On 4/15/05, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Christopher Clark <christopher.w.clark@gmail.com> [2005-04-15
13:44]:
> > Hi Ryan
> >
> > OK, I''ve got a system I can work on now.
> > Could you please modify the GRANT_DEBUG and GRANT_DEBUG_VERBOSE to be
> > set to 1, at the top of xen/common/grant_table.c and then make a debug
> > build of xen.
> >
> > The verbose option will slow any successful boot down to a crawl, so
> > you don''t want to leave it on unnecessarily once things are
working.
> >
> > I''d like to see:
> > all the boot messages from xen and dom0
> > the same from your attempt to boot domU
> 
> Attached, both smp and non-smp.
> 
> > If you have to option to test it on a uniprocessor box, please use
> > that as that would match what I''m using here.
> 
> I only have an SMP box setup right now, Ill repeat those dumps on a UP
> box tomorrow.
> 
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Kip Macy <kip.macy@gmail.com> [2005-04-15
16:32]:
> pass nosmp to xen at boot and you have UP
Indeed.  Of course, now the machine won''t reboot with that option
passed.  *sigh*

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ouch. I just used it this morning for a tree from last night, so I
didn''t anticipate any problems.

         -Kip


On 4/15/05, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Kip Macy <kip.macy@gmail.com> [2005-04-15 16:32]:
> > pass nosmp to xen at boot and you have UP
> 
> Indeed.  Of course, now the machine won''t reboot with that option
> passed.  *sigh*
> 
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Kip Macy <kip.macy@gmail.com> [2005-04-15
16:44]:
> Ouch. I just used it this morning for a tree from last night, so I
> didn''t anticipate any problems.
Yeah, more than likely it''s something ''unique'' about
my aging hardware.
Thanks for the info, none-the-less.

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Kip Macy <kip.macy@gmail.com> [2005-04-15
16:32]:
> pass nosmp to xen at boot and you have UP
The behavior is the same as I posted before.  I will also try on a real
UP box tomorrow as well.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Could you try without the block front end driver in dom0? Is there a
reason you need it in there?

c.

On 4/15/05, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Kip Macy <kip.macy@gmail.com> [2005-04-15 16:32]:
> > pass nosmp to xen at boot and you have UP
> 
> The behavior is the same as I posted before.  I will also try on a real
> UP box tomorrow as well.
> 
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Christopher Clark <christopher.w.clark@gmail.com> [2005-04-15
17:05]:
> Could you try without the block front end driver in dom0? Is there a
> reason you need it in there?
I don''t understand much about the front/back end driver stuff.  I just
take the default xenolinux configs and go with them.

Do you have a .config for dom0/domU that you want me to use?

Or at least what CONFIG_ options should be on/off for the dump?

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel