Xen devel - Jan 2008 - userspace block backend / gntdev problems

Hi,

I''m running into trouble over and over again with my userspace block
backend daemon (blkbackd) developed as part of the xenner project.

First problem is the fixed limit of 128 slots.  The frontend submits up
to 32 requests, with up to 11 grants each.  With the shared ring this
sums up to 353 grants per block device.  When is blkbackd running in aio
mode, thus many requests are in flight at the same time and thus also
many grants mapped at the same time, the 128 limit is easily reached.  I
don''t even need to stress the disk with bonnie or something, just
booting the virtual machine is enougth.  Any chance replace the
fix-sized array with a list to remove that hard-coded limit?  Or at
least raise the limit to -- say -- 1024 grants?

Second problem is that batched grant mappings (using
xc_gnttab_map_grant_refs) don''t work reliable.  Symtoms I see are
random
failures with ENOMEM for no obvious reason (128 grant limit is *far*
away).  Also host kernel crashes (kernel 2.6.21-2952.fc8xen).

When using xc_gnttab_map_grant_ref only (no batching) and limiting the
number requests in flight to 8 (so we stay below the 128 grant limit)
everything works nicely though.

cheers,
  Gerd

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Gerd,

On 4 Jan 2008, at 13:48, Gerd Hoffmann wrote:
> First problem is the fixed limit of 128 slots.  The frontend  
> submits up
> to 32 requests, with up to 11 grants each.  With the shared ring this
> sums up to 353 grants per block device.  When is blkbackd running  
> in aio
> mode, thus many requests are in flight at the same time and thus also
> many grants mapped at the same time, the 128 limit is easily  
> reached.  I
> don''t even need to stress the disk with bonnie or something, just
> booting the virtual machine is enougth.  Any chance replace the
> fix-sized array with a list to remove that hard-coded limit?  Or at
> least raise the limit to -- say -- 1024 grants?
The 128-grant limit is fairly arbitrary, and I wanted to see how  
people were using gntdev before changing this. The reason for using a  
fixed-size array is that it gives us O(1)-time mapping and unmapping  
of single grants, which I anticipated would be the most frequently- 
used case. I''ll prepare a patch that enables the configuration of  
this limit when the device is opened.
> Second problem is that batched grant mappings (using
> xc_gnttab_map_grant_refs) don''t work reliable.  Symtoms I see are
> random
> failures with ENOMEM for no obvious reason (128 grant limit is *far*
> away).
If it''s failing with ENOMEM, a possible reason is that the address  
space for mapping grants within gntdev (the array I mentioned above)  
is becoming fragmented. Are you combining the mapping of single  
grants and batches within the same gntdev instance? A possible  
workaround would be to use separate gntdev instances for mapping the  
single grants, and for mapping the batches. That way, the  
fragmentation should not occur, if the batches are all of the same size.
> Also host kernel crashes (kernel 2.6.21-2952.fc8xen).
When does this happen? Could you post the kernel OOPS?
> When using xc_gnttab_map_grant_ref only (no batching) and limiting the
> number requests in flight to 8 (so we stay below the 128 grant limit)
> everything works nicely though.
That''s good to know, thanks!

Regards,

Derek Murray.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Derek Murray wrote:
> The 128-grant limit is fairly arbitrary, and I wanted to see how people
> were using gntdev before changing this. The reason for using a
> fixed-size array is that it gives us O(1)-time mapping and unmapping of
> single grants, which I anticipated would be the most frequently-used
> case.
Ok, try a hash instead of a list then ;)
>> Second problem is that batched grant mappings (using
>> xc_gnttab_map_grant_refs) don''t work reliable.  Symtoms I see
are random
>> failures with ENOMEM for no obvious reason (128 grant limit is *far*
>> away).
> 
> If it''s failing with ENOMEM, a possible reason is that the address
space
> for mapping grants within gntdev (the array I mentioned above) is
> becoming fragmented. Are you combining the mapping of single grants and
> batches within the same gntdev instance?
Yes, I''m mixing up single and batched maps (the later can have
different
sizes too, depending on the requests coming in, in the 1 -> 11 range).
But I''ve seen ENOMEM failures with *only* the shared ring being mapped,
i.e. one of 128 slots being used.  That can''t be fragmentation ...
>> Also host kernel crashes (kernel 2.6.21-2952.fc8xen).
> 
> When does this happen? Could you post the kernel OOPS?
Dunno what exactly triggers it.  Oops attached.

cheers,
  Gerd




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Derek Murray <Derek.Murray@cl.cam.ac.uk> writes:
> Hi Gerd,
>
> On 4 Jan 2008, at 13:48, Gerd Hoffmann wrote:
>> First problem is the fixed limit of 128 slots.  The frontend
>> submits up
>> to 32 requests, with up to 11 grants each.  With the shared ring this
>> sums up to 353 grants per block device.  When is blkbackd running
>> in aio
>> mode, thus many requests are in flight at the same time and thus also
>> many grants mapped at the same time, the 128 limit is easily
>> reached.  I
>> don''t even need to stress the disk with bonnie or something,
just
>> booting the virtual machine is enougth.  Any chance replace the
>> fix-sized array with a list to remove that hard-coded limit?  Or at
>> least raise the limit to -- say -- 1024 grants?
>
> The 128-grant limit is fairly arbitrary, and I wanted to see how
> people were using gntdev before changing this. The reason for using a
> fixed-size array is that it gives us O(1)-time mapping and unmapping
> of single grants, which I anticipated would be the most frequently- 
> used case. I''ll prepare a patch that enables the configuration of
> this limit when the device is opened.
Any news on this?  I''d like to try converting the PV framebuffer to
use grants.  I need to map ~2000-5000 pages, depending on the pvfb''s
resolution.

[...]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Markus Armbruster wrote:
> Derek Murray <Derek.Murray@cl.cam.ac.uk> writes:
>
>   
>> Hi Gerd,
>>
>> On 4 Jan 2008, at 13:48, Gerd Hoffmann wrote:
>>     
>>> First problem is the fixed limit of 128 slots.  The frontend
>>> submits up
>>> to 32 requests, with up to 11 grants each.  With the shared ring
this
>>> sums up to 353 grants per block device.  When is blkbackd running
>>> in aio
>>> mode, thus many requests are in flight at the same time and thus
also
>>> many grants mapped at the same time, the 128 limit is easily
>>> reached.  I
>>> don''t even need to stress the disk with bonnie or
something, just
>>> booting the virtual machine is enougth.  Any chance replace the
>>> fix-sized array with a list to remove that hard-coded limit?  Or at
>>> least raise the limit to -- say -- 1024 grants?
>>>       
>> The 128-grant limit is fairly arbitrary, and I wanted to see how
>> people were using gntdev before changing this. The reason for using a
>> fixed-size array is that it gives us O(1)-time mapping and unmapping
>> of single grants, which I anticipated would be the most frequently- 
>> used case. I''ll prepare a patch that enables the configuration
of
>> this limit when the device is opened.
>>     
>
> Any news on this?  I''d like to try converting the PV framebuffer
to
> use grants.  I need to map ~2000-5000 pages, depending on the
pvfb''s
> resolution.
>
> [...]
>   
In my latest post on "Dynamic modes support for PV xenfb" I am using
grants to map an extended framebuffer. I have a single grant ref that
points to 10 other refs. The other refs contain MFNs. Same technique as
the current framebuffer pd array but avoids the 64bit long issue. Kind
of a hybrid approach. I am able to map a 22MB framebuffer when running a
64 bit guest and 44MB when running a 32 bit guest. When the backend is
done with the mapping it sends a message to the frontend to free up the
refs.

I did try to map the whole framebuffers via grants, failed. Like you say
you need a whole bunch of them.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pat Campbell <plc@novell.com> writes:

[...]
> In my latest post on "Dynamic modes support for PV xenfb" I am
using
> grants to map an extended framebuffer. I have a single grant ref that
> points to 10 other refs. The other refs contain MFNs. Same technique as
> the current framebuffer pd array but avoids the 64bit long issue. Kind
> of a hybrid approach. I am able to map a 22MB framebuffer when running a
> 64 bit guest and 44MB when running a 32 bit guest. When the backend is
> done with the mapping it sends a message to the frontend to free up the
> refs.
Uhm, I fear I didn''t get the advantage of your hybrid approach.  Could
you explain?
> I did try to map the whole framebuffers via grants, failed. Like you say
> you need a whole bunch of them.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/1/08 08:41, "Markus Armbruster" <armbru@redhat.com> wrote:
>> In my latest post on "Dynamic modes support for PV xenfb" I
am using
>> grants to map an extended framebuffer. I have a single grant ref that
>> points to 10 other refs. The other refs contain MFNs. Same technique as
>> the current framebuffer pd array but avoids the 64bit long issue. Kind
>> of a hybrid approach. I am able to map a 22MB framebuffer when running
a
>> 64 bit guest and 44MB when running a 32 bit guest. When the backend is
>> done with the mapping it sends a message to the frontend to free up the
>> refs.
> 
> Uhm, I fear I didn''t get the advantage of your hybrid approach. 
Could
> you explain?
Presumably it allows creation of huge framebuffers without using up lots of
grants, or slots in accounting tables that Xen maintains. Given that those
tables can dynamically grow, I''m not sure how useful the two-level
grant
table would be.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 26/1/08 08:41, "Markus Armbruster" <armbru@redhat.com>
wrote:
>
>   
>>> In my latest post on "Dynamic modes support for PV xenfb"
I am using
>>> grants to map an extended framebuffer. I have a single grant ref
that
>>> points to 10 other refs. The other refs contain MFNs. Same
technique as
>>> the current framebuffer pd array but avoids the 64bit long issue.
Kind
>>> of a hybrid approach. I am able to map a 22MB framebuffer when
running a
>>> 64 bit guest and 44MB when running a 32 bit guest. When the backend
is
>>> done with the mapping it sends a message to the frontend to free up
the
>>> refs.
>>>       
>> Uhm, I fear I didn''t get the advantage of your hybrid
approach.  Could
>> you explain?
>>     
>
> Presumably it allows creation of huge framebuffers without using up lots of
> grants, or slots in accounting tables that Xen maintains. Given that those
> tables can dynamically grow, I''m not sure how useful the two-level
grant
> table would be.
>
>  -- Keir
>
>
>   
Well, turns out my email did not really get sent.  Keir is right, it is
a two level grant table.  Solution might not be useful in the general
case but for this device I think it fits the bill. Existing xenfb code
is already doing a two level table which has to be maintained for
backward compatibility reasons, might as well be consistent.  Also by
using a two level grant table we don''t have to extend the event
structure which might be a compatibility issue.

I will get my patches sent up for your review directly.

Pat

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel