Xen devel - Jan 2013 - [PATCH] xen/grant-table: Force to use v1 of grants.

We have the framework to use v2, but there are no backends that
actually use it. The end result is that on PV we use v2 grants
and on PVHVM v1. The v1 has a capacity of 512 grants per page while
the v2 has 256 grants per page. This means we lose about 50%
capacity - and if we want more than 16 VIFs (each VIF takes
512 grants), then we are hitting the max per guest of 32.

Note: The hypervisor provides an over-ride for the default
of 32 frames (pages) per guest: gnttab_max_nr_frames=X

Oracle-bug: 16039922
CC: annie.li@oracle.com
CC: msw@amazon.com
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/grant-table.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
index 95ce9d0..568fd8f 100644
--- a/drivers/xen/grant-table.c
+++ b/drivers/xen/grant-table.c
@@ -1108,10 +1108,8 @@ static void gnttab_request_version(void)
 	int rc;
 	struct gnttab_set_version gsv;
 
-	if (xen_hvm_domain())
-		gsv.version = 1;
-	else
-		gsv.version = 2;
+	gsv.version = 1;
+
 	rc = HYPERVISOR_grant_table_op(GNTTABOP_set_version, &gsv, 1);
 	if (rc == 0 && gsv.version == 2) {
 		grant_table_version = 2;
-- 
1.8.0.2

On Wed, Jan 16, 2013 at 04:22:49PM -0500, Konrad Rzeszutek Wilk
wrote:
> We have the framework to use v2, but there are no backends that
> actually use it. The end result is that on PV we use v2 grants
> and on PVHVM v1. The v1 has a capacity of 512 grants per page while
> the v2 has 256 grants per page. This means we lose about 50%
> capacity - and if we want more than 16 VIFs (each VIF takes
> 512 grants), then we are hitting the max per guest of 32.
> 
> Note: The hypervisor provides an over-ride for the default
> of 32 frames (pages) per guest: gnttab_max_nr_frames=X
I think it would be better if the dom0/service domain and hypervisor
prevent the guest from setting the version to 2 if the backend isn''t
going to make use of them. In the short term, we could add a Xen
option to limit the supported version to 1.

Matt
> Oracle-bug: 16039922
> CC: annie.li@oracle.com
> CC: msw@amazon.com
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  drivers/xen/grant-table.c | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
> index 95ce9d0..568fd8f 100644
> --- a/drivers/xen/grant-table.c
> +++ b/drivers/xen/grant-table.c
> @@ -1108,10 +1108,8 @@ static void gnttab_request_version(void)
>  	int rc;
>  	struct gnttab_set_version gsv;
>  
> -	if (xen_hvm_domain())
> -		gsv.version = 1;
> -	else
> -		gsv.version = 2;
> +	gsv.version = 1;
> +
>  	rc = HYPERVISOR_grant_table_op(GNTTABOP_set_version, &gsv, 1);
>  	if (rc == 0 && gsv.version == 2) {
>  		grant_table_version = 2;
> -- 
> 1.8.0.2

>>> On 17.01.13 at 13:22, Matt Wilson <msw@amazon.com> wrote:
> On Wed, Jan 16, 2013 at 04:22:49PM -0500, Konrad Rzeszutek Wilk wrote:
>> We have the framework to use v2, but there are no backends that
>> actually use it. The end result is that on PV we use v2 grants
>> and on PVHVM v1. The v1 has a capacity of 512 grants per page while
>> the v2 has 256 grants per page. This means we lose about 50%
>> capacity - and if we want more than 16 VIFs (each VIF takes
>> 512 grants), then we are hitting the max per guest of 32.
>> 
>> Note: The hypervisor provides an over-ride for the default
>> of 32 frames (pages) per guest: gnttab_max_nr_frames=X
> 
> I think it would be better if the dom0/service domain and hypervisor
> prevent the guest from setting the version to 2 if the backend
isn''t
> going to make use of them. In the short term, we could add a Xen
> option to limit the supported version to 1.
But that''s something each guest has to decide for itself.

Anyway - if no-one''s using v2, what did it get introduced for
(not only in the hypervisor, but also in the kernel)? I thought
the networking drivers were intended to make use of it...

Jan

On Wed, 2013-01-16 at 21:22 +0000, Konrad Rzeszutek Wilk
wrote:
> Note: The hypervisor provides an over-ride for the default
> of 32 frames (pages) per guest: gnttab_max_nr_frames=X
Should we consider increasing this value?

Ian.

On 1/17/2013 8:05 AM, Jan Beulich wrote:
>>>> On 17.01.13 at 13:22, Matt Wilson<msw@amazon.com>  wrote:
>> On Wed, Jan 16, 2013 at 04:22:49PM -0500, Konrad Rzeszutek Wilk wrote:
>>> We have the framework to use v2, but there are no backends that
>>> actually use it. The end result is that on PV we use v2 grants
>>> and on PVHVM v1. The v1 has a capacity of 512 grants per page while
>>> the v2 has 256 grants per page. This means we lose about 50%
>>> capacity - and if we want more than 16 VIFs (each VIF takes
>>> 512 grants), then we are hitting the max per guest of 32.
>>>
>>> Note: The hypervisor provides an over-ride for the default
>>> of 32 frames (pages) per guest: gnttab_max_nr_frames=X
>> I think it would be better if the dom0/service domain and hypervisor
>> prevent the guest from setting the version to 2 if the backend
isn''t
>> going to make use of them. In the short term, we could add a Xen
>> option to limit the supported version to 1.
> But that''s something each guest has to decide for itself.
>
> Anyway - if no-one''s using v2, what did it get introduced for
> (not only in the hypervisor, but also in the kernel)? I thought
> the networking drivers were intended to make use of it...
That was the idea, but the work on that stalled. I think that once the 
netfront or netback has the right
code, then we can revert this patch - or just make this be more 
automatic and change itself to V2 if
it can.
> Jan
>

On Thu, Jan 17, 2013 at 01:12:36PM +0000, Ian Campbell
wrote:
> On Wed, 2013-01-16 at 21:22 +0000, Konrad Rzeszutek Wilk wrote:
> > Note: The hypervisor provides an over-ride for the default
> > of 32 frames (pages) per guest: gnttab_max_nr_frames=X
> 
> Should we consider increasing this value?
The fuzzy math is that for a VIF, you end up using 512 grants. For a VBD
it is 352 grants. With V1 you have 512 grants in a page, with V2 you have
256. So roughly the math is that you can at max 31VIFs and 1VBD with V1
(PVHVM guests) and 15VIFs and 1VBD with V2 (so all PV guests).

I would increase the value, but I don''t know what a good value
is.
> 
> Ian.
>