Xen devel - Jun 2006 - balloon question

Running with Xen option dom0_mem=6G and kernel option mem=8G, I would have
expected that I would see the system with a
2G balloon right after boot. However, the balloon is empty. Briefly looking at
this, it would seem to me that, with all
pages being reserved when the system starts and only the populated ones getting
the reserved status cleared in
__free_pages_bootmem(), the condition to put pages into the balloon in
balloon_init() is inverted.

Also, dying in xen_destroy_contiguous_region() (and similarly in the failure
recovery code in
xen_create_contiguous_region()) if the space just unmapped can''t be
re-populated seems overly harsh (we''re having a case
where this actually happens); wouldn''t it make sense to make this fatal
only when the page can''t be replaced by a free
one (allocated inside the domain, with the page stolen for this then simply
pushed into the balloon, similar to
balloon_alloc_empty_page_range() except without freeing the memory to Xen)?

Thanks for clarification,
Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8 Jun 2006, at 14:51, Jan Beulich wrote:
> Running with Xen option dom0_mem=6G and kernel option mem=8G, I would 
> have expected that I would see the system with a
> 2G balloon right after boot. However, the balloon is empty. Briefly 
> looking at this, it would seem to me that, with all
> pages being reserved when the system starts and only the populated 
> ones getting the reserved status cleared in
> __free_pages_bootmem(), the condition to put pages into the balloon in 
> balloon_init() is inverted.
i386 deliberately marks pages beyond start_info->nr_pages as not 
Reserved so they get picked up by the balloon driver. Either x86/64 
needs to do the same, or we stop doing that in i386. But then there 
will be pages that are legitimately Reserved that the balloon driver 
will erroneously pick up. Or we could make the balloon driver 
explicitly pick up all pages from start_info->nr_pages. Or have an 
interface to allow arch code to register page ranges with the balloon 
driver. This needs a little thought....
> Also, dying in xen_destroy_contiguous_region() (and similarly in the 
> failure recovery code in
> xen_create_contiguous_region()) if the space just unmapped can''t
be
> re-populated seems overly harsh (we''re having a case
> where this actually happens); wouldn''t it make sense to make this 
> fatal only when the page can''t be replaced by a free
> one (allocated inside the domain, with the page stolen for this then 
> simply pushed into the balloon, similar to
> balloon_alloc_empty_page_range() except without freeing the memory to 
> Xen)?
As it happens I''m fixing this exact problem right now. :-) Expect a fix
in -unstable and -testing in the next few days.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 08.06.06 18:25
>>>
>
>On 8 Jun 2006, at 14:51, Jan Beulich wrote:
>
>> Running with Xen option dom0_mem=6G and kernel option mem=8G, I would 
>> have expected that I would see the system with a
>> 2G balloon right after boot. However, the balloon is empty. Briefly 
>> looking at this, it would seem to me that, with all
>> pages being reserved when the system starts and only the populated 
>> ones getting the reserved status cleared in
>> __free_pages_bootmem(), the condition to put pages into the balloon in 
>> balloon_init() is inverted.
>i386 deliberately marks pages beyond start_info->nr_pages as not 
>Reserved so they get picked up by the balloon driver. Either x86/64 
>needs to do the same, or we stop doing that in i386. But then there 
>will be pages that are legitimately Reserved that the balloon driver 
Where would, under Xen, such pages originate from? Namely beyond
start_info->nr_pages?
>will erroneously pick up. Or we could make the balloon driver 
>explicitly pick up all pages from start_info->nr_pages. Or have an 
It is already starting its scan at start_info->nr_pages.
>interface to allow arch code to register page ranges with the balloon 
>driver. This needs a little thought....
>
>> Also, dying in xen_destroy_contiguous_region() (and similarly in the 
>> failure recovery code in
>> xen_create_contiguous_region()) if the space just unmapped
can''t be
>> re-populated seems overly harsh (we''re having a case
>> where this actually happens); wouldn''t it make sense to make
this
>> fatal only when the page can''t be replaced by a free
>> one (allocated inside the domain, with the page stolen for this then 
>> simply pushed into the balloon, similar to
>> balloon_alloc_empty_page_range() except without freeing the memory to 
>> Xen)?
>
>As it happens I''m fixing this exact problem right now. :-) Expect a
fix
>in -unstable and -testing in the next few days.
Great!

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 9 Jun 2006, at 08:00, Jan Beulich wrote:
>> i386 deliberately marks pages beyond start_info->nr_pages as not
>> Reserved so they get picked up by the balloon driver. Either x86/64
>> needs to do the same, or we stop doing that in i386. But then there
>> will be pages that are legitimately Reserved that the balloon driver
>
> Where would, under Xen, such pages originate from? Namely beyond
> start_info->nr_pages?
>
>> will erroneously pick up. Or we could make the balloon driver
>> explicitly pick up all pages from start_info->nr_pages. Or have an
>
> It is already starting its scan at start_info->nr_pages.
I don''t think there will currently be anything beyond ->nr_pages. So
we
could just get rid of the PageReserved test altogether.

But I think it''s prefereable to take the following loop from 
i386/mm/init-xen.c and put it in x86/64''s mem_init() somewhere:
  for (i = ...->nr_pages; i < max_pfn; i++) {
      ClearPageReserved(...);
      set_page_count(..., 1);
   }

That''ll allow any architecture to reserve stuff beyond ->nr_pages 
without breaking the balloon driver. Could you try adding the above 
code and see how it works out?

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>But I think it''s prefereable to take the following loop from 
>i386/mm/init-xen.c and put it in x86/64''s mem_init() somewhere:
>  for (i = ...->nr_pages; i < max_pfn; i++) {
>      ClearPageReserved(...);
>      set_page_count(..., 1);
>   }
Did you deliberately leave out the increment of totalram_pages that is in
i386''s respective loop? I have to admit that I can''t see why
i386 is doing
that, but for symmetry I added it to x86-64''s loop, too (balloon_init()
re-writes the variable anyway).
>That''ll allow any architecture to reserve stuff beyond
->nr_pages
>without breaking the balloon driver. Could you try adding the above 
>code and see how it works out?
Works out as expected (i.e. balloon now contains all the extra pages).
Patch attached.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 9 Jun 2006, at 10:52, Jan Beulich wrote:
>> But I think it''s prefereable to take the following loop from
>> i386/mm/init-xen.c and put it in x86/64''s mem_init()
somewhere:
>>  for (i = ...->nr_pages; i < max_pfn; i++) {
>>      ClearPageReserved(...);
>>      set_page_count(..., 1);
>>   }
>
> Did you deliberately leave out the increment of totalram_pages that is 
> in
> i386''s respective loop? I have to admit that I can''t see
why i386 is
> doing
> that, but for symmetry I added it to x86-64''s loop, too
(balloon_init()
> re-writes the variable anyway).
The increment is bogus. Gets overwritten by the balloon driver anyway. 
I''ll remove it from your patch and fix i386 too.

Thanks!

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel