Xen devel - Feb 2007 - How much memory does a domain REALLY use...

Apologies if this has been discussed before, but I''ve been trying to
work out just how much memory domains consume (so as to be able to
provision memory properly) - this all started when I noticed that a
system running JUST my 768MB Dom0 was actually using about 890MB
according to xenstat - an extra 122MB seemed a little extreme to me so I
went digging; I think I''ve found something over half so far but
I''m not
sure about the remaining so I was hoping folks could educate me...

Here''s what I think I''ve found (all this is i386, 32-bit
PAE...):

First, from the domain/dma heap zones, in addition to the actual domain
memory allocated with alloc_chunk() in 2MB pieces:

. First, the biggie -- swiotlb; if I''m not mistaken, this allocates
64MB
from the domain
  heap (in addition to the 64MB of low memory in the domain itself).
Allocated with
  xen_create_contiguous_region in 256KB chunks.
. paging_init() allocates the m2p translation table -- 2MB
. The Linux dma_alloc_coherent() routine also calls
xen_create_contiguous_region -- each call
  to this routine will allocate a minimum of one page from the domain
heap -- every single
  driver that does DMA will do this, so there is the possibility of some
significant
  allocations due to DMA buffers. I''m wondering if this comprises the
remainder of the
  122MB I see...
. There are a few other single page allocations but nothing huge

Then, from the Xen heap zone a bunch of small stuff:

. 128KB for console debug trace (debugtrace_init)
. 16KB for serial tx buffer
. 16KB for page directory pages (arch_domain_create)
. 16KB for stack per VCPU (do_boot_cpu in smpboot)
. 16KB per grant table (one per domain - _grant_table_create)
. 4KB for grant table references (can grow to 128KB -
__gnttab_map_grant_ref)
. max of 128KB for xenoprofile buffer space - presumably zero until I
actually
  start profiling.

So - I''ve found about 69MB for sure, plus an unknown quantity of DMA
common buffer allocations - still seems a bit shy of the 122MB I see...
all thoughts welcome...

Thanks,
Simon


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/2/07 01:21, "Graham, Simon" <Simon.Graham@stratus.com>
wrote:
> . First, the biggie -- swiotlb; if I''m not mistaken, this
allocates 64MB
> from the domain
>   heap (in addition to the 64MB of low memory in the domain itself).
> Allocated with
>   xen_create_contiguous_region in 256KB chunks.
Xen_create_contiguous_region() gives up the existing RAM and replaces with a
new contiguous extent, so it shouldn''t allocate net extra memory.

How are you determining amount of RAM used: xm info and looking at freemem?
Or xm list and look at memory usage for dom0?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > . First, the biggie -- swiotlb; if I''m not mistaken, this
allocates
> 64MB
> > from the domain
> >   heap (in addition to the 64MB of low memory in the domain itself).
> > Allocated with
> >   xen_create_contiguous_region in 256KB chunks.
> 
> Xen_create_contiguous_region() gives up the existing RAM and replaces
> with a
> new contiguous extent, so it shouldn''t allocate net extra memory.
> 
Ah - missed that; thanks! So - for the swiotlb it re-allocates the 64MB
of low memory using 256KB contiguous chunks then...
> How are you determining amount of RAM used: xm info and looking at
> freemem?
> Or xm list and look at memory usage for dom0?
I''m using xenstat and looking at the difference between total memory
and
free memory (which is in turn obtained from avail_domheap_pages via
xc_physinfo... Hmm - does this mean the free memory in the xen heap zone
is not included? Seems like this is pretty small though (8MB total???)

Simon

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/2/07 14:15, "Graham, Simon" <Simon.Graham@stratus.com>
wrote:
> I''m using xenstat and looking at the difference between total
memory and
> free memory (which is in turn obtained from avail_domheap_pages via
> xc_physinfo... Hmm - does this mean the free memory in the xen heap zone
> is not included? Seems like this is pretty small though (8MB total???)
12MB on i386; 16MB on x86/64. Either way it''s much less than 100+ MB.

How much memory does the machine have?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > I''m using xenstat and looking at the difference between total
memory
> and
> > free memory (which is in turn obtained from avail_domheap_pages via
> > xc_physinfo... Hmm - does this mean the free memory in the xen heap
> zone
> > is not included? Seems like this is pretty small though (8MB
> total???)
> 
> 12MB on i386; 16MB on x86/64. Either way it''s much less than 100+
MB.
> 
> How much memory does the machine have?
Hmm - it does indeed seem to be somewhat dependent on the size of the
machine; I have a bunch of systems from 2GB to 16GB all of which are
running a 768MB Dom0 currently; here''s what I see for used memory
(total/free as reported by xenstat) for some of these when just Dom0 is
running:

. 2GB system; 794MB used
. 6GB system; 830MB used
. 8GB system; 840MB used
. 16GB system; 890MB used

Simon

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/2/07 19:14, "Graham, Simon" <Simon.Graham@stratus.com>
wrote:
> Hmm - it does indeed seem to be somewhat dependent on the size of the
> machine; I have a bunch of systems from 2GB to 16GB all of which are
> running a 768MB Dom0 currently; here''s what I see for used memory
> (total/free as reported by xenstat) for some of these when just Dom0 is
> running:
> 
> . 2GB system; 794MB used
> . 6GB system; 830MB used
> . 8GB system; 840MB used
> . 16GB system; 890MB used
Yes, there''s an overhead of 7MB per 1GB on i386, and 12MB per 1GB on
x86/64.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Hmm - it does indeed seem to be somewhat dependent on the size of
the
> > machine; I have a bunch of systems from 2GB to 16GB all of which are
> > running a 768MB Dom0 currently; here''s what I see for used
memory
> > (total/free as reported by xenstat) for some of these when just Dom0
> is
> > running:
> >
> > . 2GB system; 794MB used
> > . 6GB system; 830MB used
> > . 8GB system; 840MB used
> > . 16GB system; 890MB used
> 
> Yes, there''s an overhead of 7MB per 1GB on i386, and 12MB per 1GB
on
> x86/64.
What is the overhead for and where would I find it allocated?

Thanks,
Simon

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/2/07 19:26, "Graham, Simon" <Simon.Graham@stratus.com>
wrote:
>> Yes, there''s an overhead of 7MB per 1GB on i386, and 12MB per
1GB on
>> x86/64.
> 
> What is the overhead for and where would I find it allocated?
24 bytes per ''struct page_info'' and 4 bytes per
machine_to_phys array entry.
The former is allocated at start-of-day near the top of arch/x86/mm.c. The
array of page_info structs is pointed at by ''frame_table''.

The overheads on x86/64 are 40 bytes per ''struct page_info''
and 8 bytes per
machine_to_phys array entry.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>  On 24/2/07 19:14, "Graham, Simon"
<Simon.Graham@stratus.com> wrote:
> 
> > Hmm - it does indeed seem to be somewhat dependent on the size of the
> > machine; I have a bunch of systems from 2GB to 16GB all of which are
> > running a 768MB Dom0 currently; here''s what I see for used
memory
> > (total/free as reported by xenstat) for some of these when just Dom0
is
> > running:
> > 
> > . 2GB system; 794MB used
> > . 6GB system; 830MB used
> > . 8GB system; 840MB used
> > . 16GB system; 890MB used
> 
>  Yes, there''s an overhead of 7MB per 1GB on i386, and 12MB per 1GB
>  on x86/64.
Some time ago (xen 3.0.2 I think) I measured the overhead on i386 to be

  12 MB per domain
  plus 6MB per GB host RAM

I guess it has gone up since the above figures fit the 7MB amount much
better
>>> for total in (2, 6, 8, 16): print
"Total",total,"Calculated used",total*7+12+768
... 
Total 2 Calculated used 794
Total 6 Calculated used 822
Total 8 Calculated used 836
Total 16 Calculated used 892

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/2/07 12:41, "Nick Craig-Wood" <nick@craig-wood.com> wrote:
>>  Yes, there''s an overhead of 7MB per 1GB on i386, and 12MB per
1GB
>>  on x86/64.
> 
> Some time ago (xen 3.0.2 I think) I measured the overhead on i386 to be
> 
>   12 MB per domain
Do you mean 12MB static overhead for Xen itself? There is no 12MB overhead
per guest, unless you are talking about HVM guests in which case there is
some shadow-pagetable overhead (I''m not sure it''s as much as
12MB though).

 --Keir
>   plus 6MB per GB host RAM
> 
> I guess it has gone up since the above figures fit the 7MB amount much
> better
> 
>>>> for total in (2, 6, 8, 16): print
"Total",total,"Calculated
>>>> used",total*7+12+768
> ... 
> Total 2 Calculated used 794
> Total 6 Calculated used 822
> Total 8 Calculated used 836
> Total 16 Calculated used 892

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser <keir@xensource.com> wrote:
>  On 26/2/07 12:41, "Nick Craig-Wood" <nick@craig-wood.com>
wrote:
> >>  Yes, there''s an overhead of 7MB per 1GB on i386, and
12MB per 1GB
> >>  on x86/64.
> > 
> > Some time ago (xen 3.0.2 I think) I measured the overhead on i386 to
be
> > 
> >   12 MB per domain
> 
>  Do you mean 12MB static overhead for Xen itself? There is no 12MB overhead
>  per guest, unless you are talking about HVM guests in which case there is
>  some shadow-pagetable overhead (I''m not sure it''s as
much as 12MB
>  though).
Yes 12 MB for Xen is what I meant, not per domain - sorry!
> >   plus 6MB per GB host RAM
> > 
> > I guess it has gone up since the above figures fit the 7MB amount much
> > better
-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel