Xen devel - Jun 2005 - comment request: dom0 dma on large memory systems

purpose:

The patches i''m working on is taking more work then i was expecting;
this
email''s purpose is to get confirmation that the solution i''m
working on
is the preferred one.


problem:

On x86_64 with 6gig ram, dom0''s initial allocation is from memory
above the pci hole (referred to as "high memory" in this email) if
dom0_mem is set to 2g or higher. The only problem is that most io/dma
devices (non-dac) can only dma to the first 32bits worth of machine
addresses--thus for some configurations, dom0 has no memory which is
dma-able.

This is realized, in my experience, with dom0 unable to find any
partitions on scan and panicking because it can''t mount root.


proposed solution to xen:

The attached patch is a (non-functional) prototype of my proposed
solution to the xen. The idea is to add a third MEMZONE for
dma''able memory, change the alloc_domheap_pages() interface so the
caller can request only dma memory if needed. Finally, internal to
common/page_alloc.c, split memory between MEMZONE_DOM and MEMZONE_DMADOM
based on the address.


proposed solution to dom0:

Dom0 might have memory allocated to it from both high and low memory.
The easiest solution would probably be to scan for and preallocate a
chunk of memory that will work for dma, something like what pci-gart.c
does. (or work on letting pci-gart.c work under xen)


sRp

-- 
Scott Parish
Signed-off-by: srparish@us.ibm.com


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yes, something like this is obviously required, and introducing a 
concept of zones seems a sensible way to go. I haven''t looked at your 
patch in great detail but it looks along the right lines. :-)

  -- Keir

On 3 Jun 2005, at 08:35, Scott Parish wrote:
> The attached patch is a (non-functional) prototype of my proposed
> solution to the xen. The idea is to add a third MEMZONE for
> dma''able memory, change the alloc_domheap_pages() interface so the
> caller can request only dma memory if needed. Finally, internal to
> common/page_alloc.c, split memory between MEMZONE_DOM and 
> MEMZONE_DMADOM
> based on the address.
>
>
> proposed solution to dom0:
>
> Dom0 might have memory allocated to it from both high and low memory.
> The easiest solution would probably be to scan for and preallocate a
> chunk of memory that will work for dma, something like what pci-gart.c
> does. (or work on letting pci-gart.c work under xen)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Yes, something like this is obviously required, and 
> introducing a concept of zones seems a sensible way to go. I 
> haven''t looked at your patch in great detail but it looks 
> along the right lines. :-)
While working up the patch it''s worth bearing in mind the closely
related NUMA issue: We''ll want a separate heap for each node''s
memory,
and a way of requesting a prefered node in the alloc function, falling
back to a ''close'' node if req''d.

It might even be worth adding in a zone for ISA DMA memory, so we could
get some old PCMCIA cards working, though probably not worth it.

Ian

 
>   -- Keir
> 
> On 3 Jun 2005, at 08:35, Scott Parish wrote:
> 
> > The attached patch is a (non-functional) prototype of my proposed 
> > solution to the xen. The idea is to add a third MEMZONE for 
> dma''able 
> > memory, change the alloc_domheap_pages() interface so the 
> caller can 
> > request only dma memory if needed. Finally, internal to 
> > common/page_alloc.c, split memory between MEMZONE_DOM and 
> > MEMZONE_DMADOM based on the address.
> >
> >
> > proposed solution to dom0:
> >
> > Dom0 might have memory allocated to it from both high and 
> low memory.
> > The easiest solution would probably be to scan for and 
> preallocate a 
> > chunk of memory that will work for dma, something like what 
> pci-gart.c 
> > does. (or work on letting pci-gart.c work under xen)
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Scott
Parish
>Sent: Friday, June 03, 2005 3:35 PM
>
>On x86_64 with 6gig ram, dom0''s initial allocation is from memory
>above the pci hole (referred to as "high memory" in this email) if
>dom0_mem is set to 2g or higher. The only problem is that most io/dma
>devices (non-dac) can only dma to the first 32bits worth of machine
>addresses--thus for some configurations, dom0 has no memory which is
>dma-able.
IIRC, 2 or 3 months ago, Keir said that default memory allocation for
Dom0 is all available memory. And then CP has to decrease by balloon
interface before creating other domains. If this still holds true, I''m
not sure whether above problem still exists, since all avail memory
including both <4G and >4G belonging to Dom0 then. (XEN itself only
consumes a small trunk). However after looking at your patch and then
the source, it seems that only the max available order, meaning must be
continuous, is allocated to Dom0 currently. So did I misunderstand this
concept? If it really only means maximum continuous trunk, then you
patch definitely shoots straight on the real problem on all 64bit
platform. ;-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, Jun 04, 2005 at 11:48:16AM +0800, Tian, Kevin wrote:
> ----Original Message-----
> >From: xen-devel-bounces@lists.xensource.com
> >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Scott
> Parish
> >Sent: Friday, June 03, 2005 3:35 PM
> >
> >On x86_64 with 6gig ram, dom0''s initial allocation is from
memory
> >above the pci hole (referred to as "high memory" in this
email) if
> >dom0_mem is set to 2g or higher. The only problem is that most io/dma
> >devices (non-dac) can only dma to the first 32bits worth of machine
> >addresses--thus for some configurations, dom0 has no memory which is
> >dma-able.
> 
> IIRC, 2 or 3 months ago, Keir said that default memory allocation for
> Dom0 is all available memory. And then CP has to decrease by balloon
> interface before creating other domains. If this still holds true,
I''m
> not sure whether above problem still exists, since all avail memory
> including both <4G and >4G belonging to Dom0 then. (XEN itself only
> consumes a small trunk). However after looking at your patch and then
> the source, it seems that only the max available order, meaning must be
> continuous, is allocated to Dom0 currently. So did I misunderstand this
> concept? If it really only means maximum continuous trunk, then you
> patch definitely shoots straight on the real problem on all 64bit
> platform. ;-)
Right, there are several hacks around this problem, a couple i''ve
thought of are:

  + enforce dom0 take all memory
  + drop the max order size for MEMZONEs to 18 (in which case
    alloc_largest should always allocate from the lower memory)
  + prealloc X amount of low memory (128M for instance) and add
    it into the dom0 allocation

You nailed it when you mentioned driver domains (next email); the long
term goal is to make sure we''re able to support them and hopefully
avoid
the hogging of that memory unnecessarily for non-dma uses. Thanks for
noticing ;^)  

(i was also glad Ian brought up numa, i had forgotten about it and this
is probably a good time to think about that while i''m tearing up this
code)

sRp

-- 
Scott Parish
Signed-off-by: srparish@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Aside from following question (just about the concept), actually to have
DMA sense within XEN is really necessary especially for a driver domain
N which has direct physical device access. Currently that domain N
invokes generic increase_reservation interface to get machine contiguous
pages. However that generic interface has no sense about the 4G
limitation for old DMA controller on 64bit platform. So such patch is
really necessary not only for domain0, but also all domains which has
direct relationship with physical devices. Also that hypercall interface
may also be changed a bit to contain flag whether DMA-able.

Thanks,
Kevin
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin
>Sent: Saturday, June 04, 2005 11:48 AM
>To: Scott Parish; xen-devel@lists.xensource.com
>Subject: RE: [Xen-devel] comment request: dom0 dma on large memory
>systems
>
>----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Scott
>Parish
>>Sent: Friday, June 03, 2005 3:35 PM
>>
>>On x86_64 with 6gig ram, dom0''s initial allocation is from
memory
>>above the pci hole (referred to as "high memory" in this
email) if
>>dom0_mem is set to 2g or higher. The only problem is that most io/dma
>>devices (non-dac) can only dma to the first 32bits worth of machine
>>addresses--thus for some configurations, dom0 has no memory which is
>>dma-able.
>
>IIRC, 2 or 3 months ago, Keir said that default memory allocation for
>Dom0 is all available memory. And then CP has to decrease by balloon
>interface before creating other domains. If this still holds true,
I''m
>not sure whether above problem still exists, since all avail memory
>including both <4G and >4G belonging to Dom0 then. (XEN itself only
>consumes a small trunk). However after looking at your patch and then
>the source, it seems that only the max available order, meaning must be
>continuous, is allocated to Dom0 currently. So did I misunderstand this
>concept? If it really only means maximum continuous trunk, then you
>patch definitely shoots straight on the real problem on all 64bit
>platform. ;-)
>
>Thanks,
>Kevin
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>-----Original Message-----
>From: Scott Parish [mailto:srparish@us.ibm.com]
>Sent: Saturday, June 04, 2005 12:00 PM
>To: Tian, Kevin
>
>>
>> IIRC, 2 or 3 months ago, Keir said that default memory allocation for
>> Dom0 is all available memory. And then CP has to decrease by balloon
>> interface before creating other domains. If this still holds true,
I''m
>> not sure whether above problem still exists, since all avail memory
>> including both <4G and >4G belonging to Dom0 then. (XEN itself
only
>> consumes a small trunk). However after looking at your patch and then
>> the source, it seems that only the max available order, meaning must
be
>> continuous, is allocated to Dom0 currently. So did I misunderstand
this
>> concept? If it really only means maximum continuous trunk, then you
>> patch definitely shoots straight on the real problem on all 64bit
>> platform. ;-)
>
>Right, there are several hacks around this problem, a couple i''ve
>thought of are:
>
>  + enforce dom0 take all memory
Just a rough thought. If dom0 can take all the memory, one alternative
is to rely on dom0 to support DMA related allocation. At that time, dom0
can handle all internal requests itself, without HV''s intervention.
Then
a similar component like balloon driver resides within dom0, to handle
requests from other driver domains. A new event channel will be created
to pass zone information about request between Dom0 and DomN. Then when
driver domain wants to allocate DMA-able pages, the request will go to
dom0, instead of HV. Finally the balloon-like driver will allocate
DMA-able page from Dom0''s memory allocator, and then update driver
domain''s mapping table. Yes, this adds some overhead for more context
switch. But HV can pre-requested DMA pool from dom0, and then accelerate
the process.

NUMA is somehow different, and I have no clear picture whether this
direction applies to it yet. ;-P
>  + drop the max order size for MEMZONEs to 18 (in which case
>    alloc_largest should always allocate from the lower memory)
>  + prealloc X amount of low memory (128M for instance) and add
>    it into the dom0 allocation
IMO, to add zone info as your patch is better than simply hacks.
>
>You nailed it when you mentioned driver domains (next email); the long
>term goal is to make sure we''re able to support them and hopefully
avoid
>the hogging of that memory unnecessarily for non-dma uses. Thanks for
>noticing ;^)
>
>(i was also glad Ian brought up numa, i had forgotten about it and this
>is probably a good time to think about that while i''m tearing up
this
>code)
>
>sRp
Yep. ;v)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel