Xen devel - Nov 2007 - eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

>> We saw this issue on our boxes too.
>>
http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00479.html
>> I am trying to figure out how to write the copy-to-low-memory path.
>> Keir, could you give me some suggestions?
>
>In gnttab_transfer(), if the foreign domain (e) is 32-on-64 and the page
>being stolen from the local domain (d) is above 166GB then allocate anothr
>domheap page for e, copy the stolen page contents to it. Then free the
>stolen page and the new page takes its place.
I think page allocation in this path isn''t nice, at least not without
success
guarantee (not the least because because netback doesn''t check return
values). I would therefore rather see a solution in placing the burden of
ensuring accessibility on the producer (netback) of the page, and fail the
transfer if the destination domain can''t access the page (whether to be
nice and try an allocate-and-copy operation here is a secondary thing).

Netback would then need to determine the address size of netfront''s
domain
(just like blkback and blktap do, except that HVM domains should also be
treated as not requiring address restriction), and have two pools of pages
for use in transfers - one unrestricted and one limited to 37 address bits (the
two could be folded for resource efficiency if the machine has less than
128G). Besides that, netback would also start checking return values of the
multicall pieces.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 27/11/07 08:43, "Jan Beulich" <jbeulich@novell.com> wrote:
> I think page allocation in this path isn''t nice, at least not
without success
> guarantee (not the least because because netback doesn''t check
return
> values). I would therefore rather see a solution in placing the burden of
> ensuring accessibility on the producer (netback) of the page, and fail the
> transfer if the destination domain can''t access the page (whether
to be
> nice and try an allocate-and-copy operation here is a secondary thing).
> 
> Netback would then need to determine the address size of
netfront''s domain
> (just like blkback and blktap do, except that HVM domains should also be
> treated as not requiring address restriction), and have two pools of pages
> for use in transfers - one unrestricted and one limited to 37 address bits
> (the
> two could be folded for resource efficiency if the machine has less than
> 128G). Besides that, netback would also start checking return values of the
> multicall pieces.
I don''t get how your netback approach works. The pages we transfer do
not
originate from netback, so it has little control over them. And, even if it
did, when we allocate pages for network receive we do not know which
domain''s packet will end up in each buffer.

Personally I think doing it in Xen is perfectly good enough for supporting
this very out-of-date network receive mechanism.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 09:56
>>>
>On 27/11/07 08:43, "Jan Beulich" <jbeulich@novell.com>
wrote:
>
>> I think page allocation in this path isn''t nice, at least not
without success
>> guarantee (not the least because because netback doesn''t check
return
>> values). I would therefore rather see a solution in placing the burden
of
>> ensuring accessibility on the producer (netback) of the page, and fail
the
>> transfer if the destination domain can''t access the page
(whether to be
>> nice and try an allocate-and-copy operation here is a secondary thing).
>> 
>> Netback would then need to determine the address size of
netfront''s domain
>> (just like blkback and blktap do, except that HVM domains should also
be
>> treated as not requiring address restriction), and have two pools of
pages
>> for use in transfers - one unrestricted and one limited to 37 address
bits
>> (the
>> two could be folded for resource efficiency if the machine has less
than
>> 128G). Besides that, netback would also start checking return values of
the
>> multicall pieces.
>
>I don''t get how your netback approach works. The pages we transfer
do not
>originate from netback, so it has little control over them. And, even if it
>did, when we allocate pages for network receive we do not know which
>domain''s packet will end up in each buffer.
Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
netback could take care of this by doing the copying there, as at that point i
already knows the destination domain.
>Personally I think doing it in Xen is perfectly good enough for supporting
>this very out-of-date network receive mechanism.
I''m not just concerned about netback here. The interface exists, and
other
users might show up and/or exist already. Whether it would be acceptable
for them to do allocation and copying is unknown. You''d therefore
either
need a way to prevent future users of the transfer mechanism, or set proper
requirements on its use. I think that placing extra requirements on the user
of the interface is better than introducing extra (possibly hard to reproduce/
recognize/debug) possibilities of failure.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>> I don''t get how your netback approach works. The pages we
transfer do not
>> originate from netback, so it has little control over them. And, even
if it
>> did, when we allocate pages for network receive we do not know which
>> domain''s packet will end up in each buffer.
> 
> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
> netback could take care of this by doing the copying there, as at that
point i
> already knows the destination domain.
You may not know constraints on that domain''s max_mfn though. We could
add
an interface to Xen to interrogate that, but generally it''s not
something we
probably want to expose outside of Xen and the guest itself.
>> Personally I think doing it in Xen is perfectly good enough for
supporting
>> this very out-of-date network receive mechanism.
> 
> I''m not just concerned about netback here. The interface exists,
and other
> users might show up and/or exist already. Whether it would be acceptable
> for them to do allocation and copying is unknown. You''d therefore
either
> need a way to prevent future users of the transfer mechanism, or set proper
> requirements on its use. I think that placing extra requirements on the
user
> of the interface is better than introducing extra (possibly hard to
reproduce/
> recognize/debug) possibilities of failure.
The interface is obsolete.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21
>>>
>On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com>
wrote:
>
>>> I don''t get how your netback approach works. The pages we
transfer do not
>>> originate from netback, so it has little control over them. And,
even if it
>>> did, when we allocate pages for network receive we do not know
which
>>> domain''s packet will end up in each buffer.
>> 
>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
Nevertheless
>> netback could take care of this by doing the copying there, as at that
point i
>> already knows the destination domain.
>
>You may not know constraints on that domain''s max_mfn though. We
could add
>an interface to Xen to interrogate that, but generally it''s not
something we
>probably want to expose outside of Xen and the guest itself.
What constraints other than the guest''s address size influence its
max_mfn?
Of course, if there''s anything beyond the address size, then having a
way to
obtain the constraint explicitly would be desirable. But otherwise (and as
fallback) using 37 bits (128G) seems quite reasonable.
>>> Personally I think doing it in Xen is perfectly good enough for
supporting
>>> this very out-of-date network receive mechanism.
>> 
>> I''m not just concerned about netback here. The interface
exists, and other
>> users might show up and/or exist already. Whether it would be
acceptable
>> for them to do allocation and copying is unknown. You''d
therefore either
>> need a way to prevent future users of the transfer mechanism, or set
proper
>> requirements on its use. I think that placing extra requirements on the
user
>> of the interface is better than introducing extra (possibly hard to
reproduce/
>> recognize/debug) possibilities of failure.
>
>The interface is obsolete.
Then it should be clearly indicated as such, e.g. by a mechanism similar to
deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in netback
should
then probably be conditional upon an extra config option, which could at once
be used to provide a note to Xen that the feature isn''t being used so
that the
function could return -ENOSYS and the clipping could be avoided/reverted.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Has there been any more thought on this subject? The discussion seems to 
have stalled, and we''re hoping to find a way past this 166G limit...

Jan Beulich wrote:
>>>>Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21
>>>
>>>>        
>>>>
>>On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com>
wrote:
>>
>>    
>>
>>>>I don''t get how your netback approach works. The pages
we transfer do not
>>>>originate from netback, so it has little control over them. And,
even if it
>>>>did, when we allocate pages for network receive we do not know
which
>>>>domain''s packet will end up in each buffer.
>>>>        
>>>>
>>>Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
Nevertheless
>>>netback could take care of this by doing the copying there, as at
that point i
>>>already knows the destination domain.
>>>      
>>>
>>You may not know constraints on that domain''s max_mfn though.
We could add
>>an interface to Xen to interrogate that, but generally it''s not
something we
>>probably want to expose outside of Xen and the guest itself.
>>    
>>
>
>What constraints other than the guest''s address size influence its
max_mfn?
>Of course, if there''s anything beyond the address size, then having
a way to
>obtain the constraint explicitly would be desirable. But otherwise (and as
>fallback) using 37 bits (128G) seems quite reasonable.
>
>  
>
>>>>Personally I think doing it in Xen is perfectly good enough for
supporting
>>>>this very out-of-date network receive mechanism.
>>>>        
>>>>
>>>I''m not just concerned about netback here. The interface
exists, and other
>>>users might show up and/or exist already. Whether it would be
acceptable
>>>for them to do allocation and copying is unknown. You''d
therefore either
>>>need a way to prevent future users of the transfer mechanism, or set
proper
>>>requirements on its use. I think that placing extra requirements on
the user
>>>of the interface is better than introducing extra (possibly hard to
reproduce/
>>>recognize/debug) possibilities of failure.
>>>      
>>>
>>The interface is obsolete.
>>    
>>
>
>Then it should be clearly indicated as such, e.g. by a mechanism similar to
>deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in netback
should
>then probably be conditional upon an extra config option, which could at
once
>be used to provide a note to Xen that the feature isn''t being used
so that the
>function could return -ENOSYS and the clipping could be avoided/reverted.
>
>Jan
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>

-- 
Elizabeth Kon (Beth)
IBM Linux Technology Center
Open Hypervisor Team
email: eak@us.ibm.com


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I''ll get something in for 3.2.0.

 -- Keir


On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:
> Has there been any more thought on this subject? The discussion seems to
> have stalled, and we''re hoping to find a way past this 166G
limit...
> 
> Jan Beulich wrote:
> 
>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21
>>>
>>>>>        
>>>>> 
>>> On 27/11/07 09:00, "Jan Beulich"
<jbeulich@novell.com> wrote:
>>> 
>>>    
>>> 
>>>>> I don''t get how your netback approach works. The
pages we transfer do not
>>>>> originate from netback, so it has little control over them.
And, even if
>>>>> it
>>>>> did, when we allocate pages for network receive we do not
know which
>>>>> domain''s packet will end up in each buffer.
>>>>>        
>>>>> 
>>>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
Nevertheless
>>>> netback could take care of this by doing the copying there, as
at that
>>>> point i
>>>> already knows the destination domain.
>>>>      
>>>> 
>>> You may not know constraints on that domain''s max_mfn
though. We could add
>>> an interface to Xen to interrogate that, but generally
it''s not something we
>>> probably want to expose outside of Xen and the guest itself.
>>>    
>>> 
>> 
>> What constraints other than the guest''s address size influence
its max_mfn?
>> Of course, if there''s anything beyond the address size, then
having a way to
>> obtain the constraint explicitly would be desirable. But otherwise (and
as
>> fallback) using 37 bits (128G) seems quite reasonable.
>> 
>>  
>> 
>>>>> Personally I think doing it in Xen is perfectly good enough
for supporting
>>>>> this very out-of-date network receive mechanism.
>>>>>        
>>>>> 
>>>> I''m not just concerned about netback here. The
interface exists, and other
>>>> users might show up and/or exist already. Whether it would be
acceptable
>>>> for them to do allocation and copying is unknown.
You''d therefore either
>>>> need a way to prevent future users of the transfer mechanism,
or set proper
>>>> requirements on its use. I think that placing extra
requirements on the
>>>> user
>>>> of the interface is better than introducing extra (possibly
hard to
>>>> reproduce/
>>>> recognize/debug) possibilities of failure.
>>>>      
>>>> 
>>> The interface is obsolete.
>>>    
>>> 
>> 
>> Then it should be clearly indicated as such, e.g. by a mechanism
similar to
>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
netback
>> should
>> then probably be conditional upon an extra config option, which could
at once
>> be used to provide a note to Xen that the feature isn''t being
used so that
>> the
>> function could return -ENOSYS and the clipping could be
avoided/reverted.
>> 
>> Jan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>  
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Beth,
I am working on it.
I spoke to Keir about it at the Xen summit.
I hope to have a patch that will fix the issue and have the high-to-low memory
copy working soon.

Thanks
Raj
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of beth kon
>Sent: Monday, December 03, 2007 2:50 PM
>Cc: xen-devel@lists.xensource.com
>Subject: Re: eliminating 166G limit (was Re: [Xen-devel]
>Problem with nr_nodeson large memory NUMA machine)
>
>Has there been any more thought on this subject? The
>discussion seems to have stalled, and we''re hoping to find a
>way past this 166G limit...
>
>Jan Beulich wrote:
>
>>>>>Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21
>>>
>>>>>
>>>>>
>>>On 27/11/07 09:00, "Jan Beulich"
<jbeulich@novell.com> wrote:
>>>
>>>
>>>
>>>>>I don''t get how your netback approach works. The
pages we transfer
>>>>>do not originate from netback, so it has little control over
them.
>>>>>And, even if it did, when we allocate pages for network
receive we
>>>>>do not know which domain''s packet will end up in
each buffer.
>>>>>
>>>>>
>>>>Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
>>>>Nevertheless netback could take care of this by doing the
copying
>>>>there, as at that point i already knows the destination domain.
>>>>
>>>>
>>>You may not know constraints on that domain''s max_mfn
>though. We could
>>>add an interface to Xen to interrogate that, but generally
it''s not
>>>something we probably want to expose outside of Xen and the
>guest itself.
>>>
>>>
>>
>>What constraints other than the guest''s address size
>influence its max_mfn?
>>Of course, if there''s anything beyond the address size, then
having a
>>way to obtain the constraint explicitly would be desirable. But
>>otherwise (and as
>>fallback) using 37 bits (128G) seems quite reasonable.
>>
>>
>>
>>>>>Personally I think doing it in Xen is perfectly good enough
for
>>>>>supporting this very out-of-date network receive mechanism.
>>>>>
>>>>>
>>>>I''m not just concerned about netback here. The
interface
>exists, and
>>>>other users might show up and/or exist already. Whether it would
be
>>>>acceptable for them to do allocation and copying is unknown.
You''d
>>>>therefore either need a way to prevent future users of the
transfer
>>>>mechanism, or set proper requirements on its use. I think that
>>>>placing extra requirements on the user of the interface is
better
>>>>than introducing extra (possibly hard to reproduce/
>>>>recognize/debug) possibilities of failure.
>>>>
>>>>
>>>The interface is obsolete.
>>>
>>>
>>
>>Then it should be clearly indicated as such, e.g. by a mechanism
>>similar to
>>deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
>>netback should then probably be conditional upon an extra config
>>option, which could at once be used to provide a note to Xen that the
>>feature isn''t being used so that the function could return
>-ENOSYS and the clipping could be avoided/reverted.
>>
>>Jan
>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>
>
>
>--
>Elizabeth Kon (Beth)
>IBM Linux Technology Center
>Open Hypervisor Team
>email: eak@us.ibm.com
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Try xen-unstable changeset 16548.

 -- Keir

On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:
> Has there been any more thought on this subject? The discussion seems to
> have stalled, and we''re hoping to find a way past this 166G
limit...
> 
> Jan Beulich wrote:
> 
>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21
>>>
>>>>>        
>>>>> 
>>> On 27/11/07 09:00, "Jan Beulich"
<jbeulich@novell.com> wrote:
>>> 
>>>    
>>> 
>>>>> I don''t get how your netback approach works. The
pages we transfer do not
>>>>> originate from netback, so it has little control over them.
And, even if
>>>>> it
>>>>> did, when we allocate pages for network receive we do not
know which
>>>>> domain''s packet will end up in each buffer.
>>>>>        
>>>>> 
>>>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
Nevertheless
>>>> netback could take care of this by doing the copying there, as
at that
>>>> point i
>>>> already knows the destination domain.
>>>>      
>>>> 
>>> You may not know constraints on that domain''s max_mfn
though. We could add
>>> an interface to Xen to interrogate that, but generally
it''s not something we
>>> probably want to expose outside of Xen and the guest itself.
>>>    
>>> 
>> 
>> What constraints other than the guest''s address size influence
its max_mfn?
>> Of course, if there''s anything beyond the address size, then
having a way to
>> obtain the constraint explicitly would be desirable. But otherwise (and
as
>> fallback) using 37 bits (128G) seems quite reasonable.
>> 
>>  
>> 
>>>>> Personally I think doing it in Xen is perfectly good enough
for supporting
>>>>> this very out-of-date network receive mechanism.
>>>>>        
>>>>> 
>>>> I''m not just concerned about netback here. The
interface exists, and other
>>>> users might show up and/or exist already. Whether it would be
acceptable
>>>> for them to do allocation and copying is unknown.
You''d therefore either
>>>> need a way to prevent future users of the transfer mechanism,
or set proper
>>>> requirements on its use. I think that placing extra
requirements on the
>>>> user
>>>> of the interface is better than introducing extra (possibly
hard to
>>>> reproduce/
>>>> recognize/debug) possibilities of failure.
>>>>      
>>>> 
>>> The interface is obsolete.
>>>    
>>> 
>> 
>> Then it should be clearly indicated as such, e.g. by a mechanism
similar to
>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
netback
>> should
>> then probably be conditional upon an extra config option, which could
at once
>> be used to provide a note to Xen that the feature isn''t being
used so that
>> the
>> function could return -ENOSYS and the clipping could be
avoided/reverted.
>> 
>> Jan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>  
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I tested this changeset on Unisys ES7000 with 256G ram and 64 processors
and it works:

xentop - 06:30:59   Xen 3.2-unstable
1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 268172340k total, 7669456k used, 260502884k free    CPUs: 64 @
3400MHz

I will be running our full test suite on this configuration today.

Thanks,
Sue Krysan
Linux Systems Group
Unisys Corporation
 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
Sent: Thursday, December 06, 2007 8:40 AM
To: eak@us.ibm.com
Cc: xen-devel@lists.xensource.com
Subject: Re: eliminating 166G limit (was Re: [Xen-devel] Problem
withnr_nodes on large memory NUMA machine)

Try xen-unstable changeset 16548.

 -- Keir

On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:
> Has there been any more thought on this subject? The discussion seems
to
> have stalled, and we''re hoping to find a way past this 166G
limit...
> 
> Jan Beulich wrote:
> 
>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21
>>>
>>>>>        
>>>>> 
>>> On 27/11/07 09:00, "Jan Beulich"
<jbeulich@novell.com> wrote:
>>> 
>>>    
>>> 
>>>>> I don''t get how your netback approach works. The
pages we transfer
do not
>>>>> originate from netback, so it has little control over them.
And,
even if
>>>>> it
>>>>> did, when we allocate pages for network receive we do not
know
which
>>>>> domain''s packet will end up in each buffer.
>>>>>        
>>>>> 
>>>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
Nevertheless
>>>> netback could take care of this by doing the copying there, as
at
that
>>>> point i
>>>> already knows the destination domain.
>>>>      
>>>> 
>>> You may not know constraints on that domain''s max_mfn
though. We
could add
>>> an interface to Xen to interrogate that, but generally
it''s not
something we
>>> probably want to expose outside of Xen and the guest itself.
>>>    
>>> 
>> 
>> What constraints other than the guest''s address size influence
its
max_mfn?
>> Of course, if there''s anything beyond the address size, then
having a
way to
>> obtain the constraint explicitly would be desirable. But otherwise
(and as
>> fallback) using 37 bits (128G) seems quite reasonable.
>> 
>>  
>> 
>>>>> Personally I think doing it in Xen is perfectly good enough
for
supporting
>>>>> this very out-of-date network receive mechanism.
>>>>>        
>>>>> 
>>>> I''m not just concerned about netback here. The
interface exists,
and other
>>>> users might show up and/or exist already. Whether it would be
acceptable
>>>> for them to do allocation and copying is unknown.
You''d therefore
either
>>>> need a way to prevent future users of the transfer mechanism,
or
set proper
>>>> requirements on its use. I think that placing extra
requirements on
the
>>>> user
>>>> of the interface is better than introducing extra (possibly
hard to
>>>> reproduce/
>>>> recognize/debug) possibilities of failure.
>>>>      
>>>> 
>>> The interface is obsolete.
>>>    
>>> 
>> 
>> Then it should be clearly indicated as such, e.g. by a mechanism
similar to
>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
netback
>> should
>> then probably be conditional upon an extra config option, which could
at once
>> be used to provide a note to Xen that the feature isn''t being
used so
that
>> the
>> function could return -ENOSYS and the clipping could be
avoided/reverted.
>> 
>> Jan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>  
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,
We were able to bring up a 250G machine with this changeset.
Raj
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
>Sent: Thursday, December 06, 2007 8:40 AM
>To: eak@us.ibm.com
>Cc: xen-devel@lists.xensource.com
>Subject: Re: eliminating 166G limit (was Re: [Xen-devel]
>Problem withnr_nodes on large memory NUMA machine)
>
>Try xen-unstable changeset 16548.
>
> -- Keir
>
>On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:
>
>> Has there been any more thought on this subject? The
>discussion seems
>> to have stalled, and we''re hoping to find a way past this
>166G limit...
>>
>> Jan Beulich wrote:
>>
>>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07
10:21 >>>
>>>>>>
>>>>>>
>>>> On 27/11/07 09:00, "Jan Beulich"
<jbeulich@novell.com> wrote:
>>>>
>>>>
>>>>
>>>>>> I don''t get how your netback approach works.
The pages
>we transfer
>>>>>> do not originate from netback, so it has little control
>over them.
>>>>>> And, even if it did, when we allocate pages for network
>receive we
>>>>>> do not know which domain''s packet will end up
in each buffer.
>>>>>>
>>>>>>
>>>>> Oh, right, I mixed up old_mfn and new_mfn in
netbk_gop_frag().
>>>>> Nevertheless netback could take care of this by doing the
copying
>>>>> there, as at that point i already knows the destination
domain.
>>>>>
>>>>>
>>>> You may not know constraints on that domain''s max_mfn
though. We
>>>> could add an interface to Xen to interrogate that, but
generally
>>>> it''s not something we probably want to expose outside
of
>Xen and the guest itself.
>>>>
>>>>
>>>
>>> What constraints other than the guest''s address size
>influence its max_mfn?
>>> Of course, if there''s anything beyond the address size,
>then having a
>>> way to obtain the constraint explicitly would be desirable. But
>>> otherwise (and as
>>> fallback) using 37 bits (128G) seems quite reasonable.
>>>
>>>
>>>
>>>>>> Personally I think doing it in Xen is perfectly good
enough for
>>>>>> supporting this very out-of-date network receive
mechanism.
>>>>>>
>>>>>>
>>>>> I''m not just concerned about netback here. The
interface exists,
>>>>> and other users might show up and/or exist already. Whether
it
>>>>> would be acceptable for them to do allocation and copying
is
>>>>> unknown. You''d therefore either need a way to
prevent
>future users
>>>>> of the transfer mechanism, or set proper requirements on
>its use. I
>>>>> think that placing extra requirements on the user of the
>interface
>>>>> is better than introducing extra (possibly hard to
reproduce/
>>>>> recognize/debug) possibilities of failure.
>>>>>
>>>>>
>>>> The interface is obsolete.
>>>>
>>>>
>>>
>>> Then it should be clearly indicated as such, e.g. by a mechanism
>>> similar to
>>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
>>> netback should then probably be conditional upon an extra config
>>> option, which could at once be used to provide a note to
>Xen that the
>>> feature isn''t being used so that the function could return
-ENOSYS
>>> and the clipping could be avoided/reverted.
>>>
>>> Jan
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>
>
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Dec 7, 2007 at 6:20 AM, Krysan, Susan <KRYSANS@unisys.com>
wrote:
> I tested this changeset on Unisys ES7000 with 256G ram and 64 processors
> and it works:
>
> xentop - 06:30:59   Xen 3.2-unstable
> 1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
> shutdown
> Mem: 268172340k total, 7669456k used, 260502884k free    CPUs: 64 @
> 3400MHz
>
> I will be running our full test suite on this configuration today.
>
> Thanks,
> Sue Krysan
> Linux Systems Group
> Unisys Corporation
>
>
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
> Sent: Thursday, December 06, 2007 8:40 AM
> To: eak@us.ibm.com
> Cc: xen-devel@lists.xensource.com
> Subject: Re: eliminating 166G limit (was Re: [Xen-devel] Problem
> withnr_nodes on large memory NUMA machine)
>
> Try xen-unstable changeset 16548.
Susan,

how about TOM? i mean MMIO size...

my system with 256g/32cores, and 0xe0000000 - 0x100000000 is mmio
mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
TOM: 00000000e0000000 aka 3584M
TOM2: 0000004020000000 aka 262656M

only can work with mem=200g

mem=224g, xen kernel will say alloc_bootmem_low can not get range..

xen kernel is from

http://people.redhat.com/dzickus/el5/92.el5/x86_64/

YH

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel