Xen devel - Dec 2005 - RE: Guest-visible phys2mach part of Xen arch-neutral API? was: Uses of &frame_table[xfn]

> > Yes, that's the current status: No way to see underlying 
> > machine address in other domains and thus no way for driver domains. 
> Itanium cluster (~1000 nodes). I'm not sure if it would be possible to
> use Xen on the cluster without driver domains though. Just food for
Just to clarify: This doesn't say that there can't be driver domains.
Driver domains would need to be implemented p==m, same as domain0, except
they would need to be given a different EFI memory map.  They have
not been implemented on Xen/ia64 because they have not yet re-appeared
in Xen/x86.

But I am intrigued by your statement... are you assuming that
Xen and domain0 are both single-system-image across the 1000+
nodes?  I think one of the difficulties with implementing driver
domains is that Xen (in the hypervisor) needs to discover and
somehow partition all the devices.  This could be a real challenge
for huge clusters.

Thanks,
Dan
> -----Original Message-----
> From: Kevin Fox [mailto:Kevin.Fox@pnl.gov] 
> Sent: Friday, December 30, 2005 9:35 AM
> To: Tian, Kevin
> Cc: Keir Fraser; Magenheimer, Dan (HP Labs Fort Collins); Xen 
> Mailing List
> Subject: RE: Guest-visible phys2mach part of Xen arch-neutral 
> API? was:[Xen-devel] Uses of &frame_table[xfn]
> 
> There has been some discussion about some day using Xen on our large
> Itanium cluster (~1000 nodes). I'm not sure if it would be possible to
> use Xen on the cluster without driver domains though. Just food for
> thought.
> 
> Kevin
> 
> PS. I'm fairly disappointed in the lack of focus that driver 
> domains are
> receiving in Xen. I use them on several machines (one in 
> production) and
> I think it counts among Xen's greatest features.
> 
> On Fri, 2005-12-30 at 10:16 +0800, Tian, Kevin wrote:
> > >From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> > >Sent: 2005年12月29日 21:48
> > >
> > >
> > >On 29 Dec 2005, at 01:59, Tian, Kevin wrote:
> > >
> > >> 	IMO, I see the phys2mach mapping as a basic 
> virtualization policy,
> > >> instead of an architecture specific requirement. After adding
> > >> phys2mach concept to XEN/IA64, we can reuse more common 
> code without
> > >> ifdef. Then correspondingly also need to add several 
> necessary changes
> > >> like x86: DMA, SWIOTLB, AGP, etc, to ensure legal machine
address
> > >> written into physical devices.
> > >
> > >This seems to make sense to me. How does ia64/xen work right now?
> > >Machine addresses visible to domain0 and full virtualisation of
> > >addresses exposed to other domains (with no way of seeing 
> underlying
> > >machine addresses)?
> > >
> > >  -- Keir
> > 
> > Yes, that's the current status: No way to see underlying 
> machine address in other domains and thus no way for driver domains.
> > 
> > Thanks,
> > Kevin
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 2005-12-30 at 11:14 -0800, Magenheimer, Dan (HP Labs Fort
Collins) wrote:
> > > Yes, that''s the current status: No way to see underlying
> > > machine address in other domains and thus no way for driver
domains.
> 
> > Itanium cluster (~1000 nodes). I''m not sure if it would be
possible to
> > use Xen on the cluster without driver domains though. Just food for
> 
> Just to clarify: This doesn''t say that there can''t be
driver domains.
> Driver domains would need to be implemented p==m, same as domain0, except
> they would need to be given a different EFI memory map.  They have
> not been implemented on Xen/ia64 because they have not yet re-appeared
> in Xen/x86.
I am not really familiar with the internals of Xen. Just thought that
driver domains were being brushed off as unimportant again. Sorry. We do
a lot of high performance computing here and without something like
driver domains, the performance hit can''t be justified.
> 
> But I am intrigued by your statement... are you assuming that
> Xen and domain0 are both single-system-image across the 1000+
> nodes?
That particular cluster is made up of dual processor HP RX2600''s. It
may
go to quad processors at some point. Each node has its own OS.

We''d really like to run two domU''s on each node, one for jobs
and
another for some services. The jobs need to be really close to the Elan4
cards since latency kills and the other domU needs to be close to disk.

Your right, having a Xen/Dom0 across a box that large would be
difficult. Largest box I know of that might be able to do that would be
NASA''s alticies (What the heck is the plural of Altix anyway? :) which
are 512 processors. I have heard rumor of larger alticies in the pipe.
An Altix does have some hardware to allow slicing up a physical box into
a few smaller boxes, but it is rather basic. Having something like Xen
run on top of it could be useful.
>   I think one of the difficulties with implementing driver
> domains is that Xen (in the hypervisor) needs to discover and
> somehow partition all the devices.  This could be a real challenge
> for huge clusters.
Could not Dom0 detect the hardware and then pci hot remove it or
something and hand a reference over to Xen for handing to a DomU?

Linux already scales to ccNuma''s of 512 processors. I believe SGI even
got the kernel changes into mainline to do so.

Kevin
> 
> Thanks,
> Dan
> 
> > -----Original Message-----
> > From: Kevin Fox [mailto:Kevin.Fox@pnl.gov] 
> > Sent: Friday, December 30, 2005 9:35 AM
> > To: Tian, Kevin
> > Cc: Keir Fraser; Magenheimer, Dan (HP Labs Fort Collins); Xen 
> > Mailing List
> > Subject: RE: Guest-visible phys2mach part of Xen arch-neutral 
> > API? was:[Xen-devel] Uses of &frame_table[xfn]
> > 
> > There has been some discussion about some day using Xen on our large
> > Itanium cluster (~1000 nodes). I''m not sure if it would be
possible to
> > use Xen on the cluster without driver domains though. Just food for
> > thought.
> > 
> > Kevin
> > 
> > PS. I''m fairly disappointed in the lack of focus that driver 
> > domains are
> > receiving in Xen. I use them on several machines (one in 
> > production) and
> > I think it counts among Xen''s greatest features.
> > 
> > On Fri, 2005-12-30 at 10:16 +0800, Tian, Kevin wrote:
> > > >From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> > > >Sent: 2005年12月29日 21:48
> > > >
> > > >
> > > >On 29 Dec 2005, at 01:59, Tian, Kevin wrote:
> > > >
> > > >> 	IMO, I see the phys2mach mapping as a basic 
> > virtualization policy,
> > > >> instead of an architecture specific requirement. After
adding
> > > >> phys2mach concept to XEN/IA64, we can reuse more common 
> > code without
> > > >> ifdef. Then correspondingly also need to add several 
> > necessary changes
> > > >> like x86: DMA, SWIOTLB, AGP, etc, to ensure legal
machine address
> > > >> written into physical devices.
> > > >
> > > >This seems to make sense to me. How does ia64/xen work right
now?
> > > >Machine addresses visible to domain0 and full virtualisation
of
> > > >addresses exposed to other domains (with no way of seeing 
> > underlying
> > > >machine addresses)?
> > > >
> > > >  -- Keir
> > > 
> > > Yes, that''s the current status: No way to see underlying
> > machine address in other domains and thus no way for driver domains.
> > > 
> > > Thanks,
> > > Kevin
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Dec 30, 2005 at 01:00:41PM -0800, Kevin Fox wrote:
> NASA''s alticies (What the heck is the plural of Altix anyway? :)
Altixen.  Or should I say, AltiXen ;-)

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel