Xen devel - Mar 2012 - Accessing Dom0 Physical memory from xen, via direct mappings (PML4:262-271)

In config.h (include/asm-x86/config.h) I found this:

#if __x86_64__
...
 *  0xffff830000000000 - 0xffff87ffffffffff [5TB, 5*2^40 bytes, PML4:262-271]
 *    1:1 direct mapping of all physical memory.                   
...

I was wondering if it''s possible for dom0 to malloc a huge chunk of
memory
and let xen know the starting address of this range. 
Inside xen, I can translate dom0 virt address to a virt address in the above 
range and access the entire chunk via these virtual addresses.

The catch here is that I want this virtual address range to be accessible 
across all vcpu contexts in xen (whether it''s servicing a hypercall
from dom0
or a vmx fault caused by Guest).

So far, I have only been able to achieve the former. In the latter case,
where the "current" vcpu belongs to a guest (eg in a vmx fault
handler),
I can''t access this address range inside xen. Do I have to add EPT
mappings to guest''s p2m to do this ? Or can I do something else ?


thanks
shriram

At 08:45 -0700 on 13 Mar (1331628358), Shriram Rajagopalan
wrote:
> In config.h (include/asm-x86/config.h) I found this:
> 
> #if __x86_64__
> ...
>  *  0xffff830000000000 - 0xffff87ffffffffff [5TB, 5*2^40 bytes,
PML4:262-271]
>  *    1:1 direct mapping of all physical memory.                   
> ...
> 
> I was wondering if it''s possible for dom0 to malloc a huge chunk
of memory
> and let xen know the starting address of this range. 
> Inside xen, I can translate dom0 virt address to a virt address in the
above
> range and access the entire chunk via these virtual addresses.
Eh, maybe?  But it''s harder than you''d think.  Memory
malloc()ed in dom0
may not be contiguous in PFN-space, and dom0 PFNs may not be contiguous
in MFN-space, so you can''t just translate the base of the buffer and
use
it with offsets, you have to translate again for every 4k page.  Also, you
need to make sure dom0 doesn''t page out or relocate your user-space
buffer while Xen is accessing the MFNs.  mlock() might do what you need
but AIUI there''s no guarantee that it won''t, e.g., move the
buffer
around to defragment memory.

If you do handle all that, the correct way to get at the mappings in
this range is with map_domain_page().  But remember, this only works
like this on 64-bit Xen.  On 32-bit, only a certain amount of memory can
be mapped at one time so if the buffer is really big, you''ll need to
map
an unmap parts of it on demand.

But maybe back up a bit: why do you want to do this?  What''s the buffer
for?  Is it something you could do more easily by having Xen allocate
the buffer and let dom0 map it?
> The catch here is that I want this virtual address range to be accessible 
> across all vcpu contexts in xen (whether it''s servicing a
hypercall from dom0
> or a vmx fault caused by Guest).
> 
> So far, I have only been able to achieve the former. In the latter case,
> where the "current" vcpu belongs to a guest (eg in a vmx fault
handler),
> I can''t access this address range inside xen. Do I have to add EPT
> mappings to guest''s p2m to do this ? Or can I do something else ?
If you really have got a pointer into the 1-1 mapping it should work
from any vcpu.  But again, that''s not going to work on 32-bit Xen.
There, you have to use map_domain_page_global() to get a mapping that
persists across all vcpus, and that''s even more limited in how much it
can map at once.

Cheers,

Tim.

On Tue, Mar 13, 2012 at 9:08 AM, Tim Deegan <tim@xen.org> wrote:
> At 08:45 -0700 on 13 Mar (1331628358), Shriram Rajagopalan wrote:
> > In config.h (include/asm-x86/config.h) I found this:
> >
> > #if __x86_64__
> > ...
> >  *  0xffff830000000000 - 0xffff87ffffffffff [5TB, 5*2^40 bytes,
> PML4:262-271]
> >  *    1:1 direct mapping of all physical memory.
> > ...
> >
> > I was wondering if it''s possible for dom0 to malloc a huge
chunk of
> memory
> > and let xen know the starting address of this range.
> > Inside xen, I can translate dom0 virt address to a virt address in the
> above
> > range and access the entire chunk via these virtual addresses.
>
> Eh, maybe?  But it''s harder than you''d think.  Memory
malloc()ed in dom0
> may not be contiguous in PFN-space, and dom0 PFNs may not be contiguous
> in MFN-space, so you can''t just translate the base of the buffer
and use
> it with offsets, you have to translate again for every 4k page.  Also, you
> need to make sure dom0 doesn''t page out or relocate your
user-space
> buffer while Xen is accessing the MFNs.  mlock() might do what you need
> but AIUI there''s no guarantee that it won''t, e.g., move
the buffer
> around to defragment memory.
>
>
Yep. I am aware of the above issues. As far as contiguity is concerned,
I was hoping (*naively/lazily*) that if I allocate a huge chunk (1G or so)
using posix_memalign, it would start at page boundary and also be contiguous
*most* of the time.  I need this setup only for some temporary analysis and
not for a production quality system.
And the machine has more than enough ram, with swap usage being 0 all the
time.


If you do handle all that, the correct way to get at the mappings
in
> this range is with map_domain_page().  But remember, this only works
> like this on 64-bit Xen.  On 32-bit, only a certain amount of memory can
> be mapped at one time so if the buffer is really big, you''ll need
to map
> an unmap parts of it on demand.
>
>
64-bit. The comments I pointed out were under the #if x86_64 region.

But maybe back up a bit: why do you want to do this?  What''s the
buffer
> for?  Is it something you could do more easily by having Xen allocate
> the buffer and let dom0 map it?
>
>
Well, the buffer acts as a huge log dirty "byte" map (a byte per
word).
I am skipping the reason for doing this huge byte map, for the sake of
brevity.

Can I have xen allocate this huge buffer ? (a byte per 8-byte word means
about
128M for a 1G guest). And if I were to have this byte-map per-vcpu, it
would mean
512M worth of RAM, for a 4-vcpu guest.

Is there a way I could increase the xen heap size to be able to allocate
this much memory?
And how do I map the xen memory in dom0 ? I vaguely remember seeing similar
code in
xentrace, but if you could point me in the right direction, it would be
great.


 > The catch here is that I want this virtual address range to be
accessible
> > across all vcpu contexts in xen (whether it''s servicing a
hypercall from
> dom0
> > or a vmx fault caused by Guest).
> >
> > So far, I have only been able to achieve the former. In the latter
case,
> > where the "current" vcpu belongs to a guest (eg in a vmx
fault handler),
> > I can''t access this address range inside xen. Do I have to
add EPT
> > mappings to guest''s p2m to do this ? Or can I do something
else ?
>
> If you really have got a pointer into the 1-1 mapping it should work
> from any vcpu.
>
 But again, that''s not going to work on 32-bit
Xen.
> There, you have to use map_domain_page_global() to get a mapping that
> persists across all vcpus, and that''s even more limited in how
much it
> can map at once.
>
> Cheers,
>
> Tim.
>
>
cheers
shriram


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

At 09:32 -0700 on 13 Mar (1331631148), Shriram Rajagopalan
wrote:
> Yep. I am aware of the above issues. As far as contiguity is concerned,
> I was hoping (*naively/lazily*) that if I allocate a huge chunk (1G or so)
> using posix_memalign, it would start at page boundary and also be
contiguous
> *most* of the time.
I think you''re playing with fire -- you have no way of knowing whether
you were lucky enough to get a contiguous region and if not you''ll
silently corrupt memory _and_ pollute your results.
> Well, the buffer acts as a huge log dirty "byte" map (a byte per
word).
> I am skipping the reason for doing this huge byte map, for the sake of
> brevity.
> 
> Can I have xen allocate this huge buffer ? (a byte per 8-byte word means
> about
> 128M for a 1G guest). And if I were to have this byte-map per-vcpu, it
> would mean
> 512M worth of RAM, for a 4-vcpu guest.
> 
> Is there a way I could increase the xen heap size to be able to allocate
> this much memory?
IIRC on x64 xen heap is the same as domheap so it already includes all
free space.  Or you could pick a 1GB region at boot time, make sure it
doesn''t get given to the memory allocators, and then mark it all as
owned by dom0 in the frametable.
> And how do I map the xen memory in dom0 ? I vaguely remember seeing similar
> code in
> xentrace, but if you could point me in the right direction, it would be
> great.
I don''t recall the details, but xentrace isn''t that big.  If I
were
doing it, I''d use share_xen_pages_with_guest() or similar to allow dom0
to see it, and then map it with ordinary mapping hypercalls.

Tim.

> Date: Tue, 13 Mar 2012 09:32:28 -0700
> From: Shriram Rajagopalan <rshriram@cs.ubc.ca>
> To: Tim Deegan <tim@xen.org>
> Cc: "xen-devel@lists.xensource.com"
<xen-devel@lists.xensource.com>
> Subject: Re: [Xen-devel] Accessing Dom0 Physical memory from xen, via
> 	direct mappings (PML4:262-271)
> Message-ID:
> 	<CAP8mzPPJTK_tg5h2JUTKOHptrG1C2giok3Yy9OO+=zwWEKq4Pg@mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Tue, Mar 13, 2012 at 9:08 AM, Tim Deegan <tim@xen.org> wrote:
>
>> At 08:45 -0700 on 13 Mar (1331628358), Shriram Rajagopalan wrote:
>> > In config.h (include/asm-x86/config.h) I found this:
>> >
>> > #if __x86_64__
>> > ...
>> >  *  0xffff830000000000 - 0xffff87ffffffffff [5TB, 5*2^40 bytes,
>> PML4:262-271]
>> >  *    1:1 direct mapping of all physical memory.
>> > ...
>> >
>> > I was wondering if it''s possible for dom0 to malloc a
huge chunk of
>> memory
>> > and let xen know the starting address of this range.
>> > Inside xen, I can translate dom0 virt address to a virt address in
the
>> above
>> > range and access the entire chunk via these virtual addresses.
>>
>> Eh, maybe?  But it''s harder than you''d think.  Memory
malloc()ed in dom0
>> may not be contiguous in PFN-space, and dom0 PFNs may not be contiguous
>> in MFN-space, so you can''t just translate the base of the
buffer and use
>> it with offsets, you have to translate again for every 4k page.  Also,
>> you
>> need to make sure dom0 doesn''t page out or relocate your
user-space
>> buffer while Xen is accessing the MFNs.  mlock() might do what you need
>> but AIUI there''s no guarantee that it won''t, e.g.,
move the buffer
>> around to defragment memory.
>>
>>
> Yep. I am aware of the above issues. As far as contiguity is concerned,
> I was hoping (*naively/lazily*) that if I allocate a huge chunk (1G or so)
> using posix_memalign, it would start at page boundary and also be
> contiguous
> *most* of the time.  I need this setup only for some temporary analysis
> and
> not for a production quality system.
> And the machine has more than enough ram, with swap usage being 0 all the
> time.
>
>
> If you do handle all that, the correct way to get at the mappings in
>> this range is with map_domain_page().  But remember, this only works
>> like this on 64-bit Xen.  On 32-bit, only a certain amount of memory
can
>> be mapped at one time so if the buffer is really big, you''ll
need to map
>> an unmap parts of it on demand.
>>
>>
> 64-bit. The comments I pointed out were under the #if x86_64 region.
>
> But maybe back up a bit: why do you want to do this?  What''s the
buffer
>> for?  Is it something you could do more easily by having Xen allocate
>> the buffer and let dom0 map it?
>>
>>
> Well, the buffer acts as a huge log dirty "byte" map (a byte per
word).
> I am skipping the reason for doing this huge byte map, for the sake of
> brevity.
>
> Can I have xen allocate this huge buffer ? (a byte per 8-byte word means
> about
> 128M for a 1G guest). And if I were to have this byte-map per-vcpu, it
> would mean
> 512M worth of RAM, for a 4-vcpu guest.
>
> Is there a way I could increase the xen heap size to be able to allocate
> this much memory?
> And how do I map the xen memory in dom0 ? I vaguely remember seeing
> similar
> code in
> xentrace, but if you could point me in the right direction, it would be
> great.
Have you looked into XENMEM_exchange? There might be some size constraints
to this hypercall, but assuming you meet them, you could
1. have your user-space dom0 tool mmap an fd from your kernel driver
2. your kernel driver get_free_page()s as many as necessary
3. It then calls XENMEM_exchange. The hypervisor will take the mfn''s
and
hand back a contiguous mfn range.
4. Hook up the resulting mfn''s into the user-space mmap

Now you have un-swappable, page-aligned, machine-contiguous memory, mapped
into your dom0 tool. And any other actor can easily identify this region
with the base mfn and the page count. I don''t think a guest vcpu could
map
this, though, being dom0 pages, but certainly the hypervisor can map it
and store information as appropriate.

I bring this up because it would be far easier than having Xen remember
all the mfn''s in your array, which I''m suspicious you might
need for your
byte-dirty use case.

Hope that helps
Andres
>
>
>  > The catch here is that I want this virtual address range to be
> accessible
>> > across all vcpu contexts in xen (whether it''s servicing a
hypercall
>> from
>> dom0
>> > or a vmx fault caused by Guest).
>> >
>> > So far, I have only been able to achieve the former. In the latter
>> case,
>> > where the "current" vcpu belongs to a guest (eg in a vmx
fault
>> handler),
>> > I can''t access this address range inside xen. Do I have
to add EPT
>> > mappings to guest''s p2m to do this ? Or can I do
something else ?
>>
>> If you really have got a pointer into the 1-1 mapping it should work
>> from any vcpu.
>>
>  But again, that''s not going to work on 32-bit Xen.
>> There, you have to use map_domain_page_global() to get a mapping that
>> persists across all vcpus, and that''s even more limited in how
much it
>> can map at once.
>>
>> Cheers,
>>
>> Tim.
>>
>>
> cheers
> shriram

On Tue, Mar 13, 2012 at 9:43 AM, Tim Deegan <tim@xen.org> wrote:
> At 09:32 -0700 on 13 Mar (1331631148), Shriram Rajagopalan wrote:
> > Yep. I am aware of the above issues. As far as contiguity is
concerned,
> > I was hoping (*naively/lazily*) that if I allocate a huge chunk (1G or
> so)
> > using posix_memalign, it would start at page boundary and also be
> contiguous
> > *most* of the time.
>
> I think you''re playing with fire -- you have no way of knowing
whether
> you were lucky enough to get a contiguous region and if not you''ll
> silently corrupt memory _and_ pollute your results.
>
> > Well, the buffer acts as a huge log dirty "byte" map (a byte
per word).
> > I am skipping the reason for doing this huge byte map, for the sake of
> > brevity.
> >
> > Can I have xen allocate this huge buffer ? (a byte per 8-byte word
means
> > about
> > 128M for a 1G guest). And if I were to have this byte-map per-vcpu, it
> > would mean
> > 512M worth of RAM, for a 4-vcpu guest.
> >
> > Is there a way I could increase the xen heap size to be able to
allocate
> > this much memory?
>
> IIRC on x64 xen heap is the same as domheap so it already includes all
> free space.  Or you could pick a 1GB region at boot time, make sure it
> doesn''t get given to the memory allocators, and then mark it all
as
> owned by dom0 in the frametable.
>
>
xen heap seems to be the same domheap. however, when I try to allocate
more than 128MB of memory, it punts!
 I am using the alloc_xenheap_pages(order, 0).
Am I missing something here ?

I am not sure if Andres'' XENMEM_exchange mechanism allows >128MB.
I think going for a boot time allocation seems to be the only way, if I want
a huge region like 1GB.

>  > And how do I map the xen memory in dom0 ? I vaguely remember seeing
> similar
> > code in
> > xentrace, but if you could point me in the right direction, it would
be
> > great.
>
> I don''t recall the details, but xentrace isn''t that big. 
If I were
> doing it, I''d use share_xen_pages_with_guest() or similar to allow
dom0
> to see it, and then map it with ordinary mapping hypercalls.
>
> Tim.
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

At 20:04 -0700 on 13 Mar (1331669099), Shriram Rajagopalan
wrote:
> On Tue, Mar 13, 2012 at 9:43 AM, Tim Deegan <tim@xen.org> wrote:
> > At 09:32 -0700 on 13 Mar (1331631148), Shriram Rajagopalan wrote:
> > > Can I have xen allocate this huge buffer ? (a byte per 8-byte
word means
> > > about
> > > 128M for a 1G guest). And if I were to have this byte-map
per-vcpu, it
> > > would mean
> > > 512M worth of RAM, for a 4-vcpu guest.
> > >
> > > Is there a way I could increase the xen heap size to be able to
allocate
> > > this much memory?
> >
> > IIRC on x64 xen heap is the same as domheap so it already includes all
> > free space.  Or you could pick a 1GB region at boot time, make sure it
> > doesn''t get given to the memory allocators, and then mark it
all as
> > owned by dom0 in the frametable.
> >
> >
> xen heap seems to be the same domheap. however, when I try to allocate
> more than 128MB of memory, it punts!
>  I am using the alloc_xenheap_pages(order, 0).
> Am I missing something here ?
> 
> I am not sure if Andres'' XENMEM_exchange mechanism allows
>128MB.
It has to use the allocator to get the contiguous memory, so if you
can''t get that memory directle, xenmem_exchange won''t work
either.

Tim.