Xen devel - Sep 2012 - additional domain.c memory allocation causes "xm create" to fail

Hello Xen Developers,

I am currently attempting an experiment to modify Xen in such a way that it
will flush the L2 cache every time it performs a context switch.  Thus far
I have implemented my code inside the __context_switch() function, of
domain.c (xen/arch/x86/domain.c), and while my modifications work fine for
memory allocations of up to 512KB, it fails whenever I increase this to 1MB.

Specifically, when the memory buffer is increased to 1MB, the machine will
force a hard reset whenever xm tries to create a new domain.  When I try to
create a new domain (a sparse squeeze install, Dom0 is running Ubuntu
12.04) it gets as far as completing the scripts/init-bottom call before it
crashes, which makes me think it is going down during the following init
call.

I have narrowed down the section of code that is failing.  The curious
thing is that the failure threshold seems to be dependent on the number of
iterations in the loop, rather than the specific amount of memory (i.e. 1MB
of memory will work when ''i'' is incremented by 128 rather than
64, whereas
512KB of memory will work when ''i'' is 64):

  cache_mem_size = 1048576;  // Size of L2 cache
  cache_mem = xmalloc_array(char,cache_mem_size);

  for (i=0;i<cache_mem_size;i+=64)
    cache_mem[i] += 1;

  xfree(cache_mem);

If anyone has a suggestion as to what may be causing this failure, or
insight into the runtime limitations of this section of the architecture,
kernel, or scheduler, I would appreciate the information.

Thanks,

-Misiu


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 04/09/12 19:22, misiu godfrey wrote:
> Hello Xen Developers,
>
> I am currently attempting an experiment to modify Xen in such a way
> that it will flush the L2 cache every time it performs a context switch.
For what purpose?  There was once a bug which caused this to happen and
it caused Xen to slow to a glacial pace.  We got bored of debugging HVM
domains after several hours and the BIOS has still not loaded the
bootloader.
>  Thus far I have implemented my code inside the __context_switch()
> function, of domain.c (xen/arch/x86/domain.c), and while my
> modifications work fine for memory allocations of up to 512KB, it
> fails whenever I increase this to 1MB.
>
> Specifically, when the memory buffer is increased to 1MB, the machine
> will force a hard reset whenever xm tries to create a new domain.
>  When I try to create a new domain (a sparse squeeze install, Dom0 is
> running Ubuntu 12.04) it gets as far as completing the
> scripts/init-bottom call before it crashes, which makes me think it is
> going down during the following init call.
>
> I have narrowed down the section of code that is failing.  The curious
> thing is that the failure threshold seems to be dependent on the
> number of iterations in the loop, rather than the specific amount of
> memory (i.e. 1MB of memory will work when ''i'' is
incremented by 128
> rather than 64, whereas 512KB of memory will work when
''i'' is 64):
>
>   cache_mem_size = 1048576;  // Size of L2 cache
>   cache_mem = xmalloc_array(char,cache_mem_size);
You don''t check the return value, so what happens when the allocation
fails?  I would say that calling *alloc() is not a sensible thing to be
doing in __context_switch() anyway, as you are sitting doing a long
operation while in a critical section of Xen code.
>
>   for (i=0;i<cache_mem_size;i+=64)
>     cache_mem[i] += 1;
>
>   xfree(cache_mem);
Furthermore, this algorithm has no guarantee to clear the L2 cache.  In
fact is almost certainly will not.
>
> If anyone has a suggestion as to what may be causing this failure, or
> insight into the runtime limitations of this section of the
> architecture, kernel, or scheduler, I would appreciate the information.
>
> Thanks,
>
> -Misiu 
-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

> For what purpose?  There was once a bug which caused this to happen and
> it caused Xen to slow to a glacial pace.  We got bored of debugging HVM
> domains after several hours and the BIOS has still not loaded the
> bootloader.
>
I''m working on an experiment to do with cache-based side channels in
Cloud
environments.  Part of it involves measuring the effect of flushing the
cache every time there is a VM switch.

You don''t check the return value, so what happens when the
allocation
> fails?  I would say that calling *alloc() is not a sensible thing to be
> doing in __context_switch() anyway, as you are sitting doing a long
> operation while in a critical section of Xen code.
>
Unfortunately, I can chalk that up to my inexperience with C programming.
 Thanks for pointing that out.

As for the sensibility of the plan, it is still in rather early stages and
not as robust as I would like it.  As I get more working I was planning on
leaving the memory buffer permanently allocated so as not to spend time
managing it in a critical section.  If you have a suggestion for a more
practical solution I''m all ears.

Furthermore, this algorithm has no guarantee to clear the L2 cache. 
In
> fact is almost certainly will not.
>
This is the code that has worked in all of my prior experiments and has
been ratified by others I have worked with.  Are you sure it wouldn''t
work?
 While, for simplicity''s sake, I have removed portions of the code
designed
to prevent pre-fetching and perhaps left out something important, my
understanding of cache-coloring, however, would still imply that the data
in the cache should be dirty, or flushed after this loop terminates.

Perhaps I have misused the term "flush".  My objective is to make each
cache line dirty, or flush it to main memory.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 04/09/12 20:45, misiu godfrey wrote:
>
>     For what purpose?  There was once a bug which caused this to
>     happen and
>     it caused Xen to slow to a glacial pace.  We got bored of
>     debugging HVM
>     domains after several hours and the BIOS has still not loaded the
>     bootloader.
>
>
> I''m working on an experiment to do with cache-based side channels
in
> Cloud environments.  Part of it involves measuring the effect of
> flushing the cache every time there is a VM switch.
Ok.  My suspicion is that it will be unreasonably slow, although should
not be the glacial pace we saw with the bug (which was flushing the L2
cache on every instruction).
>
>     You don''t check the return value, so what happens when the
allocation
>     fails?  I would say that calling *alloc() is not a sensible thing
>     to be
>     doing in __context_switch() anyway, as you are sitting doing a long
>     operation while in a critical section of Xen code.
>
>
> Unfortunately, I can chalk that up to my inexperience with C
> programming.  Thanks for pointing that out.
>
> As for the sensibility of the plan, it is still in rather early stages
> and not as robust as I would like it.  As I get more working I was
> planning on leaving the memory buffer permanently allocated so as not
> to spend time managing it in a critical section.  If you have a
> suggestion for a more practical solution I''m all ears.
>
>     Furthermore, this algorithm has no guarantee to clear the L2
>     cache.  In
>     fact is almost certainly will not.
>
>
> This is the code that has worked in all of my prior experiments and
> has been ratified by others I have worked with.  Are you sure it
> wouldn''t work?  While, for simplicity''s sake, I have
removed portions
> of the code designed to prevent pre-fetching and perhaps left out
> something important, my understanding of cache-coloring, however,
> would still imply that the data in the cache should be dirty, or
> flushed after this loop terminates.
>
> Perhaps I have misused the term "flush".  My objective is to make
each
> cache line dirty, or flush it to main memory.
"flush" is the correct term.

However, the structure of caches work against you.  With a set
associative cache, you have no control over which of the sets gets used
for your cache line.  So on an N way set associate cache, your worst
case may only dirty 1/N of the actual lines in the cache.

After that, your L1 cache inclusion policy is going to affect how you
dirty your L2 cache, as well as whether you have joined caches or split
instruction and data caches.

Furthermore, on newer processes, multiple cores may share an L2 or L3
cache, and context switches are unlike to occur at exactly the same time
on each core, meaning that a context switch on one core is going to
(attempt to) nuke the L2 cache of the VM which is in mid run on another
core.  Conversely, even executing the loop trying to dirty the cache
will mean that you dont get all of it, and having another core executing
on the same L2 cache means it will pull its data back during your
dirtying loop.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> "flush" is the correct term.
>
> However, the structure of caches work against you.  With a set associative
> cache, you have no control over which of the sets gets used for your cache
> line.  So on an N way set associate cache, your worst case may only dirty
> 1/N of the actual lines in the cache.
>
> After that, your L1 cache inclusion policy is going to affect how you
> dirty your L2 cache, as well as whether you have joined caches or split
> instruction and data caches.
>
> Furthermore, on newer processes, multiple cores may share an L2 or L3
> cache, and context switches are unlike to occur at exactly the same time on
> each core, meaning that a context switch on one core is going to (attempt
> to) nuke the L2 cache of the VM which is in mid run on another core.
> Conversely, even executing the loop trying to dirty the cache will mean
> that you dont get all of it, and having another core executing on the same
> L2 cache means it will pull its data back during your dirtying loop.
>
I have some more robust code that takes account of the set-associativity of
the cache, code that I originally thought was going to be superfluous in
this situation.  Now that I have managed to execute this basic loop I can
address a more complex environment with a set-associative.  Currently, I
don''t need to worry about an L3 cache because my test machine has no
shared
cache between cores (nothing higher than an L2).  Although I will keep this
in mind, as it will need to be addressed once I get beyond the proof of
concept.

Anyway, validating the xmalloc return value seems to have addressed my
problem, although the log I am printing to seems to imply that xmalloc is
never failing.  I''ll look further into it once I get more things
working.
 Thanks a lot for your advice, Andrew.  Sorry my problem ended up being so
trivial.

-Misiu


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel