Dan Magenheimer
2010-May-05 18:14 UTC
[Xen-devel] Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?
Has anyone tried using a ramdisk in a PV domain? I''m trying it in an experiment to compare swapping to a ramdisk against tmem (frontswap) performance (suggested as a performance test on lkml). Writing to the ramdisk appears to be VERY VERY slow, elapsed time in the guest is several times larger than user+sys, and xentop shows the guest consuming vcpu seconds at about the user+sys rate. Note that this is when tmem is turned off and there is no vhd swap disk configured. I''m suspecting that writing to ramdisk must be causing some interesting/expensive PV pagetable behavior? Or maybe somehow /dev/ram0 is getting routed through qemu? Or ?? Thanks, Dan P.S. Yes, I realize a ramdisk has no real value in a guest... this is just a performance experiment. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2010-May-05 20:16 UTC
[Xen-devel] RE: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?
It appears that ramswap is bad for HV domains also. On one run: real 11m38.017s user 0m6.696s sys 0m27.865s I tried on bare-metal Linux and there is a similar pattern but not nearly as bad: real 0m54.978s user 0m8.577s sys 0m3.094s Maybe there is some horrible TLB thrashing going on even on bare-metal that is made even worse by virtualization?> -----Original Message----- > From: Dan Magenheimer > Sent: Wednesday, May 05, 2010 12:14 PM > To: Xen-Devel (xen-devel@lists.xensource.com) > Subject: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!? > > Has anyone tried using a ramdisk in a PV domain? > > I''m trying it in an experiment to compare swapping to > a ramdisk against tmem (frontswap) performance > (suggested as a performance test on lkml). > > Writing to the ramdisk appears to be VERY VERY slow, > elapsed time in the guest is several times larger than > user+sys, and xentop shows the guest consuming vcpu > seconds at about the user+sys rate. Note that > this is when tmem is turned off and there is no > vhd swap disk configured. > > I''m suspecting that writing to ramdisk must be causing > some interesting/expensive PV pagetable behavior? > Or maybe somehow /dev/ram0 is getting routed through > qemu? Or ?? > > Thanks, > Dan > > P.S. Yes, I realize a ramdisk has no real value in > a guest... this is just a performance experiment._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-May-06 17:42 UTC
Re: [Xen-devel] Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?
On 05/05/2010 11:14 AM, Dan Magenheimer wrote:> Has anyone tried using a ramdisk in a PV domain? > > I''m trying it in an experiment to compare swapping to > a ramdisk against tmem (frontswap) performance > (suggested as a performance test on lkml). > > Writing to the ramdisk appears to be VERY VERY slow, > elapsed time in the guest is several times larger than > user+sys, and xentop shows the guest consuming vcpu > seconds at about the user+sys rate. Note that > this is when tmem is turned off and there is no > vhd swap disk configured. > > I''m suspecting that writing to ramdisk must be causing > some interesting/expensive PV pagetable behavior? > Or maybe somehow /dev/ram0 is getting routed through > qemu? Or ?? >I haven''t looked at ramdisk, but I''m pretty sure there''s nothing special about accessing it. The only thing I can think of is that if you''re using a 32bit highmem system then you may be being hit by lots of kmap overhead. But on a 64-bit system, AFAIK, it should just be memory copies. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2010-May-06 21:02 UTC
RE: [Xen-devel] Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?
> Subject: Re: [Xen-devel] Writing to a ramdisk in a PV domain is > SLLLOOOWWW?!? > > > Writing to the ramdisk appears to be VERY VERY slow, > > elapsed time in the guest is several times larger than > > user+sys, and xentop shows the guest consuming vcpu > > seconds at about the user+sys rate. Note that > > this is when tmem is turned off and there is no > > vhd swap disk configured. > > > > I''m suspecting that writing to ramdisk must be causing > > some interesting/expensive PV pagetable behavior? > > Or maybe somehow /dev/ram0 is getting routed through > > qemu? Or ?? > > > > I haven''t looked at ramdisk, but I''m pretty sure there''s nothing > special > about accessing it. The only thing I can think of is that if you''re > using a 32bit highmem system then you may be being hit by lots of kmap > overhead. But on a 64-bit system, AFAIK, it should just be memory > copies.Thanks for the reply Jeremy. I tried a 64-bit guest (and bare-metal) and saw the same problem. I guess I''ll start collecting some statistics. Maybe since ramdisk (via the swap code) is still going through the blockio layer, there is some kind of cache or TLB overhead that normally would only be necessary for memory used for DMA? (I don''t know much about I/O so this is pure speculation.) Or maybe there is an assumption that since blockio is asynchronous, a timer is set to some minimum value and the write-to-ramdisk isn''t completed until the timer fires? (Hmmm... but this wouldn''t explain why virtual is much worse than bare-metal.) Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2010-May-06 22:04 UTC
RE: [Xen-devel] Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?
This is complete wild guess, but could it be that you are doing multi-page I/O with non contiguous machine pages? I believe the DMA API for pv guests uses swioltlb which does extra copies to a bounce buffer if you request a dma address to a multi-page region (as the pages may be non contiguous). Renato> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of > Dan Magenheimer > Sent: Thursday, May 06, 2010 2:02 PM > To: Jeremy Fitzhardinge > Cc: Xen-Devel (xen-devel@lists.xensource.com) > Subject: RE: [Xen-devel] Writing to a ramdisk in a PV domain > is SLLLOOOWWW?!? > > > Subject: Re: [Xen-devel] Writing to a ramdisk in a PV domain is > > SLLLOOOWWW?!? > > > > > Writing to the ramdisk appears to be VERY VERY slow, > elapsed time in > > > the guest is several times larger than > > > user+sys, and xentop shows the guest consuming vcpu > > > seconds at about the user+sys rate. Note that this is > when tmem is > > > turned off and there is no vhd swap disk configured. > > > > > > I''m suspecting that writing to ramdisk must be causing some > > > interesting/expensive PV pagetable behavior? > > > Or maybe somehow /dev/ram0 is getting routed through qemu? Or ?? > > > > > > > I haven''t looked at ramdisk, but I''m pretty sure there''s nothing > > special > > about accessing it. The only thing I can think of is that if you''re > > using a 32bit highmem system then you may be being hit by > lots of kmap > > overhead. But on a 64-bit system, AFAIK, it should just be memory > > copies. > > Thanks for the reply Jeremy. > > I tried a 64-bit guest (and bare-metal) and saw the same problem. > I guess I''ll start collecting some statistics. > > Maybe since ramdisk (via the swap code) is still going > through the blockio layer, there is some kind of cache or > TLB overhead that normally would only be necessary for > memory used for DMA? (I don''t know much about I/O so > this is pure speculation.) Or maybe there is an assumption > that since blockio is asynchronous, a timer is set to > some minimum value and the write-to-ramdisk isn''t completed > until the timer fires? (Hmmm... but this wouldn''t explain > why virtual is much worse than bare-metal.) > > Dan > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-May-06 22:26 UTC
Re: [Xen-devel] Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?
On 05/06/2010 03:04 PM, Santos, Jose Renato G wrote:> This is complete wild guess, but could it be that you are doing multi-page I/O with non contiguous machine pages? > I believe the DMA API for pv guests uses swioltlb which does extra copies to a bounce buffer if you request a dma address to a multi-page region (as the pages may be non contiguous). >Ramdisk access shouldn''t be going anywhere near DMA API or swiotlb. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel