Xen users - Jul 2008 - Measure disk activity in full-virtualization.

I''ve been doing tests to compare the performance isolation between
para-virtualized and full-virtualized guests using Intel VT. First of
all we performed tests over 2 VM''s (paravirtualized guest). Some bench
stressing memory and disk were done measured by xenmon and virt-top.
These two measure tools helped to determine CPU allocations, blocked
and waited times, and everything worked fine.

For measure disk usage, I understand that xenmon measures the transfer
of memory pages between DomU and Dom0, and virt-top could show VBD_RD
and VBD_WR for each VM. The problem arrived when measuring
full-virtualized guests. I can''t measure de disk usage of a particular
VM. I know that the split device driver model used in
para-virtualization change by the complete emulation of the hardware.

I found that the process that handles the VM image is called qemu-dm.
When stressing disk activity of that VM (let''s say VM1), no disk usage
of the qemu-dm process related to the VM1 is detected (using for
example pidstat over qemu-dm).Virt-top and xenmon didn''t give me any
value reflecting disk usage when full-virtualized guest are stressed.

Is there any way to measure disk activity of a particular VM with full
virtualization?
Why virt-top and xenmon didn''t give me any value?
Anybody tried something like this?

Thanks!

Alejandro.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,
> For measure disk usage, I understand that xenmon measures the transfer
> of memory pages between DomU and Dom0, and virt-top could show VBD_RD
> and VBD_WR for each VM. The problem arrived when measuring
do you know a kubuntu repository where i can install ocaml-libvirt
from? It is a dep. for virt-top what i want to test after you
mentioned it.

regards Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Alejandro,

On Fri, Jul 18, 2008 at 3:30 PM, Alejandro Paredes <aleparedes@gmail.com>
wrote:
> I''ve been doing tests to compare the performance isolation between
> para-virtualized and full-virtualized guests using Intel VT. First of
> all we performed tests over 2 VM''s (paravirtualized guest). Some
bench
> stressing memory and disk were done measured by xenmon and virt-top.
> These two measure tools helped to determine CPU allocations, blocked
> and waited times, and everything worked fine.
>
I think this is really interesting research. Can you give us some more
insight into
your testing methodology?

Are you familiar with the work at Clarkson University regarding performance
isolation?

http://clarkson.edu/~jnm/publications/isolation_ExpCS_FINALSUBMISSION.pdf
http://todddeshane.net/research/Xen_versus_KVM_20080623.pdf


Cheers,
Todd


-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

2008/7/20 Todd Deshane <deshantm@gmail.com>:
> Are you familiar with the work at Clarkson University regarding performance
> isolation?
>
>
http://clarkson.edu/~jnm/publications/isolation_ExpCS_FINALSUBMISSION.pdf<http://clarkson.edu/%7Ejnm/publications/isolation_ExpCS_FINALSUBMISSION.pdf>
> <http://todddeshane.net/research/Xen_versus_KVM_20080623.pdf>
>
I am very surprised by this publication whose conclusion is that a full
virtualised environment (VMWare) provides a perfect isolation between VMs
while a para virtualised one (Xen) is good but not perfect.

Our experience concerns WMWare server and Xen, we done some subjective and
practical tests (measuring the time some real huge tasks take to acheive)
and listen to users complains, and the results were quite clear : on VMWare
server, when a VM is very busy, its process eats all the CPU it runs on, and
other VMs slow down significantly along with the host itself. When the
number of VCPU exceeds the number of CPU (it is the general case), you may
have a delay of several second to see the echo of the character you typed on
a VM that does nothing because another one on the same host is 100% busy.

With Xen, no such a behaviour, isolation looks like much better. That why we
migrated all our servers from VMWare to Xen.

Perhaps benchmarks don''t always reflect the real world...

Alain.

> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Jul 21, 2008 at 5:21 AM, Alain Barthe <ab266061@gmail.com> wrote:
> 2008/7/20 Todd Deshane <deshantm@gmail.com>:
>
>> Are you familiar with the work at Clarkson University regarding
>> performance isolation?
>>
>>
http://clarkson.edu/~jnm/publications/isolation_ExpCS_FINALSUBMISSION.pdf<http://clarkson.edu/%7Ejnm/publications/isolation_ExpCS_FINALSUBMISSION.pdf>
>> <http://todddeshane.net/research/Xen_versus_KVM_20080623.pdf>
>>
>
> I am very surprised by this publication whose conclusion is that a full
> virtualised environment (VMWare) provides a perfect isolation between VMs
> while a para virtualised one (Xen) is good but not perfect.
>
> Our experience concerns WMWare server and Xen, we done some subjective and
> practical tests (measuring the time some real huge tasks take to acheive)
> and listen to users complains, and the results were quite clear : on VMWare
> server, when a VM is very busy, its process eats all the CPU it runs on,
and
> other VMs slow down significantly along with the host itself. When the
> number of VCPU exceeds the number of CPU (it is the general case), you may
> have a delay of several second to see the echo of the character you typed
on
> a VM that does nothing because another one on the same host is 100% busy.
>

When you say VMWare Server, are you referring to ESX?

We used VMWare workstation in that paper, ESX is a different animal
altogether.

>
> With Xen, no such a behaviour, isolation looks like much better. That why
> we migrated all our servers from VMWare to Xen.
>
> Perhaps benchmarks don''t always reflect the real world...
>
The workload matters a lot. The version of the software and hardware
configurations can matter a lot too.

Cheers,
Todd

-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

2008/7/22 Todd Deshane <deshantm@gmail.com>:
>
> When you say VMWare Server, are you referring to ESX?
>
I talked about the actual VMWare Server (
http://www.vmware.com/products/server/), which should be closed from VMWare
Workstation.

>
>
> We used VMWare workstation in that paper, ESX is a different animal
> altogether.
>
>
>>
>> With Xen, no such a behaviour, isolation looks like much better. That
why
>> we migrated all our servers from VMWare to Xen.
>>
>> Perhaps benchmarks don''t always reflect the real world...
>>
>
> The workload matters a lot. The version of the software and hardware
> configurations can matter a lot too.
>
Sure, but concerning CPU isolation, I can''t understand, theorically,
that
full virtualisation is better than para virtualisation. And our experiment
confirms this. Can you explain this point ?

Thanks.

>
>
> Cheers,
> Todd
>
> --
> Todd Deshane
> http://todddeshane.net
> check out our book: http://runningxen.com
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 23/07/2008 10:11, Alain Barthe wrote:
> 2008/7/22 Todd Deshane <deshantm@gmail.com
<mailto:deshantm@gmail.com>>:
> 
>     When you say VMWare Server, are you referring to ESX?
> 
> I talked about the actual VMWare Server 
> (http://www.vmware.com/products/server/), which should be closed from 
> VMWare Workstation.
The free VMware Server product is essentially the previous VMware GSX 
product, rather than VMware Workstation.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>
> Sure, but concerning CPU isolation, I can''t understand,
theorically, that
> full virtualisation is better than para virtualisation. And our experiment
> confirms this. Can you explain this point ?
My understanding is that a lot of the CPU isolation comes from the scheduler.
The schedulers in modern operating systems in general should be good enough
to provide good isolation, regardless of para vs. full virtualization.

Performance isolation is different than simple measuring overall performance and
I don''t think that it is fundamental that full or para will be better
in general. The
implementation done by the scheduling mechanism both in the guest and of the
hypervisor matters. It is very possible that our CPU test was not
stressful enough.
It is also possible that the tests that you are doing stress more than
just the CPU
and the degradation noticed in the other guests is due some other factor such as
disk or network or even memory.

Our work is actually ongoing and choosing the right CPU-intensive test
will be an
important decision going forward.

Thanks for your questions/comments. Feel free to give suggestions etc.

I like to see that there is others doing performance isolation testing
now. When we
started, we didn''t know of others doing that kind of testing.

Cheers,
Todd

-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

2008/7/23 Todd Deshane <deshantm@gmail.com>:
> >
> > Sure, but concerning CPU isolation, I can''t understand,
theorically, that
> > full virtualisation is better than para virtualisation. And our
> experiment
> > confirms this. Can you explain this point ?
>
> My understanding is that a lot of the CPU isolation comes from the
> scheduler.
> The schedulers in modern operating systems in general should be good enough
> to provide good isolation, regardless of para vs. full virtualization.

I agree. I suppose that the hypervisor scheduler does the job in PV, and the
OS process scheduler does the job in FV. I am not sure they have the same
policy, nor the same goals. Also, the hypervisor scheduler seems to be more
tunable than the OS one.

> Performance isolation is different than simple measuring overall
> performance and
> I don''t think that it is fundamental that full or para will be
better
> in general. The
> implementation done by the scheduling mechanism both in the guest and of
> the
> hypervisor matters. It is very possible that our CPU test was not
> stressful enough.
> It is also possible that the tests that you are doing stress more than
> just the CPU
> and the degradation noticed in the other guests is due some other factor
> such as
> disk or network or even memory.

Yes. The test we done was roughly 5 parallel gzipped tar on big files, so I
think that mainly the CPU and disk I/O were stressed. So you are right, we
cannot compare your methodic benchmark with our simple and subjective
experiment.

> Our work is actually ongoing and choosing the right CPU-intensive test
> will be an
> important decision going forward.
>
> Thanks for your questions/comments. Feel free to give suggestions etc.
>
> I like to see that there is others doing performance isolation testing
> now. When we
> started, we didn''t know of others doing that kind of testing.
>
> Cheers,
> Todd
>
Thanks for your reply.

>
>
> --
> Todd Deshane
> http://todddeshane.net
> check out our book: http://runningxen.com
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi all, I''m going to briefly explain what I''ve done. First of
all,
I''ll mention the tools used…
-	Stress: It imposes a configurable amount of CPU, memory, I/O, and
disk stress on the system.
-	Memtester: is an effective userspace tester for stress-testing the
memory subsystem.  It is very effective at finding intermittent and
non-deterministic faults. Memtester will malloc the amount of memory
specified.
-	Bonnie++: is a program to test hard drives and file systems for
performance or the lack therof.  There are a many different types of
file system operations which different applications use to different
degrees. Bonnie++ tests some of them and for each test gives a result
of the amount of work done per second and the percentage of CPU time
this took. There are two sections to the program''s operations. The
first is to test the IO throughput in a fashion that is designed to
simulate some types of database applications.  The second is to test
creation, reading, and deleting files.
-	Sysstat: The sysstat package contains utilities to monitor system
performance and usage activity.
-	Xenmon: make use of the existing xen tracing feature to provide fine
grained reporting of various domain related metrics like: Execution
Count, CPU usage, Waiting time, etc.


The measure procedure was like this:
1-	Two clones guests were created with the following characteristics:
1024 RAM, 8 Gb disk, 8 Vitual CPUs. Lets call them VM1 and VM2
respectively.
2-	The xen credit scheduler was used, assigning the same weight to
each virtual machine.
3-	One particular stress was run over VM1 (for example CPU stress)
with the VM2 in an idle state. Here the all the metrics were taken
from inside the VM1 (using sysstat) and from dom0 (using xennom and
xentop).
4-	After that, the same test was re-run in VM1 with another parameter
stressed in VM2 (for example memory stress).
5-	A normalized value is calculated dividing the performance value
obtained from VM1 (taken from step 3) by the value from VM1 taken from
step 4. Here we can see the fall of performance in VM1 due to VM2 (or
fall of CPU due to memory).
6-	Then, step 3 and 4 are done again but the load of each virtual
machine is exchanged (VM1 memory stressed and VM2 CPU stressed). Here
we can find the fall he performance of memory due to CPU.
7-	By summing the values of step 5 and 6 we can find the overall
performance inference of memory-CPU.
8-	The same test can be run for:
CPU-disk
memory-disk
CPU-CPU
Disk-disk

Before doing this procedure, all the values of the stress tests were
taken from inside each VM, without any value seen from Dom0.
Everything seemed to be fine, but when I run memory against CPU the
results were disconcerting. The problem was that each VM had its own
virtual time, so every result was relative to the VM that run the
test, so there wasn''t any possibility of compare the results. The
scheduler played an important role at this time, so it was very
important to see this environment from outside of each VM, lets say
from Dom0. Xenmon and xentop helped a lot to solve this problem.

For measure disk usage, I understand that xenmon measures the transfer
of memory pages between DomU and Dom0, and virt-top could show VBD_RD
and VBD_WR for each VM. The problem arrived when measuring
full-virtualized guests. I can''t measure de disk usage of a particular
VM. I know that the split device driver model used in
para-virtualization change by the complete emulation of the hardware.

I found that the process that handles the VM image is called qemu-dm.
When stressing disk activity of that VM (let''s say VM1), no disk usage
of the qemu-dm process related to the VM1 is detected (using for
example pidstat over qemu-dm).Virt-top and xenmon didn''t give me any
value reflecting disk usage when full-virtualized guest are stressed.

Is there any way to measure disk activity of a particular VM with full
virtualization?
Why virt-top and xenmon didn''t give me any value?
Anybody tried something like this?

Thanks!

Alejandro.


2008/7/19 Todd Deshane <deshantm@gmail.com>:
> Hi Alejandro,
>
> On Fri, Jul 18, 2008 at 3:30 PM, Alejandro Paredes
<aleparedes@gmail.com>
> wrote:
>>
>> I''ve been doing tests to compare the performance isolation
between
>> para-virtualized and full-virtualized guests using Intel VT. First of
>> all we performed tests over 2 VM''s (paravirtualized guest).
Some bench
>> stressing memory and disk were done measured by xenmon and virt-top.
>> These two measure tools helped to determine CPU allocations, blocked
>> and waited times, and everything worked fine.
>
> I think this is really interesting research. Can you give us some more
> insight into
> your testing methodology?
>
> Are you familiar with the work at Clarkson University regarding performance
> isolation?
>
> http://clarkson.edu/~jnm/publications/isolation_ExpCS_FINALSUBMISSION.pdf
> http://todddeshane.net/research/Xen_versus_KVM_20080623.pdf
>
>
> Cheers,
> Todd
>
>
> --
> Todd Deshane
> http://todddeshane.net
> check out our book: http://runningxen.com
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users