libvirt users - Jun 2014 - data monitored by "virDomainBlockStats" API not consistent with ones reading from /proc/[pid]/io, and corrugated with /proc/diskstats

Hi, all.  
While monitoring KVM VMs’ disk utilisation using libvirt API
“virDomainBlockStats", I find that the results acquired is not consistent
with the value read from /proc/[vm_pid]/io(i only care about “rd_bytes" and
“wr_bytes” fields). Since libvirt doesn’t support host level monitoring,
i have to calculate the host’s disk utilisation by reading from the
/proc/diskstats file. What’s worse, the individual VM’s disk data may
be greater that the physical disk’s total value during some sampling time. 

It is supposed that the VM’s disk “rd_bytes" and “wr_bytes" should
never exceed the value of the physical disk respectively, right ? Doesn’t
libvirt virDomainBlockStats read disk information from “proc” ?  

I want to calculate VM’s bandwidth, here is my way:
	Firstly, we can get total disk utilisation using tools like iostat("iostat
-x”, the %util field represent the disk’s utilisation percentage)
	Then, given a sampling time period, 10s for example, i collected total bytes
read/written by each VM by virDomainBlockStats and total
		bytes of the physical disk by /proc/diskstats, then delta(VM’s rd+wr
bytes)/delta(physical disk’s rd+wr bytes) * %util is the VM’s
		disk utilization percentage during this time period.
Since the appearance of the above situation, i can’t get correct result. I have
search a lot for this question, but no answers.  Any help would
be appreciated.

On 06/24/2014 02:24 AM, coperd wrote:
>
> It is supposed that the VM’s disk “rd_bytes" and “wr_bytes"
should
never exceed the value of the physical disk respectively, right ?

That's a question for the qemu folks.
> Doesn’t
> libvirt virDomainBlockStats read disk information from “proc” ?
No, libvirt reads it from the qemu QMP command query-blockstats.  Beyond
that, you'll have to investigate how qemu comes up with that number.


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Just to clarify my problems:
For each KVM virtual machine, it is actually a process running on the host. 
The $pid in "/proc/$pid/io" represents the VM process’s PID. All my
work
is done on the host, not inside the VMs.  Take the VM process’s pid is “pid”, 
What the /proc/$pid/io records is io statistics of the VM from the host’s 
perspective. I understand that qemu might do some I/O coalescing kind of
work, but /proc/$pid/io reveals the actual IOs submitted to the host kernel.

For example, through “ps” command, i get the following info of one of my 
VMs: (the pid of the VM process is 4438)

qemu      4438     1  0 Jun24 ?        00:06:25 /usr/bin/qemu-system-x86_64 ...

[unrelevant data fields are ignored]
Command line "virsh domblkstat $vmname vda” get the results below:
	vda rd_bytes 639415808
	vda wr_bytes 728186880

Command line "cat /proc/4438/io” get the result below:
	read_bytes: 772415488 
	write_bytes: 734040064

I execute the above two commands consecutively using “&&", but we
can
that the results vary a lot, especially for the read_bytes. 

Since libvirt virDomainBlockStats API reads data from qemu  i should ask
the qemu folks for the answers. Great thanks, anyway.



On Jun 25, 2014, at 3:45 AM, Eric Blake <eblake@redhat.com> wrote:
> On 06/24/2014 02:24 AM, coperd wrote:
> 
>> 
>> It is supposed that the VM’s disk “rd_bytes" and “wr_bytes"
should
> never exceed the value of the physical disk respectively, right ?
> 
> That's a question for the qemu folks.
> 
>> Doesn’t
>> libvirt virDomainBlockStats read disk information from “proc” ?
> 
> No, libvirt reads it from the qemu QMP command query-blockstats.  Beyond
> that, you'll have to investigate how qemu comes up with that number.
> 
> 
> -- 
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>