search for: statsfile

- Use more friendly output with GiB and MiB/s. - Measure time per operation, providing finer grain stats - Add missing stats for flush I hope that these changes will help to understand and imporve virt-v2v performance. Nir Soffer (3): filters: stats: Show size in GiB, rate in MiB/s filters: stats: Measure time per operation filters: stats: Add flush stats filters/stats/stats.c | 117

...ommon disk images, and MiB/s works for common > storage throughput. > > Here is an example run with this change: > > $ ./nbdkit --foreground \ > --unix /tmp/nbd.sock \ > --exportname '' \ > --filter stats \ > file file=/var/tmp/dst.img \ > statsfile=/dev/stderr \ > --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' > (100.00/100%) > elapsed time: 2.313 s > write: 1271 ops, 1.14 GiB, 502.63 MiB/s > zero: 1027 ops, 4.86 GiB, 2153.24 MiB/s > extents: 1 ops, 2.00 GiB...

...ddk file="[datastore1] Fedora 28/Fedora 28.vmdk" \ libdir=vmware-vix-disklib-distrib \ server=vmware user=root password=+/tmp/passwd \ thumbprint=xyz \ vm=moref=3 \ --filter=stats statsfile=/dev/stderr \ --run ' unset LD_LIBRARY_PATH /home/rjones/d/qemu/qemu-img convert -p $nbd /var/tmp/out ' (with extra filters added to the command line as appropriate for each test). > I think on slow CPU and fast disk this might be even bigger &...

...ore useful way is to measure the time we spent in each > operation. > > Here is an example run with this change: > > $ ./nbdkit --foreground \ > --unix /tmp/nbd.sock \ > --exportname '' \ > --filter stats \ > file file=/var/tmp/dst.img \ > statsfile=/dev/stderr \ > --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' > (100.00/100%) > elapsed time: 2.150 s > write: 1271 ops, 1.14 GiB, 0.398 s, 2922.22 MiB/s > zero: 1027 ops, 4.86 GiB, 0.012 s, 414723.03 MiB/s > ext...

- Use more friendly output with GiB and MiB/s - Measure time per operation, providing finer grain stats - Add total stats for understanding system throughput - Add missing stats for flush I hope that these changes will help to understand and improve virt-v2v performance. Changes since v1: - Keep bytes values - Increase precision to 0.001 GiB and 0.001 MiB/s - Add total stats - Show time before

...dora 28.vmdk" \ > > libdir=vmware-vix-disklib-distrib \ > > server=vmware user=root password=+/tmp/passwd \ > > thumbprint=xyz \ > > vm=moref=3 \ > > --filter=stats statsfile=/dev/stderr \ > > --run ' > > unset LD_LIBRARY_PATH > > /home/rjones/d/qemu/qemu-img convert -p $nbd /var/tmp/out > > ' > > > >(with extra filters added to the command line as appropriate for each > >test). > &gt...

...o parse. Add also size in GiB, and show rate in MiB per second. This works well for common disk images and storage. Here is an example run with this change: $ ./nbdkit --foreground \ --unix /tmp/nbd.sock \ --exportname '' \ --filter stats \ file file=/var/tmp/dst.img \ statsfile=/dev/stderr \ --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' (100.00/100%) elapsed time: 2.055 s write: 1271 ops, 1219244032 bytes, 1.136 GiB, 565.710 MiB/s zero: 1027 ops, 5223206912 bytes, 4.864 GiB, 2423.486 MiB/s extents: 1 ops...

flush is usually called once but it can take significant time, so we need to include it. Here is an example run with this change: $ ./nbdkit --foreground \ --unix /tmp/nbd.sock \ --exportname '' \ --filter stats \ file file=/var/tmp/dst.img \ statsfile=/dev/stderr \ --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' (100.00/100%) total: 2299 ops, 2.172 s, 6442450944 bytes, 6.000 GiB, 2829.295 MiB/s write: 1271 ops, 0.356 s, 1219244032 bytes, 1.136 GiB, 3269.711 MiB/s zero: 1027 ops,...

...n each > > operation. > > > > Here is an example run with this change: > > > > $ ./nbdkit --foreground \ > > --unix /tmp/nbd.sock \ > > --exportname '' \ > > --filter stats \ > > file file=/var/tmp/dst.img \ > > statsfile=/dev/stderr \ > > --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' > > (100.00/100%) > > elapsed time: 2.150 s > > write: 1271 ops, 1.14 GiB, 0.398 s, 2922.22 MiB/s > > zero: 1027 ops, 4.86 GiB, 0.012 s,...

...and hard to parse. Using GiB for sizes works for common disk images, and MiB/s works for common storage throughput. Here is an example run with this change: $ ./nbdkit --foreground \ --unix /tmp/nbd.sock \ --exportname '' \ --filter stats \ file file=/var/tmp/dst.img \ statsfile=/dev/stderr \ --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' (100.00/100%) elapsed time: 2.313 s write: 1271 ops, 1.14 GiB, 502.63 MiB/s zero: 1027 ops, 4.86 GiB, 2153.24 MiB/s extents: 1 ops, 2.00 GiB, 885.29 MiB/s --- filters/st...

...arge > writes into smaller chunks, to magnify the effects of write delays and > allow testing to provide obvious results with a smaller image) > delay delay-write=20ms delay-zero=5ms (also to magnify the effects on a > smaller image, with writes penalized more than zeroing) > stats statsfile=/dev/stderr (to track overall time and a decent summary > of how much I/O occurred). > noextents (forces the entire image to report that it is allocated, > which eliminates any testing variability based on whether qemu-img uses > that to bypass a zeroing operation [1]) I can't hel...

...store1] Fedora 28/Fedora 28.vmdk" \ > libdir=vmware-vix-disklib-distrib \ > server=vmware user=root password=+/tmp/passwd \ > thumbprint=xyz \ > vm=moref=3 \ > --filter=stats statsfile=/dev/stderr \ > --run ' > unset LD_LIBRARY_PATH > /home/rjones/d/qemu/qemu-img convert -p $nbd /var/tmp/out > ' > >(with extra filters added to the command line as appropriate for each >test). > >> I think on slow CPU and f...

...umming ops, bytes of other operations, and using the total time. This works well for copying images to/from nbdkit. Here is an example run with this change: $ ./nbdkit --foreground \ --unix /tmp/nbd.sock \ --exportname '' \ --filter stats \ file file=/var/tmp/dst.img \ statsfile=/dev/stderr \ --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' (100.00/100%) total: 2299 ops, 2.190 s, 6442450944 bytes, 6.000 GiB, 2804.977 MiB/s write: 1271 ops, 0.359 s, 1219244032 bytes, 1.136 GiB, 3237.466 MiB/s zero: 1027 ops,...

...s is incorrect and hides the real throughput. A more useful way is to measure the time we spent in each operation. Here is an example run with this change: $ ./nbdkit --foreground \ --unix /tmp/nbd.sock \ --exportname '' \ --filter stats \ file file=/var/tmp/dst.img \ statsfile=/dev/stderr \ --run 'qemu-img convert -p -n -f raw -O raw -T none /var/tmp/fedora-30.img nbd:unix:/tmp/nbd.sock' (100.00/100%) elapsed time: 2.150 s write: 1271 ops, 1.14 GiB, 0.398 s, 2922.22 MiB/s zero: 1027 ops, 4.86 GiB, 0.012 s, 414723.03 MiB/s extents: 1 ops, 2.00 GiB, 0.000 s...

I ran some local testing on my Fedora 29 system via: $ ./nbdkit memory 1m $ for i in `seq 10`; do ./nbdkit -U - --filter=stats nbd \ statsappend=true statsfile=$file hostname=localhost port=10809 \ --run '~/libnbd/examples/threaded-reads-and-writes $unixsocket' done Pre-patch, the runs averaged 1.266s, 1.30E+08 bits/s Post-patch, the runs averaged 1.154s, 1.42E+08 bits/s This is roughly 9% performance gain, all because semaphores are li...

Hi, I''m trying to configure IMQ to work on the same machine with frottle (http://frottle.sourceforge.net). The problem is both feed themselves packets through netfilter queueing mechanism, but currently there can only be one netfilter queue per protocol family. To explain why I need IMQ in the first place I have to explain what frottle does. It is a deamon that tweaks the behaviour of a

As I've spent really too long today investigating this, I want to document this in a public email, even though there's nothing really that interesting here. One thing you find from search for VDD 6.7 / VixDiskLib_QueryAllocatedBlocks issues with Google is that we must be one of the very few users out there. And the other thing is that it's quite broken. All testing was done using

I like this change. I think we were overloading the config_complete method before to do two different things (complete configuration; do any allocation/housekeeping necessary before we can start serving). The only questions in my mind are whether we want this before 1.18, and whether the name ("get_ready") is a good one. Rich.

...ro requests, but forces large writes into smaller chunks, to magnify the effects of write delays and allow testing to provide obvious results with a smaller image) delay delay-write=20ms delay-zero=5ms (also to magnify the effects on a smaller image, with writes penalized more than zeroing) stats statsfile=/dev/stderr (to track overall time and a decent summary of how much I/O occurred). noextents (forces the entire image to report that it is allocated, which eliminates any testing variability based on whether qemu-img uses that to bypass a zeroing operation [1]) So here's my one-time setup, fo...

There's enough here to need a review; some of it probably needs backporting to stable-1.12. This probably breaks tests on Haiku or other platforms that have not been as on-the-ball about atomic CLOEXEC; feel free to report issues that arise, and I'll help come up with workarounds (even if we end up leaving a rare fd leak on less-capable systems). Meanwhile, I'm still working on my