Libguestfs - May 2021 - libnbd flamegraph Odd benchmark results

On Sun, May 30, 2021 at 12:13:28AM +0530, Abhay Raj Singh
wrote:
> That might be the reason
> 
> I have 8GB(7.7) of ram I verify 7.4 available by running in
> tty only mode and it gets filled pretty quickly (vmstat)
I think this comes back to something that Nir Soffer said last week
about number of requests and request size:

  https://listman.redhat.com/archives/libguestfs/2021-May/msg00121.html

I followed up with a discussion about L3 cache size vs default buffer size:

  https://listman.redhat.com/archives/libguestfs/2021-May/msg00122.html
  https://listman.redhat.com/archives/libguestfs/2021-May/msg00123.html

In short the default settings for nbdcopy are wrong and more
performance could be gained by having better settings.

The question (for you!) is how we can choose better settings, and
whether we should choose simply better defaults or have dynamic
settings.  My vision for nbdcopy is that it should work "as well as
possible" without end users needing to change the settings on the
command line.  But there may be cases where users need to or want to
tune particular settings.

But I'm not sure how to achieve that.  eg: Should we choose defaults
based on the number of cores and amount of memory detected at run
time?  Or is that too complicated, and maybe we can choose more
conservative defaults?  Really I'm looking for your answers here.

I do believe that qemu gets this wrong.  There are a bunch of command
line parameters that you have to set to get good performance, and
they're not even documented.
> Strange thing is that this also happens at size=1G.
I would imagine that the large default buffer size x number of buffers
would cause problems no matter the size of the virtual disk.

[...]
> -F doesn't affect the result much, it decreases the sampling rate to
avoid CPU/
> IO overload caused by perf.
My guess is sampling rate would not affect things very much, but I
didn't measure it so I could be wrong.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

On Sat, May 29, 2021 at 09:46:08PM +0100, Richard W.M. Jones
wrote:
> In short the default settings for nbdcopy are wrong and more
> performance could be gained by having better settings.
To be clearer, I mean that if we allocate (default) 32MB buffers and
have (default) 64 requests in flight and have (default) 4 connections
open, then in theory there could be 32 * 64 * 4 MB of memory being
used at once.  That is 8 GB -- although in reality it would probably
not be that much.

Also each single buffer is larger than my test machine's L3 cache.

Then there's the problem that Nir also pointed out that we don't reuse
these memory buffers, so we depend on whatever malloc() does, and may
or may not be reusing the memory in a cache-efficient way.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

> The question (for you!) is how we can choose better settings, and
> whether we should choose

 In my opinion configuration resolution will be best if resolved in the
following order of increasing priority/weightage.
1. Auto-detect depending on the system parameters.
Auto-detection of best params is a must IMO as there's
enough diversity in server hardware that one size won't fit
all can be implemented without much effort(excluding benchmarking).

2. Configuration file: A yaml, toml, or any other simple format
- Load a different config file with *nbdcopy --file=custom-config.yaml*
- Maybe generate the default config file having values from the auto-detect
phase.
nbdcopy --generate-config

~/.config/nbdcopy/config.yaml (used to get values)
~/.config/nbdcopy/example.yaml (generated for reference from auto)

3. CLI params
A quick way to override configuration file params
ex: functionally same
`nbdcopy --requests=4 --config=custom-config.yaml`
`nbdcopy --config=custom-config.yaml --requests=4`

The flow of initialization:
Read config file ----override ---> CLI params --- missing params -->
auto-detect fill
or
easier implementation:
auto-detect fill --- override ---> config file --- override ---> CLI
params


The rationale for choosing configuration file method:
- Easier to use avoiding long commands (less typing, scripts easier to read)
- Easy switch between multiple configurations and override them.
- Documentation and configuration in the same place
So easier for someone else to understand why certain parameters were chosen.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20210530/6468314d/attachment.htm>