Richard W.M. Jones
2021-Jun-07 16:14 UTC
[Libguestfs] ANNOUNCE: libnbd 1.8 and nbdkit 1.26 - high performance network block device client and server
I'm pleased to announce the release of nbdkit 1.26, a high performance
plugin-based Network Block Device (NBD) server, and libnbd 1.8, a high
performance client library.
https://en.wikipedia.org/wiki/Network_block_device
You can get these from:
tarballs: https://download.libguestfs.org/nbdkit/
https://download.libguestfs.org/libnbd/
git: https://gitlab.com/nbdkit
Key features of nbdkit:
* Multithreaded NBD server written in C with good performance.
* Minimal dependencies for the basic server.
* Liberal license (BSD) allows nbdkit to be linked to proprietary
libraries or included in proprietary code.
* Well-documented, simple plugin API with a stable ABI guarantee.
Lets you export ?unconventional? block devices easily.
* You can write plugins in C, Go, Lua, Perl, Python, OCaml, Ruby,
Rust, shell script or Tcl.
* Filters can be stacked in front of plugins to transform the output.
Key features of libnbd:
* Synchronous API for ease of use.
* Asynchronous API for writing non-blocking, multithreaded clients.
You can mix both APIs freely.
* High performance.
* Minimal dependencies for the basic library.
* Well-documented, stable API.
* Bindings in several programming languages.
* Shell (nbdsh) for command line and scripting.
* Copying tool (nbdcopy) for high performance copying and streaming.
* Query tool (nbdinfo) to query NBD servers.
* FUSE support (nbdfuse) to mount NBD in the local filesystem.
Release notes for nbdkit 1.26
nbdkit is now hosted on gitlab:
https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html
Security
There were no security issues found. All past security issues and
information about how to report new ones can be found in
nbdkit-security(1).
Plugins
nbdkit-gzip-plugin and nbdkit-tar-plugin have been removed. They were
deprecated in nbdkit 1.22. The functionality has been replaced by
nbdkit-gzip-filter(1) and nbdkit-tar-filter(1) respectively.
nbdkit-streaming-plugin(1) has been deprecated and will be removed in
nbdkit?1.30. You can use nbdcopy(1) instead.
nbdkit-curl-plugin(1) adds these features:
"followlocation=false" to
disable following redirects. "cookiefile" and
"cookiejar" to control
saving cookies.
nbdkit-ssh-plugin(1) now advertises multi-conn if the server is
OpenSSH, improving performance (Eric Blake).
nbdkit-ondemand-plugin(1) has a new "wait=true" flag allowing
you to
choose between clients waiting for an export to become free or being
immediately rejected.
nbdkit-sparse-random-plugin(1) now does not generate fully random data
within each block. Using perf we observed that the plugin was spending
34% of the total time during benchmarks just doing this. You can
enable the old behaviour using "random-content=true".
nbdkit-file-plugin(1) now checks if the file is read-only and exports a
read-only NBD connection in this case (Eric Blake).
nbdkit-vddk-plugin(1) has been tested with VDDK 7.0.2.
Filters
There is a new API for use by filters to manage "contexts".
Instead of
the previous 1-1 relationship between filter connections and plugin
connections, filters can now open more than one context into the
backend from a single connection, or share contexts between connections
(Eric Blake).
New nbdkit-multi-conn-filter(1) which allows you to add multi-conn
support to plugins which don't support it (although the emulation is
expensive), or change whether multi-conn is advertised to clients (Eric
Blake).
nbdkit-cow-filter(1) now ignores flush/FUA requests (which are not
useful for a temporary overlay). The filter now implements extents and
efficient trimming. Fine-grained locking is used around shared
structures, greatly improving parallel performance. Added support for
multi-conn.
nbdkit-cache-filter(1) advertises multi-conn when possible (Eric
Blake).
nbdkit-cache-filter and nbdkit-cow-filter can now handle arbitrary
sized disks, instead of rounding the disk size down.
nbdkit-blocksize-filter(1) supports parallel requests (Eric Blake).
nbdkit-checkwrite-filter(1) supports multi-conn (Eric Blake).
nbdkit-ext2-filter(1) no longer supports multi-conn. The filter is
single-threaded so it does not matter, but it was not safe. Fix trim,
zero and flush support (Eric Blake).
nbdkit-readahead-filter(1) fixes multi-conn support (Eric Blake).
nbdkit-gzip-filter(1) and nbdkit-xz-filter(1) now both support multi-
conn (Eric Blake).
Language bindings
nbdkit-python-plugin(3) supports the following new features:
"after_fork" and "preconnect" methods.
"nbdkit.shutdown()" binding.
nbdkit-rust-plugin(3) updated for newer Rust compiler versions (Alan
Somers).
nbdkit-golang-plugin(3) now uses and requires golang modules. Golang ?
1.13 must be used.
API
New ".cleanup" callback for plugins and filters. This is
called
shortly before ".unload", but unlike ".unload" it is
guaranteed that no
plugin or filter code has been dlclosed (Eric Blake).
New "nbdkit_extents_full" is a helper function for filters
which can be
used to read all extents from a plugin covering a region.
Tests
Old plugins that we use for testing were rebuilt so they no longer
depend on libselinux (thanks Thomas Wei?schuh).
nbdkit-ssh-plugin(1) can still be built even if ssh-keygen is
unavailable for testing it (thanks Thomas Wei?schuh).
Improve tests/test-captive.sh to avoid crashes during testing.
Test coverage on FreeBSD has been improved, fewer tests should skip
unnecessarily.
Build
nbdkit and plugins can now be built with "-fvisibility=hidden",
which
allows the compiler to make some optimizations.
Add a dependency so nbdkit is relinked if the linker script
(nbdkit.syms) changes (Eric Blake).
xorriso is now preferred over genisoimage or mkisofs (see
https://wiki.debian.org/genisoimage).
It is now possible to compile nbdkit with clang with warnings enabled.
Several warnings have been suppressed or fixed.
"MALLOC_CHECK_" and "MALLOC_PERTURB_" are only
enabled when running the
tests, not when running nbdkit from the build directory. This makes it
easier to do benchmarks.
"nbdkit_peer_*" implemented on OpenBSD.
Internals
When fuzzing nbdkit we now recommend using the "-t 1" flag
(which
disables threads). This improves the fuzz-stability of nbdkit.
AUTHORS
Authors of nbdkit 1.26:
? Alan Somers
? Eric Blake
? Michael Ablassmeier
? Richard W.M. Jones
? Timm B?der
Release notes for libnbd 1.8
libnbd is now hosted on gitlab:
https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html
Security
If you find a security issue, please read SECURITY in the source
(online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY).
To find out about previous security issues in libnbd, see
libnbd-security(3).
There was one security bug found in libnbd during this release cycle.
CVE-2021-20286 denial of service when using nbd_set_opt_mode(3)
See the full announcement here:
https://listman.redhat.com/archives/libguestfs/2021-March/msg00092.html
(Found and fixed by Eric Blake).
New APIs
nbd_get_private_data(3)
nbd_set_private_data(3)
These calls allow you to store either an unsigned integer or a
pointer in the handle for the application to use for its own
purposes.
nbd_get_uri(3)
This call constructs an NBD URI which can be used to connect back
to the same server (using nbd_connect_uri(3) or from other tools
that support the NBD URI standard).
Enhancements to existing APIs
Numeric IPv6 URIs (like "nbd://[::1]/") are now supported.
libnbd gives a better error message if the server backlog overflows
(thanks Xin Long, Lukas Doktor, Eric Blake, Martin Kletzander).
New features
Continuous integration (CI) tests now run on every commit and merge
request to the upstream repository (Martin Kletzander).
Tools
A great deal of work has been done to improve the performance of
nbdcopy(1) (Nir Soffer, Eric Blake).
On Linux, nbdcopy now uses readahead when reading from files, and takes
steps to preserve the host page cache.
nbdcopy new --request-size option to allow controlling the size of
requests made to NBD servers. Adjusting this can affect performance
(Nir Soffer).
nbdcopy now supports "null:" pseudo-target, where it discards
the
output. This is useful for benchmarking.
nbdcopy new -v flag to enable libnbd and other debugging.
nbdinfo(1) --list (which lists all exports of a server) no longer exits
early if one of the exports is inaccessible or there is a recoverable
error (Eric Blake).
nbdinfo now prints the URI of each export in the output making it
easier to connect back to each export.
nbdfuse(1) is now multithreaded and supports issuing parallel NBD
commands on a single connection as well as multi-conn, and other
enhancements have been made. This resulted in around 50% performance
improvement when tested with fio.
nbdfuse now supports nbdcopy-style "[ CMD ]", eg:
nbdfuse dir/ramdisk [ nbdkit memory 1G ]
nbdfuse now supports trimming. Also it supports fast zeroing, but this
requires a forthcoming kernel patch before it will work.
nbdfuse sets the virtual file permissions appropriately if the NBD
export is read-only.
nbdfuse new -v flag to enable libnbd and other debugging.
Tests
Added a test of interoperability with qemu-storage-daemon(1). Note
that qemu-storage-daemon was broken in qemu 6.0.0 so this test can fail
unless you update to a fixed version (thanks Stefan Hajnoczi, Daniel
Berrang?).
Added a test of nbdcopy with TLS which was previously untested.
Python code style tests are now opt-in using
./configure --enable-python-code-style
More tests should now run on FreeBSD and not be skipped.
nbdfuse tests now ensure that errors from NBD are transmitted through
FUSE back to the local client.
"MALLOC_CHECK_" and "MALLOC_PERTURB_" are only
enabled when running the
tests, not when running libnbd programs from the build directory. This
makes it easier to do benchmarks.
Other improvements and bug fixes
The minimum C compiler is now ISO C99 (previously ANSI C90).
FUSE 3 (instead of FUSE 2) is now required to build nbdfuse.
The library tries harder not to leak thread-local storage memory on
dlclose or exit. (Because of the design of Linux it is not always
possible to avoid this, especially in multithreaded programs.)
Fix the progress bar in nbdcopy(1).
UTF-8 export names are now processed and displayed properly by
nbdinfo(1).
The --help output of nbdcopy, nbdfuse and nbdinfo now displays a brief
summary of all options and has some examples.
The generator now only requires ocamlc, not ocamlfind (Abhay Raj
Singh).
Golang bindings now use and require golang modules.
Documentation
Document using libnbd with the Meson build system.
Added Python examples.
Add example of how to integrate libnbd with libev (Nir Soffer).
AUTHORS
Eric Blake
Richard W.M. Jones
Martin Kletzander
Nir Soffer
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW