Richard W.M. Jones
2023-Sep-27 12:48 UTC
[Libguestfs] ANNOUNCE: nbdkit 1.36 and libnbd 1.18 released
I'm pleased to announce new stable releases of nbdkit 1.36 and libnbd 1.18.
nbdkit is a Network Block Device (NBD) server with a stable plugin ABI
and a permissive license. libnbd is an NBD client library in
userspace.
Among the major new features are: Rust bindings for libnbd, 64 bit
extended headers (NBD protocol extension) support in libnbd, a
completely rewritten curl plugin for nbdkit, qcow2 support for nbdkit.
Complete release notes are attached below.
nbdkit 1.36.0 can be downloaded here:
https://download.libguestfs.org/nbdkit/1.36-stable/
libnbd 1.18.0 can be downloaded here:
https://download.libguestfs.org/libnbd/1.18-stable/
Release notes for nbdkit 1.36 online:
https://libguestfs.org/nbdkit-release-notes-1.36.1.html
Release notes for libnbd 1.18 online:
https://libguestfs.org/libnbd-release-notes-1.18.1.html
Rich.
----------------------------------------------------------------------
nbdkit-release-notes-1.36 - release notes for nbdkit 1.36
These are the release notes for nbdkit stable release 1.36. This
describes the major changes since 1.34.
nbdkit 1.36.0 was released on 27 September 2023.
Security
No security issues were identified in this release.
All past security issues and information about how to report new ones
can be found in nbdkit-security(1).
Plugins
New nbdkit-ones-plugin(1) which creates a fully allocated disk
containing all 0xff (all ones), or another byte of your choice.
nbdkit-curl-plugin(1) now uses a curl "multi" interface. This
enables
much better performance, and also allows the curl plugin to handle
requests in parallel. The curl plugin now falls back to making a
"GET"
request to get the size of the remote file for certain servers which do
not support "HEAD" requests. This plugin adds new options:
"ipresolve"
(force IPv4 or IPv6), "resolve" (force a particular IP
address), -D
curl.times=1 (print detailed timing stats), and -D curl.verbose.ids=1
(display connection and transfer IDs).
nbdkit-memory-plugin(1) now uses a read-write lock to protect internal
structures, resulting in improved performance for mostly read
workloads.
nbdkit-data-plugin(1) now has more optimizations.
nbdkit-file-plugin(1) now supports 4k sector sizes on Windows (Brian
Carnes).
Filters
New nbdkit-evil-filter(1) adds random but consistent data corruption to
the underlying plugin.
New nbdkit-qcow2dec-filter(1) which can decode qcow2 files (but not
write to them).
nbdkit-ip-filter(1) can now filter by client SELinux label.
nbdkit-partition-filter(1) now supports 4k sector sizes (Brian Carnes).
nbdkit-retry-request-filter(1) allows the "get_size" operation
to be
retried.
nbdkit-tar-filter(1) adds new "tar-limit" parameter which can
be used
to ensure the filter does not read indefinite amounts of input when
opening the tar file.
Filters can now append their own output to nbdkit --dump-plugin output.
Language bindings
Rust bindings add support for "after_fork",
"block_size",
"nbdkit_debug", "nbdkit_is_tls",
"nbdkit_parse_size",
"nbdkit_parse_bool" and "nbdkit_parse_probability".
The "open" method
can now return an error; note this is not backwards compatible and
requires a small source code change to Rust plugins. (Thanks Alan
Somers)
nbdkit-ocaml-plugin(3) now supports OCaml 5.
OCaml bindings add support for "nbdkit_stdio_safe",
"nbdkit_is_tls",
"nbdkit_peer_name" and
"nbdkit_peer_security_context".
nbdkit-perl-plugin(3) now supports Perl 5.38.
Shell script plugins (nbdkit-sh-plugin(3)) may now ignore stdin in
their "pwrite" method, whereas previously it was required to
read and
discard stdin along error paths (thanks Eric Blake).
Server
When using the --run option, the default is now to use a private Unix
domain socket (as if -U - was specified), whereas in nbdkit ? 1.34 the
default was to open a TCP port. The new default reflects the most
common and safest way to use the --run option. You can find out if
nbdkit has the new behaviour by checking "nbdkit --dump-config"
and
looking for "run_default_socket=Unix" in the output.
Debug strings containing control codes and other non-printable
characters are now escaped properly.
New flag -D nbdkit.environ=1 can be used to dump the server environment
in debug output.
API
New "nbdkit_parse_probability" function which can be used to
parse
probabilities in various formats, like "10%" or
"1:10".
New "nbdkit_peer_security_context" function which returns the
security
context (usually SELinux label) of the client.
Bug fixes
Fix long-standing double-free in nbdkit-file-plugin(1) when the client
closes the connection during negotiation (thanks Eric Blake).
nbdkit-file-plugin now has improved error messages when you try to open
a file with an empty name ("").
Documentation
Miscellaneous documentation improvements for Windows (Brian Carnes).
Fuzzing instructions updated for Clang 17, AFL++ 4.08c.
Tests
Ongoing CI maintenance and fixes (Laszlo Ersek, Eric Blake).
Many fixes in the test suite for MinGW and Wine.
Several internal consistency checks were added to the test suite, to
ensure, for example, that we always use "requires_run" when the
test
uses the nbdkit --run option.
Build
Optionally "zlib-ng" can be used (instead of "zlib")
for faster
decompression in various places.
No longer link with msvcrt on Windows.
"awk" is a build requirement. It was implicitly required
before
(because it is needed by the "./configure" script generated by
autoconf), but now the dependency is explicit (Eric Blake).
Internals
"podwrapper" has a new --replace parameter which makes it
easier to use
"AC_SUBST"-substitutions in documentation.
SEE ALSO
nbdkit(1).
AUTHORS
Authors of nbdkit 1.36:
Alan Somers
Brian Carnes
Eric Blake
Laszlo Ersek
Richard W.M. Jones
----------------------------------------------------------------------
libnbd-release-notes-1.18 - release notes for libnbd 1.18
These are the release notes for libnbd stable release 1.18. This
describes the major changes since 1.16.
libnbd 1.18.0 was released on 27 September 2023.
Security
Eric Blake found a case where a server could cause libnbd to crash,
although not in the normal libnbd configuration. For more information
see the announcement here:
https://listman.redhat.com/archives/libguestfs/2023-July/032035.html
Eric Blake found an issue with nbd_get_size(3) where a server that
returns a size > (uint64_t)INT64_MAX and < (uint64_t)-1 would cause
nbd_get_size to return a negative number (which is not -1 and so
callers may or may not treat it as an error). While no client code in
libnbd itself is affected by this, it could affect external clients.
libnbd ? 1.16.5 now returns an error (-1) and sets nbd_get_errno to
"EOVERFLOW" in this case. We have applied for a decision on a
CVE.
See the announcement here:
https://listman.redhat.com/archives/libguestfs/2023-September/032711.html
During routine fuzzing we found several security problems which had
been introduced during this development cycle and have subsequently
been fixed. Stable (even numbered) releases of libnbd should not be
vulnerable; do not use the development (odd numbered) releases in
production.
If you find a security issue, please read SECURITY in the source
(online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY).
To find out about previous security issues in libnbd, see
libnbd-security(3).
New APIs
nbd_block_status_64(3)
nbd_aio_block_status_64(3)
Make a 64 bit block status request, see "Protocol" below
(Eric
Blake).
nbd_block_status_filter(3)
nbd_aio_block_status_filter(3)
Send filtered block status command, see "Protocol" below
(Eric
Blake).
nbd_can_block_status_payload(3)
Find out if the server supports filtered block status command (Eric
Blake).
nbd_set_request_extended_headers(3)
nbd_get_request_extended_headers(3)
nbd_get_extended_headers_negotiated(3)
nbd_opt_extended_headers(3)
nbd_aio_opt_extended_headers(3)
Set/get whether we request extended headers from the server, and
find out if we negotiated extended headers, see "Protocol"
below
(Eric Blake).
Enhancements to existing APIs
"qemu:" meta-context constants (eg.
"qemu:dirty-bitmap" as
"LIBNBD_CONTEXT_QEMU_DIRTY_BITMAP") are now available through
the C,
Golang, OCaml and Python language bindings (Eric Blake).
nbd_shutdown(3) now works correctly when in opt mode (Eric Blake).
nbd_set_string(3) adds "LIBNBD_STRICT_AUTO_FLAG" which allows
the
client to test how servers behave when the payload length flag is
adjusted (Eric Blake).
Protocol
libnbd now supports NBD 64 bit "extended headers" and extent
sizes. In
practice this allows certain requests such as zeroing very large
sections of the disk to be implemented much more efficiently, with
servers that support this (Eric Blake).
libnbd now supports filtered block status requests (Eric Blake).
Tools
nbdcopy(1), nbdinfo(1) and nbddump(1) have been expanded to use and
report NBD 64 bit / extended header support when the server supports it
(Eric Blake).
nbdinfo --has can be used as an alias for --can, eg.
"nbdinfo --has structured-reply URI" (Eric Blake).
nbdinfo makes the export size output optional, for servers which send
an oversized one (Eric Blake).
nbdcopy now supports "human sizes" for some parameters, eg.
"nbdcopy --request-size=1M ...".
Language bindings
New Rust bindings. There is a basic API for ordinary use, and an
asynch API implemented using Tokio. Rust ? 1.69 is required. (Tage
Johansson, supported by sponsorship from Google Summer of Code 2023,
additional review and fixes by Eric Blake).
OCaml 5 is now supported.
Golang 1.21 is now supported. The minimum version is now 1.17 (Eric
Blake).
Use "gofmt" to format Golang bindings (Eric Blake).
Use "unsafe.Slice" when converting C arrays to Golang slices
(Eric
Blake).
All language bindings support NBD 64 bit / extended headers, and
examples of how to use this feature are available (Eric Blake).
Tests
Fix a couple of race conditions in tests where we did not fully consume
stdin in nbdkit-sh-plugin(3) pwrite method (Eric Blake).
Fuzzing now disables client-side strictness checks, enabling a wider
range of inputs to be fuzzed (Eric Blake).
Other improvements and bug fixes
Consistently wrap source code at 80 columns (Laszlo Ersek).
Debug messages no longer print the very verbose state transitions
inside the state machine as these are not usually useful. You can
reenable this by defining "-DLIBNBD_STATE_VERBOSE=1" at compile
time.
Completion ".callback" methods are now always called exactly
once, and
documentation is clearer on when this happens (Eric Blake).
Documentation
"podwrapper" has a new --replace parameter which makes it
easier to use
"AC_SUBST"-substitutions in documentation.
Build
Automake's "subdir-objects" option is now used (Eric
Blake).
Multiple, ongoing fixes to the CI tests (Eric Blake).
SEE ALSO
libnbd(3).
AUTHORS
Eric Blake
Laszlo Ersek
Richard W.M. Jones
Tage Johansson
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html