Libguestfs - Nov 2019 - ANNOUNCE: libnbd 1.2 & nbdkit 1.16 - high performance NBD client and server

I'm pleased to announce the releases of libnbd 1.2 and nbdkit 1.16.
These are a high performance Network Block Device (NBD) client library
and server.


Key features of libnbd:

 * Synchronous API for ease of use.
 * Asynchronous API for writing non-blocking, multithreaded clients.
   You can mix both APIs freely.
 * High performance.
 * Minimal dependencies for the basic library.
 * Well-documented, stable API.
 * Bindings in several programming languages.
 * Shell (nbdsh) for command line and scripting.

Git: https://github.com/libguestfs/libnbd
Download: http://download.libguestfs.org/libnbd/1.2-stable/
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=28807


Key features of nbdkit:

 * Multithreaded NBD server written in C with good performance.
 * Minimal dependencies for the basic server.
 * Liberal license (BSD) allows nbdkit to be linked to proprietary
   libraries or included in proprietary code.
 * Well-documented, simple plugin API with a stable ABI guarantee.
   Lets you export “unconventional” block devices easily.
 * You can write plugins in C, Lua, Perl, Python, OCaml, Ruby, Rust,
   shell script or Tcl.
 * Filters can be stacked in front of plugins to transform the output.

Git: https://github.com/libguestfs/nbdkit
Download: http://download.libguestfs.org/nbdkit/1.16-stable/
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=16469


*** Release notes for libnbd 1.2 ***

       These are the release notes for libnbd stable release 1.2.  This
       describes the major changes since 1.0.

       libnbd 1.2.0 was released on 14th November 2019.

   Security
       Two security problems were found during development of libnbd 1.2.
       Both were backported to the 1.0 stable branch.  Upgrading is highly
       advisable.

       CVE-2019-14842 protocol downgrade attack when using
       "LIBNBD_TLS_REQUIRE"

       See the full announcement and links to mitigation, tests and fixes
       here:
       https://www.redhat.com/archives/libguestfs/2019-September/msg00128.html

       remote code execution vulnerability

       See the full announcement here:
       https://www.redhat.com/archives/libguestfs/2019-October/msg00060.html

   New APIs
       nbd_can_fast_zero(3)
           Test support by the server for fast zeroing (Eric Blake).

       nbd_connect_socket(3)
       nbd_aio_connect_socket(3)
           Connect to a local connected socket which you create in your main
           program using your own chosen method.

       nbd_connect_systemd_socket_activation(3)
       nbd_aio_connect_systemd_socket_activation(3)
           Connect to local processes that support systemd socket activation.

       nbd_connect_vsock(3)
       nbd_aio_connect_vsock(3)
           Used to connect to servers over "AF_VSOCK".

       nbd_get_handshake_flags(3)
       nbd_set_handshake_flags(3)
       nbd_get_request_structured_replies(3)
       nbd_set_request_structured_replies(3)
       nbd_get_structured_replies_negotiated(3)
           Can be used when testing NBD servers to avoid various NBD features
           (Eric Blake).

       nbd_get_protocol(3)
           Get the NBD protocol variant that the server supports.

       nbd_get_tls_negotiated(3)
           Did we actually negotiate a TLS connection?

       nbd_set_uri_allow_local_file(3)
       nbd_set_uri_allow_tls(3)
       nbd_set_uri_allow_transports(3)
           These can be used to filter NBD URIs before calling
           nbd_connect_uri(3).

   New features
       New tool nbdfuse(1) lets you create a loop-mounted file backed by an
       NBD server without needing root.

       "AF_VSOCK" is now a supported protocol (thanks Stefan Hajnoczi
and
       Stefano Garzarella).

       Support for the "FAST_ZERO" flag (Eric Blake).

       Allow disabling certain protocol features, to make it easier to test
       servers (Eric Blake).

       Stack-allocated Variable Length Arrays (VLAs) are now banned throughout
       the library, making the library easier to consume from threads and
       other small stack situations.

       Reproducible builds (Chris Lamb).

       Support for filtering potentially dangerous or undesirable NBD URI
       features.

   Documentation
       Many improvements to the generated manual pages, including:

       ·   Separate "RETURN VALUE" and "ERRORS" sections for
each API
           function.

       ·   Example code.

       ·   Relevant links can be added to the "SEE ALSO" section.

       ·   Link to NBD URI specification where relevant, and improve
           documentation around what URIs libnbd supports.

       ·   Document libnbd version number scheme.

       ·   Document limits on export name length, encoding etc.

       New libnbd-security(3) man page listing past security issues and
       remediations (Eric Blake).

   Tools
       nbdsh(1) has a new --base-allocation option which can be used to
       request "base:allocation" metadata context.

       New nbdsh(1) --uri (-u) option to connect to URIs.

   Tests
       You can now fuzz libnbd using either American Fuzzy Lop or clang’s
       libFuzzer.

       Add unit tests for nbdsh(1) (Eric Blake).

       Improved interop testing with various NBD servers and features.

   Other improvements and bug fixes
       nbd_connect_tcp(3) now tries to return the correct errno(3) from the
       underlying connect(2) call when that fails.

       The nbd-protocol.h header file is now shared between libnbd and nbdkit.

       Better fork-safety in "nbd_connect_*" APIs.

       The code was analyzed with Coverity and various problems identified and
       fixed.


*** Release notes for nbdkit 1.16 ***

       These are the release notes for nbdkit stable release 1.16.  This
       describes the major changes since 1.14.

       nbdkit 1.16.0 was released on 14th November 2019.

   Security
       Two security issues were found during development of nbdkit 1.16.
       Fixes for these were backported to older stable branches.  Upgrading to
       the fixed versions is highly recommended.  The new nbdkit-security(1)
       man page contains an up to date list of past security issues.

       CVE-2019-14850 denial of service due to premature opening of back-end
       connection

       See the full announcement and links to mitigation, tests and fixes
       here:
       https://www.redhat.com/archives/libguestfs/2019-September/msg00084.html

       CVE-2019-14851 assertion failure by issuing commands in the wrong order

       This CVE was caused by the fix to the previous issue.

       See the full announcement and links to mitigation, tests and fixes
       here:
       https://www.redhat.com/archives/libguestfs/2019-September/msg00272.html

   New features
       Add support for fast zeroing.  Plugins can expose this using the new
       ".can_fast_zero" method (Eric Blake).

       nbdkit-partitioning-plugin(1) allows use of "mbr-id=default" or
       "type-guid=default" to go back to the default MBR byte or
partition
       type GUID.

       New --mask-handshake server flag can be used for testing client feature
       negotiation (Eric Blake).

       The client export name is passed to nbdkit-captive(1) --run parameter
       as $exportname (Eric Blake).

       Captive --run commands which fail (eg. aborting) now cause nbdkit to
       exit with an error instead of errors being silently ignored (Eric
       Blake).

       File descriptors can be passed to password parameters, eg:
       "password=-3" which means that the password should be read from
file
       descriptor 3.

       nbdkit can now serve over the "AF_VSOCK" protocol (thanks
Stefan
       Hajnoczi).

       New --log=null option discards error messages.

   Plugins
       Python 2 support has been dropped from nbdkit-python-plugin(3) in line
       with Python 2 end of life at the beginning of 2020.  Python ≥ 3.3 is
       required by this plugin.  If you wish to continue to use Python 2 then
       you will need to use nbdkit 1.14.

       New nbdkit-info-plugin(1) which returns various server information back
       to the client.  It can be used for testing server latency amongst other
       things.

       nbdkit-data-plugin(1) now allows you to write "BYTE*N" to get
repeated
       bytes (eg. nbdkit data data="0x55*4096").

       nbdkit-ssh-plugin(1) new parameter "compression=true|false" to
control
       transport compression.

       nbdkit-vddk-plugin(1) is no longer compiled on non-x86 platforms since
       VMware has only ever shipped VDDK on x86.

       nbdkit-sh-plugin(1) scripts can now see the client exportname and can
       use the "magic_config_key" feature.

   Filters
       New nbdkit-retry-filter(1) which can reopen the plugin transparently on
       certain types of failures (lots of help from Eric Blake).

   API
       Macros "NBDKIT_VERSION_MAJOR",
"NBDKIT_VERSION_MINOR",
       "NBDKIT_VERSION_MICRO" expose the compile-time version of
nbdkit to
       plugins and filters (Eric Blake).

       Filters (which unlike plugins do not have a public stable API) must now
       exactly match the version of nbdkit when loaded (Eric Blake).

       New ".can_fast_zero" method (Eric Blake).

       New "nbdkit_export_name" server function for reading the export
name
       passed by the client.

       New "nbdkit_peer_name" server function to return the client
address
       (like getpeername(2)).

       New server functions for safely parsing integers:
"nbdkit_parse_int",
       "nbdkit_parse_unsigned", "nbdkit_parse_int8_t",
"nbdkit_parse_uint8_t",
       "nbdkit_parse_int16_t", "nbdkit_parse_uint16_t",
       "nbdkit_parse_int32_t", "nbdkit_parse_uint32_t",
       "nbdkit_parse_int64_t", "nbdkit_parse_uint64_t".

   Bug fixes
       ".trim" with FUA flag set now works (Eric Blake).

   Documentation
       The previous release notes have been turned into man pages.

   Tests
       Several tests now optionally use nbdsh(1) instead of qemu-io.

       You can now fuzz nbdkit using either American Fuzzy Lop or clang’s
       libFuzzer.

       Several tests have had sleep times increased to make them more stable
       when run on slow or heavily loaded machines.

   Internals
       Reproducible builds (Chris Lamb).

       Compile code with -Wshadow warning (Eric Blake).

       The internal backend system has been extensively overhauled.  In
       particular this means that we now validate request ranges as requests
       are passed between filters and down to the plugin, making it easier to
       find bugs in filters early (Eric Blake).

       Plugin size and "can_*" flags are cached more aggressively by
the
       server (Eric Blake).

       Variable Length Arrays (VLAs) on stack are now banned throughout the
       code.

       The nbd-protocol.h header describing the NBD protocol is now shared
       with libnbd(3).

       Plugin ".unload" method is now called after all worker threads
have
       exited, avoiding races at server shutdown.

       Code was audited using Coverity and various problems were fixed.




-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Very nice and congrats on getting this in.

Brett

On Thu, Nov 14, 2019 at 5:53 AM Richard W.M. Jones <rjones@redhat.com>
wrote:
> I'm pleased to announce the releases of libnbd 1.2 and nbdkit 1.16.
> These are a high performance Network Block Device (NBD) client library
> and server.
>
>
> Key features of libnbd:
>
>  * Synchronous API for ease of use.
>  * Asynchronous API for writing non-blocking, multithreaded clients.
>    You can mix both APIs freely.
>  * High performance.
>  * Minimal dependencies for the basic library.
>  * Well-documented, stable API.
>  * Bindings in several programming languages.
>  * Shell (nbdsh) for command line and scripting.
>
> Git: https://github.com/libguestfs/libnbd
> Download: http://download.libguestfs.org/libnbd/1.2-stable/
> Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=28807
>
>
> Key features of nbdkit:
>
>  * Multithreaded NBD server written in C with good performance.
>  * Minimal dependencies for the basic server.
>  * Liberal license (BSD) allows nbdkit to be linked to proprietary
>    libraries or included in proprietary code.
>  * Well-documented, simple plugin API with a stable ABI guarantee.
>    Lets you export “unconventional” block devices easily.
>  * You can write plugins in C, Lua, Perl, Python, OCaml, Ruby, Rust,
>    shell script or Tcl.
>  * Filters can be stacked in front of plugins to transform the output.
>
> Git: https://github.com/libguestfs/nbdkit
> Download: http://download.libguestfs.org/nbdkit/1.16-stable/
> Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=16469
>
>
> *** Release notes for libnbd 1.2 ***
>
>        These are the release notes for libnbd stable release 1.2.  This
>        describes the major changes since 1.0.
>
>        libnbd 1.2.0 was released on 14th November 2019.
>
>    Security
>        Two security problems were found during development of libnbd 1.2.
>        Both were backported to the 1.0 stable branch.  Upgrading is highly
>        advisable.
>
>        CVE-2019-14842 protocol downgrade attack when using
>        "LIBNBD_TLS_REQUIRE"
>
>        See the full announcement and links to mitigation, tests and fixes
>        here:
>
> https://www.redhat.com/archives/libguestfs/2019-September/msg00128.html
>
>        remote code execution vulnerability
>
>        See the full announcement here:
>
> https://www.redhat.com/archives/libguestfs/2019-October/msg00060.html
>
>    New APIs
>        nbd_can_fast_zero(3)
>            Test support by the server for fast zeroing (Eric Blake).
>
>        nbd_connect_socket(3)
>        nbd_aio_connect_socket(3)
>            Connect to a local connected socket which you create in your
> main
>            program using your own chosen method.
>
>        nbd_connect_systemd_socket_activation(3)
>        nbd_aio_connect_systemd_socket_activation(3)
>            Connect to local processes that support systemd socket
> activation.
>
>        nbd_connect_vsock(3)
>        nbd_aio_connect_vsock(3)
>            Used to connect to servers over "AF_VSOCK".
>
>        nbd_get_handshake_flags(3)
>        nbd_set_handshake_flags(3)
>        nbd_get_request_structured_replies(3)
>        nbd_set_request_structured_replies(3)
>        nbd_get_structured_replies_negotiated(3)
>            Can be used when testing NBD servers to avoid various NBD
> features
>            (Eric Blake).
>
>        nbd_get_protocol(3)
>            Get the NBD protocol variant that the server supports.
>
>        nbd_get_tls_negotiated(3)
>            Did we actually negotiate a TLS connection?
>
>        nbd_set_uri_allow_local_file(3)
>        nbd_set_uri_allow_tls(3)
>        nbd_set_uri_allow_transports(3)
>            These can be used to filter NBD URIs before calling
>            nbd_connect_uri(3).
>
>    New features
>        New tool nbdfuse(1) lets you create a loop-mounted file backed by an
>        NBD server without needing root.
>
>        "AF_VSOCK" is now a supported protocol (thanks Stefan
Hajnoczi and
>        Stefano Garzarella).
>
>        Support for the "FAST_ZERO" flag (Eric Blake).
>
>        Allow disabling certain protocol features, to make it easier to test
>        servers (Eric Blake).
>
>        Stack-allocated Variable Length Arrays (VLAs) are now banned
> throughout
>        the library, making the library easier to consume from threads and
>        other small stack situations.
>
>        Reproducible builds (Chris Lamb).
>
>        Support for filtering potentially dangerous or undesirable NBD URI
>        features.
>
>    Documentation
>        Many improvements to the generated manual pages, including:
>
>        ·   Separate "RETURN VALUE" and "ERRORS"
sections for each API
>            function.
>
>        ·   Example code.
>
>        ·   Relevant links can be added to the "SEE ALSO" section.
>
>        ·   Link to NBD URI specification where relevant, and improve
>            documentation around what URIs libnbd supports.
>
>        ·   Document libnbd version number scheme.
>
>        ·   Document limits on export name length, encoding etc.
>
>        New libnbd-security(3) man page listing past security issues and
>        remediations (Eric Blake).
>
>    Tools
>        nbdsh(1) has a new --base-allocation option which can be used to
>        request "base:allocation" metadata context.
>
>        New nbdsh(1) --uri (-u) option to connect to URIs.
>
>    Tests
>        You can now fuzz libnbd using either American Fuzzy Lop or clang’s
>        libFuzzer.
>
>        Add unit tests for nbdsh(1) (Eric Blake).
>
>        Improved interop testing with various NBD servers and features.
>
>    Other improvements and bug fixes
>        nbd_connect_tcp(3) now tries to return the correct errno(3) from the
>        underlying connect(2) call when that fails.
>
>        The nbd-protocol.h header file is now shared between libnbd and
> nbdkit.
>
>        Better fork-safety in "nbd_connect_*" APIs.
>
>        The code was analyzed with Coverity and various problems identified
> and
>        fixed.
>
>
> *** Release notes for nbdkit 1.16 ***
>
>        These are the release notes for nbdkit stable release 1.16.  This
>        describes the major changes since 1.14.
>
>        nbdkit 1.16.0 was released on 14th November 2019.
>
>    Security
>        Two security issues were found during development of nbdkit 1.16.
>        Fixes for these were backported to older stable branches.
> Upgrading to
>        the fixed versions is highly recommended.  The new
> nbdkit-security(1)
>        man page contains an up to date list of past security issues.
>
>        CVE-2019-14850 denial of service due to premature opening of
> back-end
>        connection
>
>        See the full announcement and links to mitigation, tests and fixes
>        here:
>
> https://www.redhat.com/archives/libguestfs/2019-September/msg00084.html
>
>        CVE-2019-14851 assertion failure by issuing commands in the wrong
> order
>
>        This CVE was caused by the fix to the previous issue.
>
>        See the full announcement and links to mitigation, tests and fixes
>        here:
>
> https://www.redhat.com/archives/libguestfs/2019-September/msg00272.html
>
>    New features
>        Add support for fast zeroing.  Plugins can expose this using the new
>        ".can_fast_zero" method (Eric Blake).
>
>        nbdkit-partitioning-plugin(1) allows use of
"mbr-id=default" or
>        "type-guid=default" to go back to the default MBR byte or
partition
>        type GUID.
>
>        New --mask-handshake server flag can be used for testing client
> feature
>        negotiation (Eric Blake).
>
>        The client export name is passed to nbdkit-captive(1) --run
> parameter
>        as $exportname (Eric Blake).
>
>        Captive --run commands which fail (eg. aborting) now cause nbdkit to
>        exit with an error instead of errors being silently ignored (Eric
>        Blake).
>
>        File descriptors can be passed to password parameters, eg:
>        "password=-3" which means that the password should be read
from file
>        descriptor 3.
>
>        nbdkit can now serve over the "AF_VSOCK" protocol (thanks
Stefan
>        Hajnoczi).
>
>        New --log=null option discards error messages.
>
>    Plugins
>        Python 2 support has been dropped from nbdkit-python-plugin(3) in
> line
>        with Python 2 end of life at the beginning of 2020.  Python ≥ 3.3 is
>        required by this plugin.  If you wish to continue to use Python 2
> then
>        you will need to use nbdkit 1.14.
>
>        New nbdkit-info-plugin(1) which returns various server information
> back
>        to the client.  It can be used for testing server latency amongst
> other
>        things.
>
>        nbdkit-data-plugin(1) now allows you to write "BYTE*N" to
get
> repeated
>        bytes (eg. nbdkit data data="0x55*4096").
>
>        nbdkit-ssh-plugin(1) new parameter
"compression=true|false" to
> control
>        transport compression.
>
>        nbdkit-vddk-plugin(1) is no longer compiled on non-x86 platforms
> since
>        VMware has only ever shipped VDDK on x86.
>
>        nbdkit-sh-plugin(1) scripts can now see the client exportname and
> can
>        use the "magic_config_key" feature.
>
>    Filters
>        New nbdkit-retry-filter(1) which can reopen the plugin
> transparently on
>        certain types of failures (lots of help from Eric Blake).
>
>    API
>        Macros "NBDKIT_VERSION_MAJOR",
"NBDKIT_VERSION_MINOR",
>        "NBDKIT_VERSION_MICRO" expose the compile-time version of
nbdkit to
>        plugins and filters (Eric Blake).
>
>        Filters (which unlike plugins do not have a public stable API) must
> now
>        exactly match the version of nbdkit when loaded (Eric Blake).
>
>        New ".can_fast_zero" method (Eric Blake).
>
>        New "nbdkit_export_name" server function for reading the
export name
>        passed by the client.
>
>        New "nbdkit_peer_name" server function to return the
client address
>        (like getpeername(2)).
>
>        New server functions for safely parsing integers:
> "nbdkit_parse_int",
>        "nbdkit_parse_unsigned", "nbdkit_parse_int8_t",
> "nbdkit_parse_uint8_t",
>        "nbdkit_parse_int16_t", "nbdkit_parse_uint16_t",
>        "nbdkit_parse_int32_t", "nbdkit_parse_uint32_t",
>        "nbdkit_parse_int64_t", "nbdkit_parse_uint64_t".
>
>    Bug fixes
>        ".trim" with FUA flag set now works (Eric Blake).
>
>    Documentation
>        The previous release notes have been turned into man pages.
>
>    Tests
>        Several tests now optionally use nbdsh(1) instead of qemu-io.
>
>        You can now fuzz nbdkit using either American Fuzzy Lop or clang’s
>        libFuzzer.
>
>        Several tests have had sleep times increased to make them more
> stable
>        when run on slow or heavily loaded machines.
>
>    Internals
>        Reproducible builds (Chris Lamb).
>
>        Compile code with -Wshadow warning (Eric Blake).
>
>        The internal backend system has been extensively overhauled.  In
>        particular this means that we now validate request ranges as
> requests
>        are passed between filters and down to the plugin, making it easier
> to
>        find bugs in filters early (Eric Blake).
>
>        Plugin size and "can_*" flags are cached more aggressively
by the
>        server (Eric Blake).
>
>        Variable Length Arrays (VLAs) on stack are now banned throughout the
>        code.
>
>        The nbd-protocol.h header describing the NBD protocol is now shared
>        with libnbd(3).
>
>        Plugin ".unload" method is now called after all worker
threads have
>        exited, avoiding races at server shutdown.
>
>        Code was audited using Coverity and various problems were fixed.
>
>
>
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
>
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs


-- 
Brett Thurber - RHCA, RHCVA
Distinguished Engineer and Engineering Manager, Migration Engineering
Products & Technologies Group, Red Hat
Mobile: +1 (512) 547-9282