Libguestfs - Sep 2019 - Re: [NBDKIT SECURITY] Denial of Service / Amplification Attack in nbdkit

We have discovered a potential Denial of Service / Amplification Attack
in nbdkit.

Lifecycle
---------

Reported: 2019-09-11  Fixed: 2019-09-11  Published: 2019-09-12

There is no CVE number assigned for this issue yet, but the bug is
being categorized and processed by Red Hat's security team which may
result in a CVE being published later.

Credit
------

Reported and patched by Richard W.M. Jones <rjones@redhat.com>.

Reviewed by Eric Blake <eblake@redhat.com> and
Daniel Berrangé <berrange@redhat.com>.

Description
-----------

nbdkit is a Network Block Device (NBD) server with multiple plugins.

An attacker, by connecting to nbdkit’s TCP port, is able to make the
current plugin’s .open() callback run.  For some plugins this callback
does a considerable amount of work, for example connecting to a remote
machine (nbdkit-ssh-plugin) or launching a small VM
(nbdkit-guestfs-plugin).  Thus for a small amount of work done by the
attacker (opening many TCP connections but not sending any data), a
large amount of work is done by nbdkit (eg. making many full ssh
connections to a remote machine or launching hundreds of VMs).

A related problem is that if nbdkit is configured to use TLS for
client authentication, even clients which are not authorized are able
to attack nbdkit in this way because all they have to do is open a TCP
connection.

Mitigating this is that the current plugin is NOT selected by the
attacker, but is part of the server configuration.  Therefore the
effectiveness of the attack depends on the particular configuration.

There are two situations in which you will NOT be affected:

 * You are not exposing nbdkit over TCP (eg. using nbdkit -U), or the
   TCP port is not accessible to untrusted attackers (eg. private LAN,
   firewall rules).

 * The plugin you are using does not do much work in its .open()
   callback.  For example the memory plugin does nothing in its
   callback so is not vulnerable:
  
https://github.com/libguestfs/nbdkit/blob/afbe7f11098df256acf8eafa9d4ec1e3cfd32910/plugins/memory/memory.c#L110-L115

Test if nbdkit is a vulnerable version
--------------------------------------

Run the following command (requires bash):

nbdkit -fv null --run 'sleep 1 >/dev/tcp/localhost/10809' 2>&1
|
  grep -q 'open readonly' && echo vulnerable || echo patched

Workarounds
-----------

It is recommended to upgrade to a fixed version of nbdkit (see next
section).  However if this cannot be done, apply network filtering
(eg. firewall, TCP wrappers, etc.) to ensure that untrusted clients
cannot connect to nbdkit’s public TCP port.  Or use a Unix domain
socket instead of a loopback TCP connection.

Fixes
-----

This affects all versions of nbdkit.  A fix is available for 1.12,
1.14 and the current development branch.

* development branch (1.15)
 
https://github.com/libguestfs/nbdkit/commit/c05686f9577fa91b6a3a4d8c065954ca6fc3fd62
  or use nbdkit >= 1.15.1 from
  http://download.libguestfs.org/nbdkit/1.15-development/

* stable branch 1.14
 
https://github.com/libguestfs/nbdkit/commit/e06cde00659ff97182173d0e33fff784041bcb4a
  or use nbdkit >= 1.14.1 from
  http://download.libguestfs.org/nbdkit/1.14-stable/

* stable branch 1.12
 
https://github.com/libguestfs/nbdkit/commit/22b30adb796bb6dca264a38598f80b8a234ff978
  or use nbdkit >= 1.12.7 from
  http://download.libguestfs.org/nbdkit/1.12-stable/



-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On 9/12/19 12:41 PM, Richard W.M. Jones wrote:
> We have discovered a potential Denial of Service / Amplification Attack
> in nbdkit.
Unfortunately, our fix for this issue cause another potential Denial of
Service attack:
> 
> Lifecycle
> ---------
> 
> Reported: 2019-09-11  Fixed: 2019-09-11  Published: 2019-09-12
> 
> There is no CVE number assigned for this issue yet, but the bug is
> being categorized and processed by Red Hat's security team which may
> result in a CVE being published later.
> 
Reported: 2019-09-18  Fixed: 2019-09-19  Published: 2019-09-20

Also pending Red Hat security review for whether this deserves a CVE
(presumably either both issues, or neither, will have a CVE)
> Credit
> ------
> 
> Reported and patched by Richard W.M. Jones <rjones@redhat.com>.
> 
> Reviewed by Eric Blake <eblake@redhat.com> and
> Daniel Berrangé <berrange@redhat.com>.
> 
> Description
> -----------
> 
> nbdkit is a Network Block Device (NBD) server with multiple plugins.
> 
> An attacker, by connecting to nbdkit’s TCP port, is able to make the
> current plugin’s .open() callback run.  For some plugins this callback
> does a considerable amount of work, for example connecting to a remote
> machine (nbdkit-ssh-plugin) or launching a small VM
> (nbdkit-guestfs-plugin).  Thus for a small amount of work done by the
> attacker (opening many TCP connections but not sending any data), a
> large amount of work is done by nbdkit (eg. making many full ssh
> connections to a remote machine or launching hundreds of VMs).
> 
> A related problem is that if nbdkit is configured to use TLS for
> client authentication, even clients which are not authorized are able
> to attack nbdkit in this way because all they have to do is open a TCP
> connection.
The NBD specification allows a client to request details about an export
(NBD_OPT_INFO) prior to committing to use that export (NBD_OPT_GO).
However, the fix to defer a plugin's .open() callback to as late as
possible means that a compliant client handshake now triggers
back-to-back opens from the same connection rather than sharing a single
open, and the second one fails due to an assertion because the first
open did not have a matching .close() callback.  Most NBD clients do not
query info, therefore, a malicious client can use the assertion failure
by querying info to cause a denial of service to another client that
does not query info.
> 
> Mitigating this is that the current plugin is NOT selected by the
> attacker, but is part of the server configuration.  Therefore the
> effectiveness of the attack depends on the particular configuration.
Mitigating this is that if TLS is used to filter out untrusted guest,
and you can ensure that trusted guests do not query info, then the
assertion is unreachable.
> 
> There are two situations in which you will NOT be affected:
> 
>  * You are not exposing nbdkit over TCP (eg. using nbdkit -U), or the
>    TCP port is not accessible to untrusted attackers (eg. private LAN,
>    firewall rules).
> 
>  * The plugin you are using does not do much work in its .open()
>    callback.  For example the memory plugin does nothing in its
>    callback so is not vulnerable:
>   
https://github.com/libguestfs/nbdkit/blob/afbe7f11098df256acf8eafa9d4ec1e3cfd32910/plugins/memory/memory.c#L110-L115
> 
> Test if nbdkit is a vulnerable version
> --------------------------------------
> 
> Run the following command (requires bash):
> 
> nbdkit -fv null --run 'sleep 1 >/dev/tcp/localhost/10809'
2>&1 |
>   grep -q 'open readonly' && echo vulnerable || echo
patched
At this time, there are no known readily-available open source clients
which query info, thus no easy way to test for the second vulnerability
other than creating a custom client.
> 
> Workarounds
> -----------
> 
> It is recommended to upgrade to a fixed version of nbdkit (see next
> section).  However if this cannot be done, apply network filtering
> (eg. firewall, TCP wrappers, etc.) to ensure that untrusted clients
> cannot connect to nbdkit’s public TCP port.  Or use a Unix domain
> socket instead of a loopback TCP connection.
> 
> Fixes
> -----
> 
> This affects all versions of nbdkit.  A fix is available for 1.12,
> 1.14 and the current development branch.
> 
> * development branch (1.15)
>  
https://github.com/libguestfs/nbdkit/commit/c05686f9577fa91b6a3a4d8c065954ca6fc3fd62
>   or use nbdkit >= 1.15.1 from
>   http://download.libguestfs.org/nbdkit/1.15-development/
Use nbdkit >= 1.15.2, containing
https://github.com/libguestfs/nbdkit/commit/a6b88b195a959b17524d1c8353fd425d4891dc5f
> 
> * stable branch 1.14
>  
https://github.com/libguestfs/nbdkit/commit/e06cde00659ff97182173d0e33fff784041bcb4a
>   or use nbdkit >= 1.14.1 from
>   http://download.libguestfs.org/nbdkit/1.14-stable/
Use nbdkit >= 1.14.2, containing
https://github.com/libguestfs/nbdkit/commit/bf0d61883a2f02f4388ec10dc92d4c61c093679e
> 
> * stable branch 1.12
>  
https://github.com/libguestfs/nbdkit/commit/22b30adb796bb6dca264a38598f80b8a234ff978
>   or use nbdkit >= 1.12.7 from
>   http://download.libguestfs.org/nbdkit/1.12-stable/
Use nbdkit >= 1.12.8, containing
https://github.com/libguestfs/nbdkit/commit/b2bc6683ea3cd1f6be694e8a681dfa411b7d15f3

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

On 9/20/19 8:58 AM, Eric Blake wrote:
> On 9/12/19 12:41 PM, Richard W.M. Jones wrote:
>> We have discovered a potential Denial of Service / Amplification Attack
>> in nbdkit.
> 
> Unfortunately, our fix for this issue cause another potential Denial of
> Service attack:
> 
>>
>> Lifecycle
>> ---------
>>
>> Reported: 2019-09-11  Fixed: 2019-09-11  Published: 2019-09-12
>>
>> There is no CVE number assigned for this issue yet, but the bug is
>> being categorized and processed by Red Hat's security team which
may
>> result in a CVE being published later.
>>
> 
> Reported: 2019-09-18  Fixed: 2019-09-19  Published: 2019-09-20
> 
> Also pending Red Hat security review for whether this deserves a CVE
> (presumably either both issues, or neither, will have a CVE)
Both CVEs have now been assigned:
CVE-2019-14850 - denial of service due to premature .open, depending on 
plugin used
CVE-2019-14851 - denial of service due to assertion after NBD_OPT_INFO, 
independent of plugin

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org