libvirt users - Aug 2020 - Re: couple of questions

Hey folks,
I've been experimenting with native NBD live migration w/ TLS and have
a couple of questions.

1) It appears that in some cases modified default_tls_x509_cert_dir
from qemu.conf is not respected, seems like virsh always expects a
default location and does not check default_tls_x509_cert_dir:

virsh # migrate vm1 qemu+tls://ratchet.lan/system --live --persistent
--undefinesource --copy-storage-all --verbose --tls
error: internal error: unable to execute QEMU command 'object-add':
Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file
or directory

It's checking /etc/pki and not the location specified in
default_tls_x509_cert_dir. Is this a bug or am I missing something?

2) QEMU has -object tls-cipher-suites, but there does not seem to be a
way to specify TLS priority in libvirt's qemu conf. Solvable via
compile time --tls-priority flag, but that's not very convenient. Is
there a way to set TLS priority for QEMU TLS connections from libvirt
configs? This would be equivalent to libvirtd.conf's tls_priority
setting, but for QEMU, not for libvirt's own connections.

3) After setting up default_tls_x509_cert_dir and
default_tls_x509_verify =  1 (and directories as required see 1),
virsh initiated migrations with --tls flag succeed and captures show
that it's using TLS. However, they equally succeed without the flag.
Is there a way to ensure that only TLS communication is permitted
between QEMUs? I tried nbd_tls, but that did not seem to have any
effect.

Thanks a lot for your help!

On Sun, Aug 16, 2020 at 22:43:30 -0700, Vjaceslavs Klimovs
wrote:
> Hey folks,
> I've been experimenting with native NBD live migration w/ TLS and have
> a couple of questions.
> 
> 1) It appears that in some cases modified default_tls_x509_cert_dir
> from qemu.conf is not respected, seems like virsh always expects a
> default location and does not check default_tls_x509_cert_dir:
> 
> virsh # migrate vm1 qemu+tls://ratchet.lan/system --live --persistent
> --undefinesource --copy-storage-all --verbose --tls
> error: internal error: unable to execute QEMU command 'object-add':
> Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file
> or directory
> 
> It's checking /etc/pki and not the location specified in
> default_tls_x509_cert_dir. Is this a bug or am I missing something?
It would be a bug. Please note that if you modify /etc/libvirt/qemu.conf
the settings are actually loaded at startup of the libvirt daemon, thus
changing the file will not be applied unless you restart libvirtd.

If you manage to consistently reproduce it, please file an issue and
attach debug logs [1] so that we can see what is happening.

[1] https://libvirt.org/kbase/debuglogs.html
> 2) QEMU has -object tls-cipher-suites, but there does not seem to be a
> way to specify TLS priority in libvirt's qemu conf. Solvable via
> compile time --tls-priority flag, but that's not very convenient. Is
> there a way to set TLS priority for QEMU TLS connections from libvirt
> configs? This would be equivalent to libvirtd.conf's tls_priority
> setting, but for QEMU, not for libvirt's own connections.
Hmm, this might be useful. Please file a feature request.
> 3) After setting up default_tls_x509_cert_dir and
> default_tls_x509_verify =  1 (and directories as required see 1),
> virsh initiated migrations with --tls flag succeed and captures show
> that it's using TLS. However, they equally succeed without the flag.
Once you specify '--tls' both the connection for migration of the qemu
state and the NBD connection for migrating the disk storage uses TLS.
Without the --tls flag, neither of them uses it. If your tls environment
is setup properly there isn't any user visible difference, but the
traffic is encrypted only when --tls is used.

> Is there a way to ensure that only TLS communication is permitted
> between QEMUs? I tried nbd_tls, but that did not seem to have any
> effect.
Unfortunately the 'nbd_tls' field is named a bit misleadingly. The
setting refers to forcing TLS for NBD connections corresponding to
<disk> device which is accessed via NBD.

The NBD connection used for the non-shared-storage migration is
controlled by the settings for the 'migration' TLS environment. I
don't
think we have a setting to always force migration using TLS and it might
not fit well with the design of the --tls flag (as it would be
impossible to disable it then).

You can file an feature request for it though. As it is a security
focued setting and defaulting to encryption may be worthwhile in many
scenarios.

Note that in somewhat old libvirt versions there was a bug that that the
NBD connection used for the disk migration was not encrypted. This is
now addressed and newer libvirt will not allow migration from/to such
libvirt:

https://libvirt.org/news.html#v4-2-0-2018-04-01

On Mon, Aug 17, 2020 at 3:44 AM Peter Krempa <pkrempa@redhat.com>
wrote:
>
> On Sun, Aug 16, 2020 at 22:43:30 -0700, Vjaceslavs Klimovs wrote:
> > Hey folks,
> > I've been experimenting with native NBD live migration w/ TLS and
have
> > a couple of questions.
> >
> > 1) It appears that in some cases modified default_tls_x509_cert_dir
> > from qemu.conf is not respected, seems like virsh always expects a
> > default location and does not check default_tls_x509_cert_dir:
> >
> > virsh # migrate vm1 qemu+tls://ratchet.lan/system --live --persistent
> > --undefinesource --copy-storage-all --verbose --tls
> > error: internal error: unable to execute QEMU command
'object-add':
> > Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file
> > or directory
> >
> > It's checking /etc/pki and not the location specified in
> > default_tls_x509_cert_dir. Is this a bug or am I missing something?
>
> It would be a bug. Please note that if you modify /etc/libvirt/qemu.conf
> the settings are actually loaded at startup of the libvirt daemon, thus
> changing the file will not be applied unless you restart libvirtd.
>
> If you manage to consistently reproduce it, please file an issue and
> attach debug logs [1] so that we can see what is happening.
>
> [1] https://libvirt.org/kbase/debuglogs.html
It appears that that was indeed the problem, one of the libvirtds
instances was not restarted.
>
> > 2) QEMU has -object tls-cipher-suites, but there does not seem to be a
> > way to specify TLS priority in libvirt's qemu conf. Solvable via
> > compile time --tls-priority flag, but that's not very convenient.
Is
> > there a way to set TLS priority for QEMU TLS connections from libvirt
> > configs? This would be equivalent to libvirtd.conf's tls_priority
> > setting, but for QEMU, not for libvirt's own connections.
>
> Hmm, this might be useful. Please file a feature request.
Thank you for the explanation, I've filed
https://gitlab.com/libvirt/libvirt/-/issues/66.
>
> > 3) After setting up default_tls_x509_cert_dir and
> > default_tls_x509_verify =  1 (and directories as required see 1),
> > virsh initiated migrations with --tls flag succeed and captures show
> > that it's using TLS. However, they equally succeed without the
flag.
>
> Once you specify '--tls' both the connection for migration of the
qemu
> state and the NBD connection for migrating the disk storage uses TLS.
> Without the --tls flag, neither of them uses it. If your tls environment
> is setup properly there isn't any user visible difference, but the
> traffic is encrypted only when --tls is used.
>
>
> > Is there a way to ensure that only TLS communication is permitted
> > between QEMUs? I tried nbd_tls, but that did not seem to have any
> > effect.
>
> Unfortunately the 'nbd_tls' field is named a bit misleadingly. The
> setting refers to forcing TLS for NBD connections corresponding to
> <disk> device which is accessed via NBD.
>
> The NBD connection used for the non-shared-storage migration is
> controlled by the settings for the 'migration' TLS environment. I
don't
> think we have a setting to always force migration using TLS and it might
> not fit well with the design of the --tls flag (as it would be
> impossible to disable it then).
>
> You can file an feature request for it though. As it is a security
> focued setting and defaulting to encryption may be worthwhile in many
> scenarios.
Indeed. Filed https://gitlab.com/libvirt/libvirt/-/issues/67.
>
> Note that in somewhat old libvirt versions there was a bug that that the
> NBD connection used for the disk migration was not encrypted. This is
> now addressed and newer libvirt will not allow migration from/to such
> libvirt:
>
> https://libvirt.org/news.html#v4-2-0-2018-04-01
>
I've also went ahead and filed
https://gitlab.com/libvirt/libvirt/-/issues/68

Issue should be self explanatory - there should be no cleartext on the
network during migration, there are issues associated with doing that.