Pkg xen devel - Jul 2023 - Bug#1041533: xen-system-amd64: Xen fails to start hvm type VMs when a vncpasswd is set

Package: xen-system-amd64
Version: 4.17.1+2-gb773c48e36-1
Severity: important

Dear Maintainer,

after upgrading my bullseye server to bookworm I ran into the issue that all VMs
of type hvm are not starting anymore.

xl throws an error:
libxl: error: libxl_qmp.c:1399:qmp_ev_fd_callback: Domain 8:error on QMP socket:
Connection reset by peer
libxl: error: libxl_qmp.c:1438:qmp_ev_fd_callback: Domain 8:Error happened with
the QMP connection to QEMU
libxl: error: libxl_dm.c:3371:device_model_postconfig_done: Domain 8:Post DM
startup configs failed, rc=-26
libxl: error: libxl_create.c:1896:domcreate_devmodel_started: Domain 8:device
model did not start: -26
libxl: error: libxl_aoutils.c:646:libxl__kill_xs_path: Device Model already
exited
libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 8:Non-existant
domain
libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 8:Unable to
destroy guest
libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 8:Destruction of
domain failed

I started digging around in this QMP stuff and installed Xen freshly on another
server with the actual bookworm iso, but the problem is the same there, too.

After reading the log of the VM I found this error:
xen-qemu-system-i386: -vnc 172.17.2.3:1,password=on,to=99: Cipher backend does
not support DES algorithm

When disabling the vncpassword (and keeping the rest of the VM configuration
untouched), xl is able to launch the VM properly.

I searched around for a while but I did not find any configuration option for
choosing the cipher used by vnc.

Running vnc without password is a potential security risk.

I hope you have a clue to either fix this or extend the documentation on this.

Best regards
Claus


-- System Information:
Debian Release: 12.0
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'),
(500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-10-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages xen-system-amd64 depends on:
ii  xen-hypervisor-4.17-amd64  4.17.1+2-gb773c48e36-1
ii  xen-hypervisor-common      4.17.1+2-gb773c48e36-1
ii  xen-utils-4.17             4.17.1+2-gb773c48e36-1

xen-system-amd64 recommends no packages.

xen-system-amd64 suggests no packages.

-- no debconf information

Hello,

I -think- VNC auth has been removed from the last QEMU versions.
Also maybe related, QEMU in Debian is not configured with VNC_SASL 
(there was a discussion about it in #debian-xen).

Wait for confirmations, meanwhile there is another option: SSH (maybe 
even more secure ?).

The workaround is to make the VNC servers only accessible from dom0, 
then to create SSH tunnels to connect to them :

1. in the domU config file, select "127.0.0.1" as the IP address to 
listen to, and remove everything about authentication
2. from your management host, create a tunnel, something like "ssh -nN 
-L localhost:12345:localhost:59xx user at dom0"
3. from your management host, use VNC_APP:12345 to connect to the display

The "xx" for the tunnel represent the "VNC display id"
you've chosen in
your domU config file, so if you have "vnclisten = 127.0.0.1:12", the 
real IP address is "127.0.0.1:5912" (in your case, you'd pick
5901).

Hope it helps.

PS: as for documentation it will be in the new Debian Xen wiki page 
(which I'm rewriting, for now it's still an offline draft).

--
Cyril R?bert / zithro

This issue also affects the Ganeti Package indirectly - it does not allow
the user to disable the vncpasswd file/configuration which results in
unbootable instances after an Upgrade to Debian Bookworm:

gnt-instance start xen-test-instance01

Waiting for job 258 for xen-test-instance01.staging.ganeti.org ...
Job 258 for xen-test-instance01.staging.ganeti.org has failed: Failure:
command execution error:
Could not start instance 'xen-test-instance01.staging.ganeti.org':
Hypervisor error: Failed to start instance
xen-test-instance01.staging.ganeti.org: exited with exit code 3 (Parsing
config from /etc/xen/xen-test-instance01.staging.ganeti.org
WARNING: you seem to be using "kernel" directive to override HVM guest
firmware. Ignore that. Use "firmware_override" instead if you really
want a
non-default firmware
WARNING: ignoring device_model directive.
WARNING: Use "device_model_override" instead if you really want a
non-default device_model
libxl: error: libxl_qmp.c:1399:qmp_ev_fd_callback: Domain 9:error on QMP
socket: Connection reset by peer
libxl: error: libxl_qmp.c:1438:qmp_ev_fd_callback: Domain 9:Error happened
with the QMP connection to QEMU
libxl: error: libxl_dm.c:3371:device_model_postconfig_done: Domain 9:Post
DM startup configs failed, rc=-26
libxl: error: libxl_create.c:1896:domcreate_devmodel_started: Domain
9:device model did not start: -26
libxl: error: libxl_aoutils.c:646:libxl__kill_xs_path: Device Model already
exited
libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain
9:Non-existant domain
libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 9:Unable
to destroy guest
libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 9:Destruction
of domain failed
). Moved config file to
/var/log/ganeti/xen/xen-test-instance01.staging.ganeti.org-2024-05-16_16_14_29

-- 
 Rudolph Bott - bott at sipgate.de
 Telefon: +49 (0)211-63 55 55-55
 Telefax: +49 (0)211-63 55 55-22

 sipgate GmbH - Gladbacher Str. 74 - 40219 D?sseldorf
 HRB D?sseldorf 39841 - Gesch?ftsf?hrer: Thilo Salmon, Tim Mois
 Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391

 www.sipgate.de - www.sipgate.co.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20240516/86c41d9d/attachment.htm>

found 1041533 4.17.3+10-g091466ba55-1~deb12u1

Hello all,

This bug also affects me, with the same symptoms and setup that were described
earlier:

	root at hypervisor:~# cat /var/log/xen/qemu-dm-vm.log
	xen-qemu-system-i386: -vnc :0,password=on,to=99: Cipher backend does not
support DES algorithm

	root at hypervisor:~# grep vnc /etc/xen/vm
	vnc = "1"
	vncconsole = "1"
	vnclisten = ""
	vncpasswd = "some-password"

Searching for as to why this happens, I think I found the root cause:
https://gitlab.com/qemu-project/qemu/-/issues/1158

Sure enough, "regular" qemu-system-x86_64 includes both GnuTLS and
Nettle:

	root at hypervisor:~# ldd /usr/bin/qemu-system-x86_64 | grep -e crypt -e tls -e
nettle
		libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30
(0x00007f9e0b400000)
		libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007f9e0b072000)
		libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f9e0a000000)

But Xen's xen-qemu-system-i386 does not:

	root at hypervisor:~# ldd /usr/libexec/xen-qemu-system-i386 | grep -e crypt -e
tls -e nettle
		libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f63b1600000)

Both versions are the same, though:

	root at hypervisor:~# qemu-system-x86_64 --version
	QEMU emulator version 7.2.11 (Debian 1:7.2+dfsg-7+deb12u6)
	Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

	root at hypervisor:~# /usr/libexec/xen-qemu-system-i386 --version
	QEMU emulator version 7.2.11 (Debian 1:7.2+dfsg-7+deb12u6)
	Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

This leads me to believe that if xen-qemu-system-i386 were to be linked with at
least one of the mentioned DES providers, this regression would be fixed.

Best regards,
Thomas