Pkg xen devel - Jan 2019 - Bug#908453: xen-utils-common: README.comet seems to no longer apply to the current Xen packages

Package: xen-utils-common
Version: 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9
Severity: important

Dear Maintainer,

   * What led up to the situation?

I needed to build a new Xen DomU.  Since I had seen that Debian had picked
up the 4.8 "comet" changes and pvshim, I wanted to experiment with
Xen's PVH
mode and have the DomU run the Linux 4.17 kernel in stretch-backports, which
should be new enough to support PVH.

It seemed like this should work, given instructions in
/usr/share/doc/xen-utils-common/README.comet
-------- 
* Converting a PV config to a PVH config

If you have a kernel capable of booting PVH, then PVH mode is both
faster and more secure than PV or PVH-shim mode.

- Remove any reference to 'builder' (e.g.,
`builder="generic"`)
- Add the following line:
  type="pvh"
--------


   * What exactly did you do (or not do) that was effective (or
     ineffective)?

I ran xen-create-image (from the xen-tools package) with --dist=stretch to
build the DomU and its configuration.  I also used --pygrub so Xen would
boot the kernel installed in the DomU's filesystem.

I temporarily edited the DomU's Xen/xl configuration to use the pvshim, per
the instructions in README.comet (type = 'pvh' / pvshim = 1), started
the
DomU, added the stretch-backports repository, and installed
linux-image-amd64 from backports.  I then shut down the DomU, edited the
DomU's configuration to remove the pvshim = 1 line, and re-started the DomU.


   * What was the outcome of this action?

It appeared the DomU was running in PV mode, despite my having added type
'pvh' to its Xen/xl configuration.

It was difficult to tell, but I believed the DomU was running in PV mode as
its kernel printed:

 - [    0.000000] Hypervisor detected: Xen PV
 - [    0.000000] Kernel/User page tables isolation: disabled on XEN PV.

Some Internet searching also seemed to indicate that "xl list -l
<id>"
should mention "pvh" if the DomU is running in PVH mode.  However, I
only
saw:

 - "type": "pv",

There was no mention of "pvh" in the output.


   * What outcome did you expect instead?

I expected the DomU to run in PVH mode.  I wasn't sure what this was
supposed to look like, but I did some experiments (installed the old
4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 packages and worked around
bootloader being broken for PVH) and saw that in fact, with that version of
the Xen packages, the DomU kernel prints:

 - [    0.000000] Hypervisor detected: Xen HVM
 - [    0.000000] Booting paravirtualized kernel on Xen PVH
 - [    0.000000] Kernel/User page tables isolation: enabled

And xl list -l <id> shows:

 - "type": "pvh",


   * More thoughts/discussion.

It looks like the Debian packages lost support for booting DomUs in PVH mode
with version 4.8.3+xsa262+shim4.10.0+comet3-1+deb9u6.  Probably because:

    Update to new upstream version 4.8.3+xsa262+shim4.10.0+comet3.
    (This is the upstream staging-4.8 branch, which is ahead of the
    upstream CI-tested stable-4.8 branch by precisely the three
    most recent XSA fixes.  We are switching away from the special
    upstream 4.8 comet branch.)

And maybe that's fine... if the mitigation comet and the pvshim provided is
also effectively provided by XPTI changes that were present in the
stable-4.8 branch, then I guess it isn't really necessary for anybody to use
(and thus PVH-mode boot) the shim anymore.  However, if that's the case,
then it probably doesn't make sense to include README.comet and continue
packaging the shim in the Debian packages anymore.

The thing that's nefarious (and could be grounds for increasing the bug
severity), is that anybody that followed the README.comet instructions and
configured DomUs to boot a PVH-capable kernel without the shim is now,
probably to their surprise, running their DomU in PV mode.  This means
they've lost Linux's KPTI protections from Meltdown within their DomU. 
The
underlying issue is that the xl command seems to silently ignore
configuration directives it doesn't understand--which, without the 4.8
comet2 changes, includes 'type'!

This isn't a huge deal for me and my deployment (at the CMU Computer Club).
Our Xen infrastructure was running jessie/Xen 4.4 at best when the
Meltdown/Spectre news broke.  So our initial mitigation was to switch
everything to run in HVM mode (and we've continued doing so since then).  I
was interested in exploring PVH mode though, since it looked like it was
more similar to PV mode in some ways that would make it work better with
various tooling (e.g., xen-tools).  The fact it didn't work with the Debian
packages the way I thought it would was surprising, and I figured it might
be surprising for other people too.

--Keith Bare


-- System Information:
Debian Release: 9.5
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-7-amd64 (SMP w/16 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages xen-utils-common depends on:
ii  lsb-base        9.20161125
ii  python          2.7.13-2
ii  ucf             3.0036
ii  udev            232-25+deb9u4
ii  xenstore-utils  4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9

xen-utils-common recommends no packages.

xen-utils-common suggests no packages.

-- Configuration Files:
/etc/default/xendomains changed [not included]

-- no debconf information

In my initial bug submission, I indicated I saw the problem in version
4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9, as that was what my system was
running when I first noticed Xen ignoring type = 'pvh'.

I saw there was an update: 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10.  Since
it looks like it's using a newer upstream version, I wanted to check whether
it also ignores type = 'pvh'.  I could imagine maybe some of the comet
changes were merged in...

But, even with the deb9u10 package, I'm still seeing that DomUs will boot in
PV mode when their configuration specifies type = 'pvh'.

--Keith Bare

Hi Keith,

Thanks for your elaborate report. I'll context-quote to try answer a few
things below.

Summary of the info below:
* PVH is not supposed to work with Xen 4.8.
* The documentation in README.comet is wrong for the recent Xen
4.8-stable that we ship now, we should fix that.
* For 4.8 and PV guests, XPTI is also available now (and enabled by
default).
* Silently ignoring invalid configuration is bad.

On 09/10/2018 05:51 AM, Keith Bare wrote:
> Package: xen-utils-common
> Version: 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9
> Severity: important
> 
> Dear Maintainer,
> 
>    * What led up to the situation?
> 
> I needed to build a new Xen DomU.  Since I had seen that Debian had picked
> up the 4.8 "comet" changes and pvshim, I wanted to experiment
with Xen's PVH
> mode and have the DomU run the Linux 4.17 kernel in stretch-backports,
which
> should be new enough to support PVH.
Parts of the PVH code have been backported. Running PVH guests yourself
is still not supported or recommended with Xen 4.8. If it was
temporarily, in the shim-branch, it was still not recommended.
> It seemed like this should work, given instructions in
> /usr/share/doc/xen-utils-common/README.comet
> -------- 
> * Converting a PV config to a PVH config
> 
> If you have a kernel capable of booting PVH, then PVH mode is both
> faster and more secure than PV or PVH-shim mode.
> 
> - Remove any reference to 'builder' (e.g.,
`builder="generic"`)
> - Add the following line:
>   type="pvh"
> --------
We just had a little conversation about this in #debian-xen on oftc irc,
and the latest info is that the documentation is (probably) wrong. The
README.comet is not present in upstream 4.8, but was copied from the
security advisory into our packaging.

XSA-254 mentions "PVH is supported in Xen 4.8 only with the 4.8
"Comet"
security release branch."

https://xenbits.xen.org/xsa/advisory-254.html

However...
>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
> 
> I ran xen-create-image (from the xen-tools package) with --dist=stretch to
> build the DomU and its configuration.  I also used --pygrub so Xen would
> boot the kernel installed in the DomU's filesystem.
> 
> I temporarily edited the DomU's Xen/xl configuration to use the pvshim,
per
> the instructions in README.comet (type = 'pvh' / pvshim = 1),
started the
> DomU, added the stretch-backports repository, and installed
> linux-image-amd64 from backports.  I then shut down the DomU, edited the
> DomU's configuration to remove the pvshim = 1 line, and re-started the
DomU.
> 
> 
>    * What was the outcome of this action?
> 
> It appeared the DomU was running in PV mode, despite my having added type
> 'pvh' to its Xen/xl configuration.
> 
> It was difficult to tell, but I believed the DomU was running in PV mode as
> its kernel printed:
> 
>  - [    0.000000] Hypervisor detected: Xen PV
>  - [    0.000000] Kernel/User page tables isolation: disabled on XEN PV.
Yes, that's PV.
> Some Internet searching also seemed to indicate that "xl list -l
<id>"
> should mention "pvh" if the DomU is running in PVH mode. 
However, I only
> saw:
> 
>  - "type": "pv",
> 
> There was no mention of "pvh" in the output.
> 
> 
>    * What outcome did you expect instead?
> 
> I expected the DomU to run in PVH mode.  I wasn't sure what this was
> supposed to look like, but I did some experiments (installed the old
> 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 packages and worked around
> bootloader being broken for PVH) and saw that in fact, with that version of
> the Xen packages, the DomU kernel prints:
> 
>  - [    0.000000] Hypervisor detected: Xen HVM
>  - [    0.000000] Booting paravirtualized kernel on Xen PVH
>  - [    0.000000] Kernel/User page tables isolation: enabled
> 
> And xl list -l <id> shows:
> 
>  - "type": "pvh",
> 
> 
>    * More thoughts/discussion.
> 
> It looks like the Debian packages lost support for booting DomUs in PVH
mode
> with version 4.8.3+xsa262+shim4.10.0+comet3-1+deb9u6.  Probably because:
> 
>     Update to new upstream version 4.8.3+xsa262+shim4.10.0+comet3.
>     (This is the upstream staging-4.8 branch, which is ahead of the
>     upstream CI-tested stable-4.8 branch by precisely the three
>     most recent XSA fixes.  We are switching away from the special
>     upstream 4.8 comet branch.)
...yes, that would be the missing puzzle piece which explains it. So in
that case, the README.comet which is still in our package is misleading
and incorrect.
> And maybe that's fine... if the mitigation comet and the pvshim
provided is
> also effectively provided by XPTI changes that were present in the
> stable-4.8 branch, then I guess it isn't really necessary for anybody
to use
> (and thus PVH-mode boot) the shim anymore. 
Yes, using XPTI for PV would be the least bad thing now for 4.8 I guess.
It's enabled by default by the way, so if you didn't see huge
performance problems in your dom0, you might as well try to run PV
guests just like before and see how they behave with XPTI.
> However, if that's the case,
> then it probably doesn't make sense to include README.comet and
continue
> packaging the shim in the Debian packages anymore.
If someone is using the shim, we can't just go break that. :|
> The thing that's nefarious (and could be grounds for increasing the bug
> severity), is that anybody that followed the README.comet instructions and
> configured DomUs to boot a PVH-capable kernel without the shim is now,
> probably to their surprise, running their DomU in PV mode.  This means
> they've lost Linux's KPTI protections from Meltdown within their
DomU.
No, 64-bit PV does not use KPTI because Xen provides an equivalent
mechanism for the guest (and has been doing that for a long time). Xen
XPTI will add protection to break out of the guest to the hypervisor and
back to the (or another) guest. So afaics now this does not cause
unwanted negative security problems.
> The
> underlying issue is that the xl command seems to silently ignore
> configuration directives it doesn't understand--which, without the 4.8
> comet2 changes, includes 'type'!
Silently ignoring unknown or unused configuration or silently ignoring
an invalid configuration combination and doing something else is bad
practice. I certainly agree.

This is a bug report for upstream. Given the size of my current
(Debian/Xen-related) todo-list, I can't help working on that now.
> This isn't a huge deal for me and my deployment (at the CMU Computer
Club).
> Our Xen infrastructure was running jessie/Xen 4.4 at best when the
> Meltdown/Spectre news broke.  So our initial mitigation was to switch
> everything to run in HVM mode (and we've continued doing so since
then).  I
> was interested in exploring PVH mode though, since it looked like it was
> more similar to PV mode in some ways that would make it work better with
> various tooling (e.g., xen-tools).  The fact it didn't work with the
Debian
> packages the way I thought it would was surprising, and I figured it might
> be surprising for other people too.
> 
> [...]
Note: we're now working to get Xen 4.11 for Buster in shape, so at least
there's something to look forward to. :-)

Regards,
Hans van Kranenburg

tags 908453 + pending
thanks

Hi Keith,

I added a change in the packaging that removed the misleading text about
PVH from the README.comet:

https://salsa.debian.org/xen-team/debian-xen/commit/f7f02e1b2ba0dd6464b53f134f9ca13c5a5550e8

Hans

Processing commands for control at bugs.debian.org:
> tags 908453 + pending
Bug #908453 [xen-utils-common] xen-utils-common: README.comet seems to no longer
apply to the current Xen packages
Added tag(s) pending.
> thanks
Stopping processing here.

Please contact me if you need assistance.
-- 
908453: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908453
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

Your message dated Fri, 17 Jan 2020 21:17:30 +0000
with message-id <E1isYzi-000HAn-GN at fasolo.debian.org>
and subject line Bug#908453: fixed in xen 4.8.5.final+shim4.10.4-1+deb9u12
has caused the Debian Bug report #908453,
regarding xen-utils-common: README.comet seems to no longer apply to the current
Xen packages
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
908453: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908453
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Keith Bare <kbare at alumni.cmu.edu>
Subject: xen-utils-common: README.comet seems to no longer apply to the current
Xen packages
Date: Sun, 09 Sep 2018 23:51:30 -0400
Size: 7396
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200117/1c1c0e60/attachment-0002.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Ian Jackson <ijackson at chiark.greenend.org.uk>
Subject: Bug#908453: fixed in xen 4.8.5.final+shim4.10.4-1+deb9u12
Date: Fri, 17 Jan 2020 21:17:30 +0000
Size: 12525
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200117/1c1c0e60/attachment-0003.mht>