Xen devel - Jun 2013 - [TESTDAY] PV / HVM pass-through works when IOMMU present; weird failures when not

* Hardware:

Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz

Not sure exactly what mobo I have, but I get this error if I boot
without any iommu options:

Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata #47, #53

* Software:

Debian Wheezy dom0 (Stock Debian 3.2 kernel)

* Guest operating systems:

Debian Wheezy (Stock debian 3.2 kernel)

* Functionality tested:

xl pci-asssignable-add
xl pci-assignable-remove -r

Booting PV:
- xl pci-attach
- xl pci-remove [fails due to guest kernel bug]
- Shutting down the guest, starting a new one and re-assigning
- pci=[] in config file
- Shutting down guest, starting new one (with config file)

Booting HVM:
 - xl pci-attach,  xl pci-detach, re-attach
 - Shutting down guest with device assigned, starting a new one
 - pci=[] in config file
 - pci-detach after booting with config file

* Comments:

xl pci-detach fails, but this is a known issue that Konrad has fixed
upstream: http://bugs.xenproject.org/xen/bug/12

If the pci-detach fails, then afterwards you can''t attach it somewhere
else while the guest is running, but you can re-assign it after the
guest is shut down.

The big problems I see, however, are as follows:

- For PV guests there is no user-visible indication that the IOMMU has
been disabled.  All of the commands work, albeit with slightly
different configurations (e.g., using GSIs instead of MSIs).

- For HVM guests, the only user-visible indication tha the IOMMU has
been disabled is the following error message on the command-line:

# xl pci-attach h0 07:00.0
libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed

However, the device itself ends up passed-through to the guest anyway;
the guest seems to be able to see it and interact with it normally.
This is particularly scary, as in theory this should not be possible
without a working IOMMU.

I don''t think this is a blocker for 4.3, but we should definitely
release note it, and for 4.4 add a check to see if there is a
functioning IOMMU and only add a device if there''s an override set.

 -George

>>> On 28.06.13 at 17:37, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> - For HVM guests, the only user-visible indication tha the IOMMU has
> been disabled is the following error message on the command-line:
> 
> # xl pci-attach h0 07:00.0
> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
> 
> However, the device itself ends up passed-through to the guest anyway;
> the guest seems to be able to see it and interact with it normally.
> This is particularly scary, as in theory this should not be possible
> without a working IOMMU.
> 
> I don''t think this is a blocker for 4.3, but we should definitely
> release note it, and for 4.4 add a check to see if there is a
> functioning IOMMU and only add a device if there''s an override
set.
To me this very much looks like a security problem (which I
think we should fix asap).

As I tried this the other day with a boot time assignment, and
it prevented the guest from booting (which is how it should be)
- are you also seeing the guest happily using such device when
assigned via guest config file?

Knowing that may hint at where to look for the actual problem.

Also, I can''t really see how the guest would be able to interact
with a half way assigned device properly - I could imagine you
being able to look at its config space, and perhaps load the
driver, but I can''t see an I/O to succeed, at least not as long
as any bus mastering is being used the device (this ought to
crash this guest, another guest, or the host, or deliver corrupt
data). Purely port based I/O would likely work, but other than
serial cards I can''t think of many things that would do so.

Jan

On 28/06/13 17:00, Jan Beulich wrote:
>>>> On 28.06.13 at 17:37, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
>> - For HVM guests, the only user-visible indication tha the IOMMU has
>> been disabled is the following error message on the command-line:
>>
>> # xl pci-attach h0 07:00.0
>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>
>> However, the device itself ends up passed-through to the guest anyway;
>> the guest seems to be able to see it and interact with it normally.
>> This is particularly scary, as in theory this should not be possible
>> without a working IOMMU.
>>
>> I don''t think this is a blocker for 4.3, but we should
definitely
>> release note it, and for 4.4 add a check to see if there is a
>> functioning IOMMU and only add a device if there''s an override
set.
> To me this very much looks like a security problem (which I
> think we should fix asap).
>
> As I tried this the other day with a boot time assignment, and
> it prevented the guest from booting (which is how it should be)
> - are you also seeing the guest happily using such device when
> assigned via guest config file?
>
> Knowing that may hint at where to look for the actual problem.
>
> Also, I can''t really see how the guest would be able to interact
> with a half way assigned device properly - I could imagine you
> being able to look at its config space, and perhaps load the
> driver, but I can''t see an I/O to succeed, at least not as long
> as any bus mastering is being used the device (this ought to
> crash this guest, another guest, or the host, or deliver corrupt
> data). Purely port based I/O would likely work, but other than
> serial cards I can''t think of many things that would do so.
I get basically the same results; adding
"pci=[''07:00.0'']" to the config
file:

# xl create h0
Parsing config from h0
xc: info: VIRTUAL MEMORY ARRANGEMENT:
   Loader:        0000000000100000->000000000019ee28
   Modules:       0000000000000000->0000000000000000
   TOTAL:         0000000000000000->00000001ff800000
   ENTRY ADDRESS: 0000000000100608
xc: info: PHYSICAL MEMORY ALLOCATION:
   4KB PAGES: 0x0000000000000200
   2MB PAGES: 0x00000000000009fb
   1GB PAGES: 0x0000000000000003
libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed: 
Function not implemented
Daemon running with PID 4346
kodo2:~# xl pci-assignable-list
kodo2:~# xl pci-list h0
Vdev Device
05.0 0000:07:00.0

And in the guest:

# lspci
[snip]
00:05.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network 
Connection (rev 02)
# ifconfig -a
[snip]
eth2      Link encap:Ethernet  HWaddr 00:1b:21:3e:fe:90
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
           Memory:f3200000-f3220000
[snip]
# ifup eth2
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth2/00:1b:21:3e:fe:90
Sending on   LPF/eth2/00:1b:21:3e:fe:90
Sending on   Socket/fallback
DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 8

Since the cable is not plugged in, the device never comes up.  So it''s 
quite possible that since it''s sending packets but not receiving 
anything, that it''s either not doing any DMA, or that it''s
DMA''ing out
junk but it doesn''t matter.

Either way, if someone *is* passing through a device, it is probably a 
security issue.

  -George

On Fri, 2013-06-28 at 17:00 +0100, Jan Beulich wrote:
> Also, I can''t really see how the guest would be able to interact
> with a half way assigned device properly - I could imagine you
> being able to look at its config space, and perhaps load the
> driver, but I can''t see an I/O to succeed, at least not as long
> as any bus mastering is being used the device (this ought to
> crash this guest, another guest, or the host, or deliver corrupt
> data). Purely port based I/O would likely work, but other than
> serial cards I can''t think of many things that would do so.
That''s what I was thinking. I would be useful to know what sort of
device this is and to what extent it is "working".

Ian.

On 28/06/13 17:00, Jan Beulich wrote:
>>>> On 28.06.13 at 17:37, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
>> - For HVM guests, the only user-visible indication tha the IOMMU has
>> been disabled is the following error message on the command-line:
>>
>> # xl pci-attach h0 07:00.0
>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>
>> However, the device itself ends up passed-through to the guest anyway;
>> the guest seems to be able to see it and interact with it normally.
>> This is particularly scary, as in theory this should not be possible
>> without a working IOMMU.
>>
>> I don''t think this is a blocker for 4.3, but we should
definitely
>> release note it, and for 4.4 add a check to see if there is a
>> functioning IOMMU and only add a device if there''s an override
set.
> To me this very much looks like a security problem (which I
> think we should fix asap).
Is it worth delaying the release (yet) another week for?

Probably the simplest solution at the moment, if there''s an easy way
for
the toolstack to figure out whether there is a working IOMMU or not, is 
to simply not allow pass-through without an IOMMU unless there is an 
override option.

Thoughts?
  -George

>>> On 28.06.13 at 18:10, George Dunlap
<george.dunlap@eu.citrix.com> wrote:
> Either way, if someone *is* passing through a device, it is probably a 
> security issue.
After looking around a bit, I''m convinced that the other day I must
have tried this with xm (for whatever reason) - libxl has at least
two issues here compared to xend/xm:

- Missing error handling: Errors for the domain creation case
  propagate all the way back up to domcreate_attach_pci(), at
  which point they get silently ignored. Without having looked
  deeper, my first suggestion would be to simply drop devices
  from the d_config->pcidevs[] and decrement
  d_config->num_pcidevs upon failure, thus skipping any
  respective backend setup.

- Not making use of xc_test_assign_device() (which would allow
  detecting the situation before _any_ other setup gets done for
  the device).

Of course it is all but helpful that the backend in qemu has no way
of verifying that the device is in fact owned by the guest, i.e. it
has to blindly set up things trusting that the information handed
to it is consistent.

Jan

On Mon, Jul 1, 2013 at 11:53 AM, George Dunlap
<george.dunlap@eu.citrix.com> wrote:
> On 28/06/13 17:00, Jan Beulich wrote:
>>>>>
>>>>> On 28.06.13 at 17:37, George Dunlap
<George.Dunlap@eu.citrix.com>
>>>>> wrote:
>>>
>>> - For HVM guests, the only user-visible indication tha the IOMMU
has
>>> been disabled is the following error message on the command-line:
>>>
>>> # xl pci-attach h0 07:00.0
>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>>
>>> However, the device itself ends up passed-through to the guest
anyway;
>>> the guest seems to be able to see it and interact with it normally.
>>> This is particularly scary, as in theory this should not be
possible
>>> without a working IOMMU.
>>>
>>> I don''t think this is a blocker for 4.3, but we should
definitely
>>> release note it, and for 4.4 add a check to see if there is a
>>> functioning IOMMU and only add a device if there''s an
override set.
>>
>> To me this very much looks like a security problem (which I
>> think we should fix asap).
>
>
> Is it worth delaying the release (yet) another week for?
>
> Probably the simplest solution at the moment, if there''s an easy
way for the
> toolstack to figure out whether there is a working IOMMU or not, is to
> simply not allow pass-through without an IOMMU unless there is an override
> option.
On further reflection, I think there isn''t actually a security bug
here: The promised behavior as of now is that if you really need to
have an iommu, then you should specify "iommu=force".  If I specify
iommu=force, then of course Xen doesn''t boot, and I can''t
trigger this
problem.

This is actually a pretty awful interface, and should change, but
that''s a 4.4 thing, not a 4.3 thing.  Since we haven''t had any
other
issues reported, I think we should go ahead with the scheduled release
tomorrow.

 -George

>>> On 01.07.13 at 12:53, George Dunlap
<george.dunlap@eu.citrix.com> wrote:
> On 28/06/13 17:00, Jan Beulich wrote:
>>>>> On 28.06.13 at 17:37, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
>>> - For HVM guests, the only user-visible indication tha the IOMMU
has
>>> been disabled is the following error message on the command-line:
>>>
>>> # xl pci-attach h0 07:00.0
>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device failed
>>>
>>> However, the device itself ends up passed-through to the guest
anyway;
>>> the guest seems to be able to see it and interact with it normally.
>>> This is particularly scary, as in theory this should not be
possible
>>> without a working IOMMU.
>>>
>>> I don''t think this is a blocker for 4.3, but we should
definitely
>>> release note it, and for 4.4 add a check to see if there is a
>>> functioning IOMMU and only add a device if there''s an
override set.
>> To me this very much looks like a security problem (which I
>> think we should fix asap).
> 
> Is it worth delaying the release (yet) another week for?
I would say so, but I''m open to being convinced otherwise.
> Probably the simplest solution at the moment, if there''s an easy
way for
> the toolstack to figure out whether there is a working IOMMU or not, is 
> to simply not allow pass-through without an IOMMU unless there is an 
> override option.
xend had no override option - pass-through to HVM without
IOMMU should never be allowed imo.

Adding proper error handling perhaps is indeed beyond what''s
reasonable for 4.3, but making use of xc_test_assign_device()
should result in not too big a change.

Jan

>>> On 01.07.13 at 14:15, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> On Mon, Jul 1, 2013 at 11:53 AM, George Dunlap
> <george.dunlap@eu.citrix.com> wrote:
>> On 28/06/13 17:00, Jan Beulich wrote:
>>>>>>
>>>>>> On 28.06.13 at 17:37, George Dunlap
<George.Dunlap@eu.citrix.com>
>>>>>> wrote:
>>>>
>>>> - For HVM guests, the only user-visible indication tha the
IOMMU has
>>>> been disabled is the following error message on the
command-line:
>>>>
>>>> # xl pci-attach h0 07:00.0
>>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device
failed
>>>>
>>>> However, the device itself ends up passed-through to the guest
anyway;
>>>> the guest seems to be able to see it and interact with it
normally.
>>>> This is particularly scary, as in theory this should not be
possible
>>>> without a working IOMMU.
>>>>
>>>> I don''t think this is a blocker for 4.3, but we should
definitely
>>>> release note it, and for 4.4 add a check to see if there is a
>>>> functioning IOMMU and only add a device if there''s an
override set.
>>>
>>> To me this very much looks like a security problem (which I
>>> think we should fix asap).
>>
>>
>> Is it worth delaying the release (yet) another week for?
>>
>> Probably the simplest solution at the moment, if there''s an
easy way for the
>> toolstack to figure out whether there is a working IOMMU or not, is to
>> simply not allow pass-through without an IOMMU unless there is an
override
>> option.
> 
> On further reflection, I think there isn''t actually a security bug
> here: The promised behavior as of now is that if you really need to
> have an iommu, then you should specify "iommu=force".  If I
specify
> iommu=force, then of course Xen doesn''t boot, and I can''t
trigger this
> problem.
I disagree, not the least because the behavior was different with
xend: When there''s no IOMMU, pass-through to HVM must not
happen (or we''d have to suppress bus mastering on any such
passed through device). Pass-through to PV may happen, but is
insecure (as would be pass-through to HVM with disabled bus
mastering). So to anyone migrating from xend, if we don''t change
things, this will at least be perceived as a security bug.
> This is actually a pretty awful interface, and should change, but
> that''s a 4.4 thing, not a 4.3 thing.  Since we haven''t
had any other
> issues reported, I think we should go ahead with the scheduled release
> tomorrow.
As per the above and the earlier reply I sent, I don''t think we
should release without this fixed. Let me see whether the minimal
fix I sketched out earlier works...

Jan

Jan Beulich writes ("Re: [Xen-devel] [TESTDAY] PV / HVM pass-through works
when IOMMU present; weird failures when not"):
> As per the above and the earlier reply I sent, I don''t think we
> should release without this fixed. Let me see whether the minimal
> fix I sketched out earlier works...
As I just wrote on IRC (Jan, you seem to have fallen off the channel);

14:08 <Diziet> Re Jan''s comments.  To be clear, this
isn''t a regression
               compared to 4.2, right ?
14:10 <Diziet> I think the right answer is to treat this as an
embargo-less
               security problem.  Preparing a proper security patch for that 
               probably involves fixing the error handling.
14:10 <Diziet> And that might take a week.
14:10 <Diziet> I''m really not keen on this "extra
check" approach.
14:11 <Diziet> So I would go ahead with the release tomorrow.

>>> On 01.07.13 at 15:17, Ian Jackson <Ian.Jackson@eu.citrix.com>
wrote:
> Jan Beulich writes ("Re: [Xen-devel] [TESTDAY] PV / HVM pass-through
works when
> IOMMU present; weird failures when not"):
>> As per the above and the earlier reply I sent, I don''t think
we
>> should release without this fixed. Let me see whether the minimal
>> fix I sketched out earlier works...
> 
> As I just wrote on IRC (Jan, you seem to have fallen off the channel);
> 
> 14:08 <Diziet> Re Jan''s comments.  To be clear, this
isn''t a regression
>                compared to 4.2, right ?
> 14:10 <Diziet> I think the right answer is to treat this as an
embargo-less
>                security problem.  Preparing a proper security patch for
that
>                probably involves fixing the error handling.
> 14:10 <Diziet> And that might take a week.
> 14:10 <Diziet> I''m really not keen on this "extra
check" approach.
What''s wrong with it? It allows you to bail early, thus avoiding any
other setup to be done. But yes, it doesn''t save you from (sooner
or later) doing proper error recovery.

Jan
> 14:11 <Diziet> So I would go ahead with the release tomorrow.

On Mon, 1 Jul 2013, Jan Beulich wrote:
> >>> On 01.07.13 at 14:15, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> > On Mon, Jul 1, 2013 at 11:53 AM, George Dunlap
> > <george.dunlap@eu.citrix.com> wrote:
> >> On 28/06/13 17:00, Jan Beulich wrote:
> >>>>>>
> >>>>>> On 28.06.13 at 17:37, George Dunlap
<George.Dunlap@eu.citrix.com>
> >>>>>> wrote:
> >>>>
> >>>> - For HVM guests, the only user-visible indication tha the
IOMMU has
> >>>> been disabled is the following error message on the
command-line:
> >>>>
> >>>> # xl pci-attach h0 07:00.0
> >>>> libxl: error: libxl_pci.c:949:do_pci_add: xc_assign_device
failed
> >>>>
> >>>> However, the device itself ends up passed-through to the
guest anyway;
> >>>> the guest seems to be able to see it and interact with it
normally.
> >>>> This is particularly scary, as in theory this should not
be possible
> >>>> without a working IOMMU.
> >>>>
> >>>> I don''t think this is a blocker for 4.3, but we
should definitely
> >>>> release note it, and for 4.4 add a check to see if there
is a
> >>>> functioning IOMMU and only add a device if
there''s an override set.
> >>>
> >>> To me this very much looks like a security problem (which I
> >>> think we should fix asap).
> >>
> >>
> >> Is it worth delaying the release (yet) another week for?
> >>
> >> Probably the simplest solution at the moment, if there''s
an easy way for the
> >> toolstack to figure out whether there is a working IOMMU or not,
is to
> >> simply not allow pass-through without an IOMMU unless there is an
override
> >> option.
> > 
> > On further reflection, I think there isn''t actually a
security bug
> > here: The promised behavior as of now is that if you really need to
> > have an iommu, then you should specify "iommu=force".  If I
specify
> > iommu=force, then of course Xen doesn''t boot, and I
can''t trigger this
> > problem.
> 
> I disagree, not the least because the behavior was different with
> xend: When there''s no IOMMU, pass-through to HVM must not
> happen (or we''d have to suppress bus mastering on any such
> passed through device).
> Pass-through to PV may happen, but is
> insecure (as would be pass-through to HVM with disabled bus
> mastering).
I don''t have an opinion on the release, but this is certainly true.

I have assigned this issue XSA-61.

Ian.