Xen devel - Mar 2006 - PCI delegation works, access to the delegated NIC doesn''t

Hi all,

has anyone seen such a behaviour?:

I delegated successfully my NIC to a domU, but access to it from domU is not 
possible. Device appears, but access to it via ''ifconfig ethX
up'' doesn''t
work. Ends up in dmesg output like:
tg3: tg3_reset_hw timed out for eth1, firmware will not restart magic=00000000

Delegating my USB controllers e.g. works flawlessly.

Full Bug report can be found at:
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=554


Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7 Mar 2006, at 20:34, Alexander Wilms wrote:
> has anyone seen such a behaviour?:
>
> I delegated successfully my NIC to a domU, but access to it from domU 
> is not
> possible. Device appears, but access to it via ''ifconfig ethX
up''
> doesn''t
> work. Ends up in dmesg output like:
> tg3: tg3_reset_hw timed out for eth1, firmware will not restart 
> magic=00000000
>
> Delegating my USB controllers e.g. works flawlessly.
You could try getting rid of access control on I/O memory and I/O 
ports. Modify the Xen definitions of macros ioports_access_permitted() 
and iomem_access_permitted() to always return 1. Then see if the NIC 
works. If so, the code in domain0 to work out what resources a given 
PCI device requires is not working properly for you.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7 Mar 2006, at 20:34, Alexander Wilms wrote:
>
> I delegated successfully my NIC to a domU, but access to it from domU 
> is not
> possible. Device appears, but access to it via ''ifconfig ethX
up''
> doesn''t
> work. Ends up in dmesg output like:
> tg3: tg3_reset_hw timed out for eth1, firmware will not restart 
> magic=00000000
>
> Delegating my USB controllers e.g. works flawlessly.
Another worthwhile thing to try is the following (in domain0):

cat /proc/pci
cat /sys/bus/pci/devices/0000\:02\:00.0/resource
python /usr/lib/python/xen/util/pci.py 0 2 0 0

And send us the output (or attach to the bugzilla report and let us 
know on the list).

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2006-03-08 at 11:36 +0000, Keir Fraser wrote:
> On 7 Mar 2006, at 20:34, Alexander Wilms wrote:
> 
> >
> > I delegated successfully my NIC to a domU, but access to it from domU 
> > is not
> > possible. Device appears, but access to it via ''ifconfig ethX
up''
> > doesn''t
> > work. Ends up in dmesg output like:
> > tg3: tg3_reset_hw timed out for eth1, firmware will not restart 
> > magic=00000000
> >
> > Delegating my USB controllers e.g. works flawlessly.
> 
> Another worthwhile thing to try is the following (in domain0):
> 
> cat /proc/pci
> cat /sys/bus/pci/devices/0000\:02\:00.0/resource
> python /usr/lib/python/xen/util/pci.py 0 2 0 0
> 
> And send us the output (or attach to the bugzilla report and let us 
> know on the list).
> 
>   -- Keir
Based on a quick glance at the source code of the tg3 linux device
driver, it appears that it is one of those PCI devices which doesn''t do
what the specification recommends and puts device specific registers in
the PCI configuration space. I guess there''s nothing really bad about
that, but it will cause breakage with PCI driver domains in Xen. The PCI
backend intercepts all reads/writes to the PCI configuration space. It
blocks almost all writes and allows almost all reads. I suspect that the
driver is trying to write to the configuration space on the card and is
being denied by the PCI backend.

Alex, try turning on verbose request in the backend and only enable your
ethernet card in the frontend. Do a:

echo 1 > /sys/modules/pciback/parameters/verbose_request

or boot you dom0 with "pciback.verbose_request=1" on the kernel
command-line. Then start up the domain that has the ethernet device.
This will cause a lot of debug output to appear in the kernel log in
dom0. In particular, look for lines that begin with "pciback:
0000:02:00.0: write request " in your logs. Some write requests are ok
(like the one to the PCI_COMMAND register), but most others will be
denied. If you could add this output to your bugzilla entry, that would
be helpful. You can match the register offsets with standard PCI
registers in include/linux/pci_regs.h and you may be able to match the
writes to registers specific to your card in drivers/net/tg3.h

I''ve suspected that such cards exist (that use device-specific
registers
in the configuration space), but have not yet seen one in practice. If
this is indeed the problem, one solution would be to detect the card in
the PCI backend and modify its virtual configuration space to allow
writes to those specific registers (a pciback "quirks" capability
similar to the quirks capability for PCI devices in Linux). This
wouldn''t be difficult, but it could get out-of-control if we have to do
that for a lot of cards. Another possibility is to add some kind of flag
that allows all writes to work that are not specifically blocked. I
don''t like that idea at all (default permit is not a good security
posture), but would allow people with devices like this one to get them
working (albeit with less protection). Any other suggestions?

Ryan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8 Mar 2006, at 15:25, Ryan wrote:
> I''ve suspected that such cards exist (that use device-specific 
> registers
> in the configuration space), but have not yet seen one in practice. If
> this is indeed the problem, one solution would be to detect the card in
> the PCI backend and modify its virtual configuration space to allow
> writes to those specific registers (a pciback "quirks" capability
> similar to the quirks capability for PCI devices in Linux). This
> wouldn''t be difficult, but it could get out-of-control if we have
to do
> that for a lot of cards. Another possibility is to add some kind of 
> flag
> that allows all writes to work that are not specifically blocked. I
> don''t like that idea at all (default permit is not a good security
> posture), but would allow people with devices like this one to get them
> working (albeit with less protection). Any other suggestions?
It depends what the threat model is. In most scenarios device drivers 
will only be installed by administrators or main users of a system, and 
it is reasonable to assume they are not actively malicious. The main 
aim of driver domains is to protect against stupid driver bugs not 
smart malicious drivers.

Being as restrictive as possible by default does make sense, but we 
need an easy way to make things more permissive so that people can get 
drivers working while still get some (in fact, probably most) of the 
benefit of driver domains (that being restartability when a driver 
shoots itself in the foot).

To this end I''ve checked in a ''permissive'' module
flag which allows PCI
config writes to succeed by default. Enabled by specifying 
''pciback.permissive'' as a boot parameter or:
  echo Y >/sys/modules/pciback/permissive

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2006-03-08 at 16:26 +0000, Keir Fraser wrote:
> On 8 Mar 2006, at 15:25, Ryan wrote:
> 
> > I''ve suspected that such cards exist (that use
device-specific
> > registers
> > in the configuration space), but have not yet seen one in practice. If
> > this is indeed the problem, one solution would be to detect the card
in
> > the PCI backend and modify its virtual configuration space to allow
> > writes to those specific registers (a pciback "quirks"
capability
> > similar to the quirks capability for PCI devices in Linux). This
> > wouldn''t be difficult, but it could get out-of-control if we
have to do
> > that for a lot of cards. Another possibility is to add some kind of 
> > flag
> > that allows all writes to work that are not specifically blocked. I
> > don''t like that idea at all (default permit is not a good
security
> > posture), but would allow people with devices like this one to get
them
> > working (albeit with less protection). Any other suggestions?
> 
> It depends what the threat model is. In most scenarios device drivers 
> will only be installed by administrators or main users of a system, and 
> it is reasonable to assume they are not actively malicious. The main 
> aim of driver domains is to protect against stupid driver bugs not 
> smart malicious drivers.
That is true. However, it seems to me that most people don''t put the
driver alone in the driver domain. Rather, they put the device in a domU
they wish to run many applications in. If the domain is compromised, an
attacker could cause a *little* bit of havoc on the pci bus (affecting
devices in other domains) by messing with registers like the BARs or
possibly DoS other devices by changing the interrupt pin. 
> 
> Being as restrictive as possible by default does make sense, but we 
> need an easy way to make things more permissive so that people can get 
> drivers working while still get some (in fact, probably most) of the 
> benefit of driver domains (that being restartability when a driver 
> shoots itself in the foot).
> 
> To this end I''ve checked in a ''permissive''
module flag which allows PCI
> config writes to succeed by default. Enabled by specifying 
> ''pciback.permissive'' as a boot parameter or:
>   echo Y >/sys/modules/pciback/permissive
> 
I fear that providing an easy way around the security will mean that it
will just get turned off rather than forcing developers to correct the
problem. I think this should be viewed as a workaround and some more
permanent solution be put in place (perhaps the "quirks" method? see
the
pci_fixup_device method in Linux). I''m going to try and prepare an
example of what I think is the correct way to fix this issue.

If you need to include a "permissive" mode, instead of allowing
permissive mode blankly for all devices in all driver domains, could we
instead just limit it to a single device or single domain? I think the
current permissive flag opens things up unnecessarily.

Keir, I don''t believe your patch is entirely correct. If you write a
double-word and one byte or one word of that double-word gets picked up
in a field in the virtual configuration space overlay but not the other
bytes or word, the handled flag will still get set to true, but the
remaining bytes will never be written to the device. I think correctly
implementing this is non-trivial as you''d need to put some code up
above
to track which bytes have been handled and which ones haven''t. However,
I think it''s unlikely that you''d ever have this condition
actually
affect a device given the current virtual configuration space.

Ryan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>
> To this end I''ve checked in a ''permissive''
module flag which allows PCI
> config writes to succeed by default. Enabled by specifying
> ''pciback.permissive'' as a boot parameter or:
>   echo Y >/sys/modules/pciback/permissive
>
>   -- Keir
Hi Keir, hi Ryan,

good job! I recompiled with the mentioned changeset and it works.
At the moment I''m writing this mail from my dom0 via a bridge created
with the
delegated NIC. Thank''s a lot.

I just had one issue with a kernel oops when I set up the bridge in domU

brctl addbr br0
brctl addif eth0 (the vif)
brctl addif eth1 (the delegated NIC)

ifconfig br0 192.168.1.100 up

crash

Aiee, killing interrupt....

Unfortunately I was not able to catch the output, because the domU crashed 
completely.

But this is what I could get from dom0''s dmesg after crash:

<snip>

ACPI: PCI interrupt for device 0000:02:00.0 disabled
irq 16: nobody cared (try booting with the "irqpoll" option)

Call Trace: <IRQ> <ffffffff8010722a>{hypercall_page+554}
       <ffffffff801526a5>{__report_bad_irq+53} 
<ffffffff80152923>{note_interrupt+547}
       <ffffffff80274b3a>{usb_hcd_irq+42}
<ffffffff80151f66>{__do_IRQ+214}
       <ffffffff8010dee8>{do_IRQ+72}
<ffffffff8025be55>{evtchn_do_upcall+149}
       <ffffffff8010bb19>{do_hypervisor_callback+37} <EOI>
handlers:
[<ffffffff80274b10>] (usb_hcd_irq+0x0/0x60)
Disabling IRQ #16

</snip>

After that it was also not possible to reboot the domU, so I had to reboot 
dom0 as well.

But this happened only once, after rebooting I couldn''t reproduce that
oops
anymore. Will do further tests later.

What about the info''s requested by you before that permissive-patch?
Are they
still of interest?
I did those tests before I recompiled, so if needed I can still attach them to 
the bugzilla report.

Thanks again,
Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8 Mar 2006, at 23:05, Ryan wrote:
> Keir, I don''t believe your patch is entirely correct. If you write
a
> double-word and one byte or one word of that double-word gets picked up
> in a field in the virtual configuration space overlay but not the other
> bytes or word, the handled flag will still get set to true, but the
> remaining bytes will never be written to the device. I think correctly
> implementing this is non-trivial as you''d need to put some code up
> above
> to track which bytes have been handled and which ones haven''t.
However,
> I think it''s unlikely that you''d ever have this condition
actually
> affect a device given the current virtual configuration space.
Yes, I''m aware of that limitation and it will be a pain to fix 
properly. Actually, it wouldn''t be too bad if the list of config-space 
handlers were in sorted order. Then we could deal with the necessary 
''gaps'' as we go.

Allowing permissive setting at a finer granularity is also a good idea. 
I see no problem with keeping the global flag too though. It defaults 
to safety (off), and who are we to say that a user or administrator 
should not be allowed to explicitly set such a global policy as they 
see fit?

The problem with the ''proper'' fix is that it can never really
be
completed -- there''ll always be some new piece of hardware that needs 
to be accounted for and which cannot be controlled from a driver domain 
without extra patching.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8 Mar 2006, at 23:05, Ryan wrote:
> That is true. However, it seems to me that most people don''t put
the
> driver alone in the driver domain. Rather, they put the device in a 
> domU
> they wish to run many applications in. If the domain is compromised, an
> attacker could cause a *little* bit of havoc on the pci bus (affecting
> devices in other domains) by messing with registers like the BARs or
> possibly DoS other devices by changing the interrupt pin.
I guess that is true for some users, even though it was never our model 
for driver domains. Of course the BARs are actually off limits, since 
you have explicit handlers for those. :-) But your point is taken. 
Having explicit handlers at least for some common devices is not a bad 
idea, to allow some common system configurations to run without needing 
to set permissive mode.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel