Re: [Xen-users] using USB Host Controller in domU

Marcus Brown
Tue, 02 Aug 2005 11:22:13 -0000

Hi Andreas,
Thanks for asking, I figure it is time to start organising some of my notes.
This is what I''ve got so far, perhaps you''ll find something in
them?

*****************************************************************************************************************
Using PCI devices in a driver domain (a privileged domU?)
===================================================================The examples
below use the following domains.
wench:        dom0 on P3/650 on Asus P2BVM, 512MB RAM
"Production" Xen Server
brutus:        dom0 on P2/350 on Asus P2B, 192MB ECC RAM
"Development" Xen Server
BluetoothProxy:    domU on wench (in use)
XenFirewall:    domU on brutus (WIP)

These notes are mostly to clean up my own documentation, and they
_hopefully_ contain
all the information I need to repeat the process.
It is not necessarily HOWTO CREATE DRIVER DOMAINS, but how I got mine to
work.


1. Hide the device from the hypervisor (dom0)

EXAMPLE A: brutus
-----------------------------
This machine has a hidden USB Host Controller, and two network cards,
(the second) one being hidden from brutus.
It has a custom initrd as the root is on LVM.
brutus:~# lspci
0000:00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host
bridge (rev 03)
0000:00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP
bridge (rev 03)
0000:00:04.0 ISA bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 02)
0000:00:04.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01)
0000:00:04.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 02)
0000:00:0b.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX
[Cyclone] (rev 30)
0000:01:00.0 VGA compatible controller: Silicon Integrated Systems [SiS]
86C326 5598/6326 (rev 0b)

brutus:~# tail /boot/grub/menu.lst
title           Debian Xen
root            (hd0,0)
kernel          /xen-2.0-testing.gz dom0_mem=81920 root=/dev/ram0
lvm2root=/dev/mapper/vg0-root physdev_dom0_hide=(00:0c.0)(00:04:2) ro
console=tty0 noreboot
module          /vmlinuz-2.6.11.12-xen0 root=/dev/mapper/vg0-root
console=tty0 ro noreboot
module          /initrd.img-2.6.11.12-xen0
savedefault
boot

EXAMPLE B: wench
------------------------------
This machine has a hidden USB host controller
wench:~# lspci
0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX
Host bridge (rev 03)
0000:00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX
AGP bridge (rev 03)
0000:00:06.0 Multimedia audio controller: ESS Technology ES1969 Solo-1
Audiodrive (rev 01)
0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 02)
0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE
(rev 01)
0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
0000:00:0b.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX
[Cyclone] (rev 64)
0000:01:00.0 VGA compatible controller: ATI Technologies Inc 3D Rage IIC
AGP (rev 7a)

wench:~# tail /boot/grub/menu.lst
title           Xen 2 - Kernel 2.6.11.12
root            (hd0,0)
#kernel          /xen-2.0-testing.gz dom0_mem=131072 root=/dev/hda3 ro
console=tty0
kernel          /xen-2.0-testing.gz dom0_mem=131072 root=/dev/hda3 ro
console=tty0 physdev_dom0_hide=(00:07.2)
                # Marcus appends phys_hide option so BluetoothProxy domU
can have USB Host Controller
module          /vmlinuz-2.6.11.12-xen0 root=/dev/hda3 ro console=tty0
savedefault
boot

===================================================================2. Configure
kernel (& modules) for the driver domain

EXAMPLE A: Firewall on brutus
----------------------------------------------
brutus:~# egrep "XEN|PCI|NET|VORTEX"
/boot/config-2.6.11.12-XenFirewall  | grep -v "#"
CONFIG_XEN=y
CONFIG_ARCH_XEN=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PHYSDEV_ACCESS=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_WRITABLE_PAGETABLES=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XENARCH="i386"
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_NET=y
CONFIG_INET=y
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_INET_TUNNEL=m
CONFIG_NETFILTER=y
CONFIG_BRIDGE_NETFILTER=y
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NETDEVICES=y
CONFIG_NET_ETHERNET=y
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=m
CONFIG_NET_PCI=y
CONFIG_SECURITY_NETWORK=y

EXAMPLE B: BluetoothProxy on wench
----------------------------------------------
wench:~# egrep "XEN|PCI|USB|HCI|BT|VID"
/boot/config-2.6.11.12-BluetoothProxy  | grep -v "#"
CONFIG_XEN=y
CONFIG_ARCH_XEN=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PHYSDEV_ACCESS=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_WRITABLE_PAGETABLES=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XENARCH="i386"
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m
CONFIG_BT_HCIUSB=m
CONFIG_BT_HCIUSB_SCO=y
CONFIG_BT_HCIVHCI=m
CONFIG_VIDEO_DEV=m
CONFIG_USB=m
CONFIG_USB_DEVICEFS=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_STV680=m

===================================================================3.
Configuration of driver domain

wench:~# grep = /etc/xen/BluetoothProxy
#===========================================================================kernel
= "/boot/vmlinuz-2.6.11.12-BluetoothProxy"
pci = [''00,7,2'']
memory = 80
name = "BluetoothProxy"
disk = [ ''phy:/dev/vg0/BluetoothProxy,/dev/hda3,w'']
root = "/dev/hda3 ro"
extra = "4"
restart = ''onreboot''
#===========================================================================

===================================================================4. Confirming
results

EXAMPLE A: firewall on brutus
---------------------------------------------
tba

EXAMPLE B: BluetoothProxy on wench
----------------------------------------------------------
BluetoothProxy:~# lspci
0000:00:04.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB (rev 01)
(Yep, that''s ALL it comes up with!:))

BluetoothProxy:~# lsmod
Module                  Size  Used by
stv680                 26060  0
ipt_MASQUERADE          2624  0
iptable_nat            19836  1 ipt_MASQUERADE
ip_conntrack           39224  2 ipt_MASQUERADE,iptable_nat
ip_tables              19904  2 ipt_MASQUERADE,iptable_nat
rfcomm                 35132  0
bnep                   12256  2
crc32                   3808  1 bnep
l2cap                  22340  9 rfcomm,bnep
videodev                7360  1 stv680
hci_usb                13928  2
bluetooth              44164  8 rfcomm,bnep,l2cap,hci_usb
uhci_hcd               30416  0
usbcore               107672  4 stv680,hci_usb,uhci_hcd

BluetoothProxy:~# lsusb
Bus 001 Device 004: ID 0553:0202 STMicroelectronics Imaging Division
(VLSI Vision) Aiptek PenCam 1
Bus 001 Device 003: ID 0db0:1967 Micro Star International Bluetooth Dongle
Bus 001 Device 001: ID 0000:0000

Using iPaq to connect to BluetoothProxy NAP service, we see:
BluetoothProxy:~#  tail /var/log/syslog
Aug  2 20:31:08 localhost hcid[1739]: link_key_request
(sba=00:10:DC:E9:71:7C, dba=00:04:3E:A3:C7:73)
Aug  2 20:31:08 localhost pand[9838]: New connection from
00:04:3E:A3:C7:73 bnep0

Testing the camera with pencam2 (must remember to null out stv680 in
modules)
This is the look you give when it all just works:
http://marcusbrutus.cust.internode.on.net/Computers/Me/snapshot002

===================================================================5. References

5.1 Driver Domain Configuration
The Xen Users'' Manual, chapter 7.3
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user/user.html#SECTION03230000000000000000

Flat Format: Include a list of PCI device addresses of the format:
pci = [''x,y,z'', ...]

where each element in the list is a string specifying the components of
the PCI device address, separated by commas. The components (x, y and z)
of the list may be formatted as either decimal or hexadecimal.

*****************************************************************************************************************

Andreas Seuss wrote:

> Hi Marcus,
>
> I read your mail on the Xen Users mailing list. What I try to achieve
> is to share a pci network interface card. I compiled a kernel
> according to your config shown below except that I used the NIC
> modules. I run xen-3.0 (devel). When I include the pci =
[''00,08,00'']
> statement into my config file I get the following error when trying to
> create the domain:
>
> Error: Error creating domain: global name ''bus'' is not
defined
>
> I also set the ''physdev_dom0_hide=(00:08.0)'' in the menu
list. Tried
> it on the kernel line and on the module line (not both on the same
> time;-))
> but it does not seem to work. I can still see all pci devices. Maybe
> xou could paste the respective line from your menu.lst?

The hide stanza goes in the kernel line.
Can you confirm the PCI ID with lspci in dom0?

>
> Which documentation did you use? The user manual? It contains only
> sparse information on that topic I think. Thanks in advance.

Yes, the user manual :) (see References above).
Not as verbose as my attempt, huh? :)

> Andreas.
>
>
> Marcus Brown wrote:
>
>> Heh,
>> ipaq turned off, bluez-utils not installed, no IRQ for USB... etc
>> sorry to trouble the list :) ... thanks for reading!!
>>
>> ie. PCI + USB working afaik (not fully tested but looks ok)
>>
>> M.
>>
>> Marcus Brown wrote:
>>
>>
>>
>>> OK, PCI export working now.
>>>
>>> Except that I still can''t configure the bnep0 interface.
>>> Hot plugging seems to work, but dmesg gives:
>>>   SIOCSIFADDR: No such device
>>>   bnep0: ERROR while getting interface flags: No such device
>>>   SIOCSIFNETMASK: No such device
>>>   bnep0: ERROR while getting interface flags: No such device
>>>   Failed to bring up bnep0.
>>>
>>> # lspci
>>> 0000:00:07.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB
>>> (rev 01)
>>> # lsusb
>>> Bus 001 Device 002: ID 0db0:1967 Micro Star International Bluetooth
>>> Dongle
>>> Bus 001 Device 001: ID 0000:0000
>>> # lsmod
>>> Module                  Size  Used by
>>> bnep                   12256  0
>>> crc32                   3808  1 bnep
>>> l2cap                  22340  1 bnep
>>> videodev                7360  0
>>> hci_usb                13928  0
>>> bluetooth              44164  3 bnep,l2cap,hci_usb
>>> uhci_hcd               30416  0
>>> usbcore               107672  3 hci_usb,uhci_hcd
>>>
>>> Any suggestions?
>>>
>>> (Yes, I talk to myself in the blue room too !! :))
>>>
>>> Marcus Brown wrote:
>>>
>>>
>>>
>>>
>>>
>>>> Dear, oh dear!!
>>>>
>>>> pci = [''00,07,2''] in domU config!!!
>>>> (I''m going blind, I think)
>>>>
>>>> doh!
>>>>
>>>> lol
>>>>
>>>> Marcus Brown wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Hi all.
>>>>>
>>>>> I''ve been attempting to export the USB Host
Controller to a domU,
>>>>> so that I can use a USB Bluetooth adapter (and possibly a
USB
>>>>> camera).
>>>>> I''ve been using xen-testing on Debian (dom0 is
sid, domU is sarge)
>>>>>
>>>>> I''ve managed to get the domU to boot using a few
of the following
>>>>> settings:
>>>>>
>>>>> The kernel line in grub''s menu.lst now includes:
>>>>>     physdev_dom0_hide=(00:07.2)
>>>>> The domU kernel config has:
>>>>>      XEN  --->
>>>>>           [*] Physical device access
>>>>>           [*]   Block-device backend driver
>>>>>           [ ]   Network-device backend driver
>>>>>          X86 Processor Configuration  --->
>>>>>            Bus options (PCI, PCMCIA, EISA, MCA, ISA)
--->
>>>>>             [*] PCI support
>>>>>             [*] Legacy /proc/pci interface
>>>>>             [*] PCI device name database
>>>>>          Device Drivers  --->
>>>>>           Networking support  --->
>>>>>            <M> Bluetooth subsystem support  --->
>>>>>             --- Bluetooth subsystem support
>>>>>             <M>   L2CAP protocol support
>>>>>             <M>   SCO links support
>>>>>             <M>   RFCOMM protocol support
>>>>>              [*]     RFCOMM TTY support
>>>>>             <M>   BNEP protocol support
>>>>>              [*]     Multicast filter support
>>>>>              [*]     Protocol filter support
>>>>>             <M>   HIDP protocol support
>>>>>             Bluetooth device drivers  --->
>>>>>              <M> HCI USB driver
>>>>>               [*]   SCO (voice) support
>>>>>              <M> HCI VHCI (Virtual HCI device) driver
>>>>> Device Drivers  --->
>>>>>            Multimedia devices  --->
>>>>>             <M> Video For Linux
>>>>>            USB support  --->
>>>>>             <M> Support for Host-side USB
>>>>>              [*]   USB device filesystem
>>>>>              <M> UHCI HCD (most Intel and VIA)
support
>>>>>             <M> USB STV680 (Pencam) Camera support
>>>>>
>>>>> After booting the domU, the USB modules can be loaded
manually,
>>>>> however, there are no PCI devices found, or USB devices
listed.
>>>>>
>>>>> I have come to the conclusion (from the xen-users manual)
that my
>>>>> xend-config.sxp is in error. I''ve tried appending:
>>>>> (backend (blkif))
>>>>> (device (pci (bus 00) (dev 07) (func 2)))
>>>>>
>>>>> (this could explain why I now get errors from ''xm
list -v'' now:))
>>>>>
>>>>> but during boot of the domU I still get the following:
>>>>> Starting hotplug subsystem:
>>>>>    pci
>>>>> cat: /sys/bus/pci/devices/*/class: No such file or
directory
>>>>> cat: /sys/bus/pci/devices/*/vendor: No such file or
directory
>>>>> cat: /sys/bus/pci/devices/*/device: No such file or
directory
>>>>> cat: /sys/bus/pci/devices/*/subsystem_vendor: No such file
or
>>>>> directory
>>>>> cat: /sys/bus/pci/devices/*/subsystem_device: No such file
or
>>>>> directory
>>>>> Bad PCI agent invocation
>>>>>
>>>>> Can anyone see something obviously wrong here?
>>>>> The users manual talks about placing these options
>>>>> "somewhere within the vm element of its
configuration"
>>>>> but I''m unable to find information on how to do
this.
>>>>>
>>>>> Thanks for any pointers,
>>>>>
>>>>> Marcus.
>>>>>
>>>>> _______________________________________________
>>>>> Xen-users mailing list
>>>>> Xen-users@lists.xensource.com
>>>>> http://lists.xensource.com/xen-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-users mailing list
>>>> Xen-users@lists.xensource.com
>>>> http://lists.xensource.com/xen-users
>>>>
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@lists.xensource.com
>>> http://lists.xensource.com/xen-users
>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users