thr3ads.net - Xen users - [Xen-users] About NIC passthrough to the guest system [Apr 2009]

If this information is useful, please help other people find it:
Share via:

Alex Chan

2009-Apr-27 06:04 UTC

[Xen-users] About NIC passthrough to the guest system

Hi,
        I very confuse how to make xen PCI passthrough to guest. I follow
everything but still cannot get the PCI in the guest. Please help, thanks.

HOST: CentOS 5.3
Kernel : 2.6.18-128.1.6.el5xen
xen: xen-3.0.3-80.el5

[root@hm02 xen]# lspci | grep Ethernet
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit
Ethernet (rev 12)
07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit
Ethernet (rev 12)
08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
0a:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
0a:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
0d:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit
Ethernet (rev 12)

I need "0a:00.1" to passthrough to the guest system.


[root@hm02 xen]# cat /etc/modprobe.conf
options pciback hide=(0a:00.1)
alias eth0 bnx2
alias eth1 bnx2
alias eth2 bnx2
alias eth3 e1000e
alias eth4 e1000e
alias eth5 e1000e
alias eth6 e1000e
alias scsi_hostadapter megaraid_sas
alias scsi_hostadapter1 ata_piix
alias scsi_hostadapter2 usb-storage

[root@hm02 xen]# dmesg | grep pciback
pciback 0000:0a:00.1: seizing device
pciback 0000:0a:00.1: enabling permissive mode configuration space accesses!
pciback 0000:0a:00.1: permissive mode is potentially unsafe!
pciback: vpci: 0000:0a:00.1: assign to virtual slot 0

[root@hm02 xen]# cat xen-test
name = "xen-test"
uuid = "cad318b5-be8b-2a75-bd02-9fad747f9e04"
maxmem = 1024
memory = 1024
vcpus = 1
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "c"
pae = 1
acpi = 1
apic = 1
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
device_model = "/usr/lib64/xen/bin/qemu-dm"
sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [ "file:/xen/xen-test.img,hda,w", ",hdc:cdrom,r" ]
vif = [ "mac=00:16:3e:0f:f9:e5,bridge=xenbr5" ]
pci = [ "0000:0a:00.1" ] or pci = [ "0a:00.1" ] 
serial = "pty"

[root@hm02 xen]# cat /boot/config-2.6.18-128.1.6.el5xen | grep XEN
CONFIG_X86_64_XEN=y
CONFIG_X86_XEN_GENAPIC=y
CONFIG_XEN_PCIDEV_FRONTEND=y
# CONFIG_XEN_PCIDEV_FE_DEBUG is not set
CONFIG_NETXEN_NIC=m
CONFIG_XEN=y
CONFIG_XEN_INTERFACE_VERSION=0x00030203
# XEN
CONFIG_XEN_PRIVILEGED_GUEST=y
# CONFIG_XEN_UNPRIVILEGED_GUEST is not set
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_XENBUS_DEV=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_XEN_BLKDEV_TAP=m
CONFIG_XEN_NETDEV_BACKEND=m
# CONFIG_XEN_NETDEV_PIPELINED_TRANSMITTER is not set
CONFIG_XEN_NETDEV_LOOPBACK=m
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
# CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
# CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
# CONFIG_XEN_PCIDEV_BE_DEBUG is not set
# CONFIG_XEN_TPMDEV_BACKEND is not set
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_FRAMEBUFFER=y
CONFIG_XEN_KEYBOARD=y
CONFIG_XEN_SCRUB_PAGES=y
# CONFIG_XEN_DISABLE_SERIAL is not set
CONFIG_XEN_SYSFS=y
CONFIG_XEN_COMPAT_030002_AND_LATER=y
# CONFIG_XEN_COMPAT_LATEST_ONLY is not set
CONFIG_XEN_COMPAT_030002=y
CONFIG_XEN_UTIL=y
CONFIG_XEN_BALLOON=y
CONFIG_XEN_DEVMEM=y
CONFIG_XEN_SKBUFF=y
CONFIG_XEN_REBOOT=y
CONFIG_XEN_SMPBOOT=y
[root@hm02 xen]#

In the guest:

[root@xen-test ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Class ff80: XenSource, Inc. Xen Platform Device (rev 01)
00:04.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8139/8139C/8139C+ (rev 20)

I still cannot see the PCI passthrough, please help to see where I missed, thank
you so much.

Best Regards
Alex


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Christian Tramnitz

2009-Apr-27 10:36 UTC

head link

[Xen-users] Re: About NIC passthrough to the guest system

When running hvm you need vt-d support to pass-through devices.
Apart from that I doubt that you will be able to passthrough one port of 
a dual-port nic, you may have to pass-through the whole card.

Best regards,
    Christian

Alex Chan wrote:> Hi,
>         I very confuse how to make xen PCI passthrough to guest. I 
> follow everything but still cannot get the PCI in the guest. Please 
> help, thanks.
>  
> HOST: CentOS 5.3
> Kernel : 2.6.18-128.1.6.el5xen
> xen: xen-3.0.3-80.el5
>  
> [root@hm02 xen]# lspci | grep Ethernet
> 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 
> Gigabit Ethernet (rev 12)
> 07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 
> Gigabit Ethernet (rev 12)
> 08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (rev 06)
> 08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (rev 06)
> 0a:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (rev 06)
> 0a:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (rev 06)
> 0d:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 
> Gigabit Ethernet (rev 12)
>  
> I need "0a:00.1" to passthrough to the guest system.
>  
>  
> [root@hm02 xen]# cat /etc/modprobe.conf
> options pciback hide=(0a:00.1)
> alias eth0 bnx2
> alias eth1 bnx2
> alias eth2 bnx2
> alias eth3 e1000e
> alias eth4 e1000e
> alias eth5 e1000e
> alias eth6 e1000e
> alias scsi_hostadapter megaraid_sas
> alias scsi_hostadapter1 ata_piix
> alias scsi_hostadapter2 usb-storage
>  
> [root@hm02 xen]# dmesg | grep pciback
> pciback 0000:0a:00.1: seizing device
> pciback 0000:0a:00.1: enabling permissive mode configuration space
accesses!
> pciback 0000:0a:00.1: permissive mode is potentially unsafe!
> pciback: vpci: 0000:0a:00.1: assign to virtual slot 0
>  
> [root@hm02 xen]# cat xen-test
> name = "xen-test"
> uuid = "cad318b5-be8b-2a75-bd02-9fad747f9e04"
> maxmem = 1024
> memory = 1024
> vcpus = 1
> builder = "hvm"
> kernel = "/usr/lib/xen/boot/hvmloader"
> boot = "c"
> pae = 1
> acpi = 1
> apic = 1
> localtime = 0
> on_poweroff = "destroy"
> on_reboot = "restart"
> on_crash = "restart"
> device_model = "/usr/lib64/xen/bin/qemu-dm"
> sdl = 0
> vnc = 1
> vncunused = 1
> keymap = "en-us"
> disk = [ "file:/xen/xen-test.img,hda,w", ",hdc:cdrom,r"
]
> vif = [ "mac=00:16:3e:0f:f9:e5,bridge=xenbr5" ]
> pci = [ "0000:0a:00.1" ] or pci = [ "0a:00.1" ]
> serial = "pty"
>  
> [root@hm02 xen]# cat /boot/config-2.6.18-128.1.6.el5xen | grep XEN
> CONFIG_X86_64_XEN=y
> CONFIG_X86_XEN_GENAPIC=y
> CONFIG_XEN_PCIDEV_FRONTEND=y
> # CONFIG_XEN_PCIDEV_FE_DEBUG is not set
> CONFIG_NETXEN_NIC=m
> CONFIG_XEN=y
> CONFIG_XEN_INTERFACE_VERSION=0x00030203
> # XEN
> CONFIG_XEN_PRIVILEGED_GUEST=y
> # CONFIG_XEN_UNPRIVILEGED_GUEST is not set
> CONFIG_XEN_PRIVCMD=y
> CONFIG_XEN_XENBUS_DEV=y
> CONFIG_XEN_BACKEND=y
> CONFIG_XEN_BLKDEV_BACKEND=m
> CONFIG_XEN_BLKDEV_TAP=m
> CONFIG_XEN_NETDEV_BACKEND=m
> # CONFIG_XEN_NETDEV_PIPELINED_TRANSMITTER is not set
> CONFIG_XEN_NETDEV_LOOPBACK=m
> CONFIG_XEN_PCIDEV_BACKEND=y
> CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
> # CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
> # CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
> # CONFIG_XEN_PCIDEV_BE_DEBUG is not set
> # CONFIG_XEN_TPMDEV_BACKEND is not set
> CONFIG_XEN_BLKDEV_FRONTEND=m
> CONFIG_XEN_NETDEV_FRONTEND=m
> CONFIG_XEN_FRAMEBUFFER=y
> CONFIG_XEN_KEYBOARD=y
> CONFIG_XEN_SCRUB_PAGES=y
> # CONFIG_XEN_DISABLE_SERIAL is not set
> CONFIG_XEN_SYSFS=y
> CONFIG_XEN_COMPAT_030002_AND_LATER=y
> # CONFIG_XEN_COMPAT_LATEST_ONLY is not set
> CONFIG_XEN_COMPAT_030002=y
> CONFIG_XEN_UTIL=y
> CONFIG_XEN_BALLOON=y
> CONFIG_XEN_DEVMEM=y
> CONFIG_XEN_SKBUFF=y
> CONFIG_XEN_REBOOT=y
> CONFIG_XEN_SMPBOOT=y
> [root@hm02 xen]#
> In the guest:
>  
> [root@xen-test ~]# lspci
> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev
02)
> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE 
> [Natoma/Triton II]
> 00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
> 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
> 00:03.0 Class ff80: XenSource, Inc. Xen Platform Device (rev 01)
> 00:04.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
> RTL-8139/8139C/8139C+ (rev 20)
> I still cannot see the PCI passthrough, please help to see where I 
> missed, thank you so much.
> 
> Best Regards
> Alex
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Paul Schulze

2009-Apr-27 11:49 UTC

head link

Re: [Xen-users] Re: About NIC passthrough to the guest system

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Christian and Alex,

On 27 Apr 2009, at 12:36, Christian Tramnitz wrote:
> When running hvm you need vt-d support to pass-through devices.
> Apart from that I doubt that you will be able to passthrough one  
> port of a dual-port nic, you may have to pass-through the whole card.
If the DomU is starting without displaying any error, I think it is  
safe to assume, that it is running on hardware with VT-d support,  
since in my experience, a HVM DomU does not start at all, if the guest  
config sets up a device for PCI passthrough (I could be wrong here,  
haven''t tried it lately).

That depends on the NICs design. I own a D-Link DFE-580TX (4 ports)  
and on that one, each port is more or less stand-alone, not sharing  
anything with the other chips on the board. My current setup (2 ports  
in Dom0, 2 ports passed through to my IPCop DomU) works just fine with  
that. That Intel NIC looks like a similar setup though, so it should  
work somehow. However, I am currently running my DomU in PV mode, but  
with only a few modifications, Alex should be able to do that too, if  
necessary. It is also important to take a good look at the IRQs (lspci  
- -v) and keep the sharing of IRQs between all Xen instances to a  
minimum. Oh yeah and never pass a device to a DomU that shares an IRQ  
with your HDD controller, be it SCSI, SATA or IDE... that can really  
lead to "unexpected" results.
> Alex Chan wrote:
> [...]
>>  [root@hm02 xen]# cat /etc/modprobe.conf
>> options pciback hide=(0a:00.1)
>> alias eth0 bnx2
>> alias eth1 bnx2
>> alias eth2 bnx2
>> alias eth3 e1000e
>> alias eth4 e1000e
>> alias eth5 e1000e
>> alias eth6 e1000e
>> alias scsi_hostadapter megaraid_sas
>> alias scsi_hostadapter1 ata_piix
>> alias scsi_hostadapter2 usb-storage
> [...]

And finally, I looked over your config, Alex, and it looks ok, but I  
am wondering, why you define pciback.hide in /etc/modprobe.conf  
instead of the kernel command-line. Since I assume, the device driver  
for your NIC is built as a module, and /etc/modprobe.conf should be  
accounted on loading it, this might be correct. But pciback is built  
into your kernel anyway so to avoid any doubt on whether it is or not,  
I would suggest putting it on the command-line (simply add  
pciback.hide=(0a:00.1)). This might not help at all, but at least then  
you can be sure that nothing else happens to the PCI device besides  
being seized by pciback.

I hope this helps


Paul.

- --
Paul Schulze
Mail: avlex82@gmail.com

Why can''t a programmer tell the difference between Halloween and  
Christmas?
Because OCT31 = DEC25.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iEYEARECAAYFAkn1m9YACgkQj2zIQLJNnKManQCfVhQlcLQFUsyxhwbwRqBXrota
S54An2sMkqxVCUdAW4CSv5WQqNE/Ir79
=fozf
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Alex Chan

2009-Apr-28 07:17 UTC

head link

Re: [Xen-users] Re: About NIC passthrough to the guest system

Hi Christian and Paul,

        I try to see the IRQ, /proc/interrupts , there is no IRQ that
(0a:00.1) is using. So is that mean the post is hide in the host so cannot
see the IRQ?
Also I see some web page that they add

"install xxxx /sbin/modprobe pciback ;
/sbin/modprobe --first-time --ignore-install xxxx"

in the /etc/modprobe.conf, is that need to do that ?

Also when I add "pciback.hide=(0a:00.1)" in the command line of
grub.conf,
it says "Unknown boot option pciback.hide". So I cannot add it in the
kernel.

I am wondering is that I miss some config for the PCI passthrough ?

Thank you so much.

Best Regards
Alex

----- Original Message ----- 
From: "Paul Schulze" <avlex@gmx.net>
To: "Christian Tramnitz" <chris.ace@gmx.net>
Cc: <xen-users@lists.xensource.com>
Sent: Monday, April 27, 2009 7:49 PM
Subject: Re: [Xen-users] Re: About NIC passthrough to the guest system


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Christian and Alex,

On 27 Apr 2009, at 12:36, Christian Tramnitz wrote:
> When running hvm you need vt-d support to pass-through devices.
> Apart from that I doubt that you will be able to passthrough one
> port of a dual-port nic, you may have to pass-through the whole card.
If the DomU is starting without displaying any error, I think it is
safe to assume, that it is running on hardware with VT-d support,
since in my experience, a HVM DomU does not start at all, if the guest
config sets up a device for PCI passthrough (I could be wrong here,
haven''t tried it lately).

That depends on the NICs design. I own a D-Link DFE-580TX (4 ports)
and on that one, each port is more or less stand-alone, not sharing
anything with the other chips on the board. My current setup (2 ports
in Dom0, 2 ports passed through to my IPCop DomU) works just fine with
that. That Intel NIC looks like a similar setup though, so it should
work somehow. However, I am currently running my DomU in PV mode, but
with only a few modifications, Alex should be able to do that too, if
necessary. It is also important to take a good look at the IRQs (lspci
- -v) and keep the sharing of IRQs between all Xen instances to a
minimum. Oh yeah and never pass a device to a DomU that shares an IRQ
with your HDD controller, be it SCSI, SATA or IDE... that can really
lead to "unexpected" results.
> Alex Chan wrote:
> [...]
>>  [root@hm02 xen]# cat /etc/modprobe.conf
>> options pciback hide=(0a:00.1)
>> alias eth0 bnx2
>> alias eth1 bnx2
>> alias eth2 bnx2
>> alias eth3 e1000e
>> alias eth4 e1000e
>> alias eth5 e1000e
>> alias eth6 e1000e
>> alias scsi_hostadapter megaraid_sas
>> alias scsi_hostadapter1 ata_piix
>> alias scsi_hostadapter2 usb-storage
> [...]

And finally, I looked over your config, Alex, and it looks ok, but I
am wondering, why you define pciback.hide in /etc/modprobe.conf
instead of the kernel command-line. Since I assume, the device driver
for your NIC is built as a module, and /etc/modprobe.conf should be
accounted on loading it, this might be correct. But pciback is built
into your kernel anyway so to avoid any doubt on whether it is or not,
I would suggest putting it on the command-line (simply add
pciback.hide=(0a:00.1)). This might not help at all, but at least then
you can be sure that nothing else happens to the PCI device besides
being seized by pciback.

I hope this helps


Paul.

- --
Paul Schulze
Mail: avlex82@gmail.com

Why can''t a programmer tell the difference between Halloween and
Christmas?
Because OCT31 = DEC25.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iEYEARECAAYFAkn1m9YACgkQj2zIQLJNnKManQCfVhQlcLQFUsyxhwbwRqBXrota
S54An2sMkqxVCUdAW4CSv5WQqNE/Ir79
=fozf
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Xen users - Apr 2009 - About NIC passthrough to the guest system

[Xen-users] About NIC passthrough to the guest system

[Xen-users] Re: About NIC passthrough to the guest system

Re: [Xen-users] Re: About NIC passthrough to the guest system

Re: [Xen-users] Re: About NIC passthrough to the guest system