Rajesh Kumar Mallah
2012-Aug-20 12:34 UTC
[libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
Dear List ,
We are using intel 82599EB based SRIOV capable 10GbE cards
on the host machine.The VFs are being assigned to the guest
in pass through mode and are providing the network connectivity
as expected.
However when we shutdown the guest the connectivity does not
comes back via the VFs on the guest in next boot unless
the VFs are detached from the *guest* *before* initiating a
shutdown of the guests.
(
the detach is explicitly done via virsh command
/usr/local/libvirt/bin/virsh detach-device guest01 ~/nic.xml
nic.xml contained
<hostdev mode='subsystem' type='pci'
managed='yes'>
<source>
<address domain='0x0000' bus='0x04' slot='0x10'
function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</hostdev>
)
The question is anyone else facing similar problem and
if so what solution was used?
Secondly , is there anyway to automate the detaching of the
VFs from the guest when the guest is put in the shutdown
sequence ? (or for that matter destroyed).
Regards,
Rajesh Kumar Mallah
Tel. 91-11-46710500 (Ext:303)
Cell +919811255597
Website: http://www.tradeindia.com/
\|/ \|/ \|/ stop printing \|/ \|/ \|/
| | | start planting | | |
Alex Jia
2012-Aug-21 03:01 UTC
[libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
On 08/20/2012 08:34 PM, Rajesh Kumar Mallah wrote:> Dear List , > > We are using intel 82599EB based SRIOV capable 10GbE cards > on the host machine.The VFs are being assigned to the guest > in pass through mode and are providing the network connectivity > as expected. > > However when we shutdown the guest the connectivity does not > comes back via the VFs on the guest in next boot unless > the VFs are detached from the *guest* *before* initiating a > shutdown of the guests.Dear Rajesh, Please see the following comments.> ( > the detach is explicitly done via virsh command > /usr/local/libvirt/bin/virsh detach-device guest01 ~/nic.xml > nic.xml contained > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> > </hostdev> > ) > > The question is anyone else facing similar problem and > if so what solution was used? > > Secondly , is there anyway to automate the detaching of the > VFs from the guest when the guest is put in the shutdown > sequence ? (or for that matter destroyed).When *|managed|* is *"yes"* for a PCI device, it is automatically detached from the host before being passed on to the guest, and automatically *reattached* to the host after the *guest exits* such as shutdown a guest. Regards, Alex> > Regards, > Rajesh Kumar Mallah > Tel. 91-11-46710500 (Ext:303) > Cell +919811255597 > Website: http://www.tradeindia.com/ > \|/ \|/ \|/ stop printing \|/ \|/ \|/ > | | | start planting | | | > > _______________________________________________ > libvirt-users mailing list > libvirt-users at redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20120821/6ff6985c/attachment.htm>
Rajesh Kumar Mallah
2012-Aug-21 05:30 UTC
[libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
Dear Alex , Thanks Once again for responding. I tried to replicate your steps. in our case readlink always reveals that the pci device is being handled by the pci-stub driver (as in our case ixgbevf is absent in the host) Below is an annotaded sequence of events. Check status : # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub Content of the nic.xml: root at debian-squeeze:/usr/local/libvirt/etc/libvirt/qemu# cat /root/nic.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub corrent connectivity is OK # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data. 64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.491 ms 64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.384 ms 64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.310 ms ^C --- 192.168.101.10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.310/0.395/0.491/0.074 ms # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub Detach and reattach and check connectivity in running guest. note: the guest has pci hotplug support , the ethernet interfaces are configured to allow hotplug and IP is statically assigned. # virsh detach-device new-installed /root/nic.xml Device detached successfully # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub root at debian-squeeze:/usr/local/libvirt/etc/libvirt/qemu# ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.>From 192.168.100.6 icmp_seq=9 Destination Host Unreachable >From 192.168.100.6 icmp_seq=10 Destination Host Unreachable >From 192.168.100.6 icmp_seq=11 Destination Host Unreachable^C --- 192.168.101.10 ping statistics --- 14 packets transmitted, 0 received, +3 errors, 100% packet loss, time 13096ms pipe 3 # virsh attach-device new-installed /root/nic.xml Device attached successfully # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data. 64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.507 ms 64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.388 ms 64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.314 ms 64 bytes from 192.168.101.10: icmp_req=4 ttl=64 time=0.315 ms 64 bytes from 192.168.101.10: icmp_req=5 ttl=64 time=0.335 ms ^C --- 192.168.101.10 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3998ms rtt min/avg/max/mdev = 0.314/0.371/0.507/0.076 ms # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub Above proves that detach-attach cycle was successful. # virsh detach-device new-installed /root/nic.xml Device detached successfully Please note that guest is being shutdown with nic detached. # virsh shutdown new-installed Domain new-installed is being shutdown Guest being started again : # virsh start new-installed Domain new-installed started # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data. 64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.583 ms 64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.402 ms 64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.308 ms 64 bytes from 192.168.101.10: icmp_req=4 ttl=64 time=0.316 ms 64 bytes from 192.168.101.10: icmp_req=5 ttl=64 time=0.318 ms 64 bytes from 192.168.101.10: icmp_req=6 ttl=64 time=0.247 ms ^C --- 192.168.101.10 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 4997ms rtt min/avg/max/mdev = 0.247/0.362/0.583/0.109 ms Please note on starting the guest the PCI device came up and connectivity was established. The device was also defined inside the guest XML. Now shutting down the Guest without detaching. # virsh shutdown new-installed Domain new-installed is being shutdown # virsh start new-installed Domain new-installed started # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.>From 192.168.100.6 icmp_seq=9 Destination Host Unreachable >From 192.168.100.6 icmp_seq=10 Destination Host Unreachable >From 192.168.100.6 icmp_seq=11 Destination Host Unreachable^C --- 192.168.101.10 ping statistics --- 14 packets transmitted, 0 received, +3 errors, 100% packet loss, time 13095ms pipe 3 CONNECTION IS LOST We are using latest libvirtd (0.9.13) , lastest ixgbe/ixgbevf (from sf) , lastest stable kernel (3.5.1) Regards, Rajesh Kumar Mallah Tel. 91-11-46710500 (Ext:303) Cell +919811255597 Website: http://www.tradeindia.com/ \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | start planting | | | ----- "Alex Jia" <ajia at redhat.com> wrote: | From: "Alex Jia" <ajia at redhat.com> | To: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | Cc: libvirt-users at redhat.com | Sent: Tuesday, August 21, 2012 9:34:24 AM | Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage. | | On 08/21/2012 11:22 AM, Rajesh Kumar Mallah wrote: | | Dear Alex / List , | | Thanks for the response. | | The problem is regarding detaching from the guest not host. | | I see, if you attach a PCI device into guest with 'managed' mode(managed='yes') then the device is automatically detached from guest and returned to host after guest exiting, for example. | | # lspci | grep Eth | 00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02) | | # readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f | /sys/bus/pci/drivers/e1000e | | # virsh start foo | Domain foo started | | # ping -c 3 xx.xx.xx.xx | PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data. | 64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=1 ttl=49 time=87.8 ms | 64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=2 ttl=49 time=85.2 ms | 64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=3 ttl=49 time=91.8 ms | --- xx.xx.xx.xx ping statistics --- | 3 packets transmitted, 3 received, 0% packet loss, time 2093ms | rtt min/avg/max/mdev = 85.240/88.327/91.881/2.731 ms | | # cat hostdev.xml | <hostdev mode='subsystem' type='pci' managed='yes'> | <source> | <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> | </source> | </hostdev> | | # virsh attach-device foo hostdev.xml | Device attached successfully | | # readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f | /sys/bus/pci/drivers/pci-stub | | # ping -c 3 xx.xx.xx.xx | connect: Network is unreachable | | # virsh destroy foo | Domain foo destroyed | | # readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f | /sys/bus/pci/drivers/e1000e | | # ping -c 3 xx.xx.xx.xx | PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data. | 64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=49 time=79.7 ms | 64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=49 time=83.2 ms | 64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=49 time=84.5 ms | | --- xx.xx.xx.xx ping statistics --- | 3 packets transmitted, 3 received, 0% packet loss, time 2087ms | rtt min/avg/max/mdev = 79.714/82.480/84.513/2.040 ms | | Notes, need to wait for local network recovery again then check network connectivity. | | If your network still is disconnect, it may be a bug IMHO, please show your libvirt | and qemu-kvm version, or my example is not what you want, please let me know. | | Thanks, | Alex | | | In our case since ixgbevf driver is NOT present in the host hence the virtual | interfaces never really attach themselves to the host. | | Regards, | Rajesh Kumar Mallah | Tel. 91-11-46710500 (Ext:303) | Cell +919811255597 | Website: http://www.tradeindia.com/ | \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | | start planting | | | | | | ----- "Alex Jia" <ajia at redhat.com> wrote: | | From: "Alex Jia" <ajia at redhat.com> | | To: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | | Cc: libvirt-users at redhat.com | | Sent: Tuesday, August 21, 2012 8:31:21 AM | | Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage. | | | | On 08/20/2012 08:34 PM, Rajesh Kumar Mallah wrote: Dear List , We are using intel 82599EB based SRIOV capable 10GbE cards on the host machine.The VFs are being assigned to the guest in pass through mode and are providing the network connectivity as expected. However when we shutdown the guest the connectivity does not comes back via the VFs on the guest in next boot unless the VFs are detached from the *guest* *before* initiating a shutdown of the guests. Dear Rajesh, | | | | Please see the following comments. | | | | ( the detach is explicitly done via virsh command /usr/local/libvirt/bin/virsh detach-device guest01 ~/nic.xml nic.xml contained <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> ) The question is anyone else facing similar problem and if so what solution was used? Secondly , is there anyway to automate the detaching of the VFs from the guest when the guest is put in the shutdown sequence ? (or for that matter destroyed). | | When managed is "yes" for a PCI device, it is automatically detached from the host before being passed on to the guest, and automatically reattached to the host after the guest exits such as shutdown a guest. | | | | Regards, | | Alex | | | | Regards, Rajesh Kumar Mallah Tel. 91-11-46710500 (Ext:303) Cell +919811255597 Website: http://www.tradeindia.com/ \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | start planting | | | _______________________________________________ libvirt-users mailing list libvirt-users at redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users | | | -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20120821/2c2f1edc/attachment.htm>
Rajesh Kumar Mallah
2012-Aug-21 13:13 UTC
[libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
Dear Alex / List , I did some changes which i want you to know. I installed ixgbevf on the host and libvirt does attach and detach the VFs from the host and guests. But the connectivity issue is there. root at debian-squeeze:~# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/ixgbevf root at debian-squeeze:~# /usr/local/libvirt/bin/virsh start new-installed Domain new-installed started root at debian-squeeze:~# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub root at debian-squeeze:~# ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.>From 192.168.100.6 icmp_seq=1 Destination Host Unreachable >From 192.168.100.6 icmp_seq=2 Destination Host Unreachable >From 192.168.100.6 icmp_seq=3 Destination Host Unreachable >From 192.168.100.6 icmp_seq=4 Destination Host Unreachable >From 192.168.100.6 icmp_seq=5 Destination Host Unreachable >From 192.168.100.6 icmp_seq=6 Destination Host Unreachable >From 192.168.100.6 icmp_seq=7 Destination Host Unreachable >From 192.168.100.6 icmp_seq=8 Destination Host Unreachable >From 192.168.100.6 icmp_seq=9 Destination Host Unreachable64 bytes from 192.168.101.10: icmp_req=10 ttl=64 time=812 ms 64 bytes from 192.168.101.10: icmp_req=11 ttl=64 time=0.351 ms 64 bytes from 192.168.101.10: icmp_req=12 ttl=64 time=0.293 ms 64 bytes from 192.168.101.10: icmp_req=13 ttl=64 time=0.364 ms ^C root at debian-squeeze:~# ssh 192.168.101.10 root at 192.168.101.10's password: root at new-installed:~# halt The system is going down for system halt NOW!s/0) (Tue Aug 21 18:28:27 2012): root at new-installed:~# Connection to 192.168.101.10 closed by remote host. Connection to 192.168.101.10 closed. root at debian-squeeze:~# root at debian-squeeze:~# root at debian-squeeze:~# root at debian-squeeze:~# root at debian-squeeze:~# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/ixgbevf root at debian-squeeze:~# /usr/local/libvirt/bin/virsh start new-installed Domain new-installed started root at debian-squeeze:~# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub root at debian-squeeze:~# ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.>From 192.168.100.6 icmp_seq=1 Destination Host Unreachable >From 192.168.100.6 icmp_seq=2 Destination Host Unreachable >From 192.168.100.6 icmp_seq=5 Destination Host Unreachable >From 192.168.100.6 icmp_seq=6 Destination Host Unreachable >From 192.168.100.6 icmp_seq=7 Destination Host Unreachable >From 192.168.100.6 icmp_seq=8 Destination Host Unreachable^C N ever came! Please suggest if there is any way out. Relevant kernel logs: Aug 21 18:28:34 debian-squeeze kernel: [ 290.068511] ixgbevf 0000:04:10.0: enabling device (0000 -> 0002) Aug 21 18:28:34 debian-squeeze kernel: [ 290.069216] ixgbevf 0000:04:10.0: enabling bus mastering Aug 21 18:28:34 debian-squeeze kernel: [ 290.082458] ixgbevf 0000:04:10.0: irq 127 for MSI/MSI-X Aug 21 18:28:34 debian-squeeze kernel: [ 290.082482] ixgbevf 0000:04:10.0: irq 128 for MSI/MSI-X Aug 21 18:28:34 debian-squeeze kernel: [ 290.082503] ixgbevf 0000:04:10.0: irq 129 for MSI/MSI-X Aug 21 18:28:34 debian-squeeze kernel: [ 290.141706] ixgbe 0000:04:00.0: eth0: Set MAC msg received from VF 0 Aug 21 18:28:34 debian-squeeze kernel: [ 290.142548] ixgbe 0000:04:00.0: eth0: Set LPE msg received from vf 0 Aug 21 18:28:34 debian-squeeze kernel: [ 290.150487] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready Aug 21 18:28:34 debian-squeeze kernel: [ 290.153637] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready Aug 21 18:28:45 debian-squeeze ntpdate[40135]: step time server 202.190.183.188 offset -0.004582 sec Aug 21 18:28:46 debian-squeeze snmpd[39480]: IfIndex of an interface changed. Aug 21 18:29:39 debian-squeeze kernel: [ 355.486987] pci-stub 0000:04:10.0: claimed by stub Aug 21 18:29:39 debian-squeeze kernel: [ 355.826741] pci-stub 0000:04:10.0: enabling device (0000 -> 0002) Aug 21 18:29:40 debian-squeeze kernel: [ 356.484437] assign device 0:4:10.0 Aug 21 18:29:47 debian-squeeze kernel: [ 363.296117] kvm: 40206: cpu0 unhandled rdmsr: 0xc0010001 Aug 21 18:29:55 debian-squeeze kernel: [ 371.643998] ixgbe 0000:04:00.0: eth0: Set MAC msg received from VF 0 Aug 21 18:29:55 debian-squeeze kernel: [ 371.644661] ixgbe 0000:04:00.0: eth0: Set LPE msg received from vf 0 Aug 21 18:29:55 debian-squeeze kernel: [ 371.663095] pci-stub 0000:04:10.0: irq 127 for MSI/MSI-X Aug 21 18:29:55 debian-squeeze kernel: [ 371.694983] pci-stub 0000:04:10.0: irq 127 for MSI/MSI-X Aug 21 18:29:55 debian-squeeze kernel: [ 371.695009] pci-stub 0000:04:10.0: irq 128 for MSI/MSI-X Aug 21 18:29:55 debian-squeeze kernel: [ 371.758805] pci-stub 0000:04:10.0: irq 127 for MSI/MSI-X Aug 21 18:29:55 debian-squeeze kernel: [ 371.758833] pci-stub 0000:04:10.0: irq 128 for MSI/MSI-X Aug 21 18:29:55 debian-squeeze kernel: [ 371.758856] pci-stub 0000:04:10.0: irq 129 for MSI/MSI-X Regards, Rajesh Kumar Mallah Tel. 91-11-46710500 (Ext:303) Cell +919811255597 Website: http://www.tradeindia.com/ \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | start planting | | | ----- "Rajesh Kumar Mallah" <mallah at tradeindia.com> wrote: | From: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | To: "Alex Jia" <ajia at redhat.com> | Cc: libvirt-users at redhat.com, "vikas pandey" <vikas_pandey at tradeindia.com> | Sent: Tuesday, August 21, 2012 11:00:24 AM | Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage. | | | Dear Alex , | | Thanks Once again for responding. I tried to replicate your steps. in our case readlink always reveals that the pci device is being | handled by the pci-stub driver (as in our case ixgbevf is absent in the host) | | Below is an annotaded sequence of events. | | Check status : # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub | | Content of the nic.xml: | root at debian-squeeze:/usr/local/libvirt/etc/libvirt/qemu# cat /root/nic.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub | | corrent connectivity is OK | # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data. 64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.491 ms 64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.384 ms 64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.310 ms ^C --- 192.168.101.10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.310/0.395/0.491/0.074 ms # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub | | Detach and reattach and check connectivity in running guest. | note: the guest has pci hotplug support , the ethernet interfaces | are configured to allow hotplug and IP is statically assigned. | # virsh detach-device new-installed /root/nic.xml Device detached successfully # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub root at debian-squeeze:/usr/local/libvirt/etc/libvirt/qemu# ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.>From 192.168.100.6 icmp_seq=9 Destination Host Unreachable >From 192.168.100.6 icmp_seq=10 Destination Host Unreachable >From 192.168.100.6 icmp_seq=11 Destination Host Unreachable^C --- 192.168.101.10 ping statistics --- 14 packets transmitted, 0 received, +3 errors, 100% packet loss, time 13096ms pipe 3 # virsh attach-device new-installed /root/nic.xml Device attached successfully # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data. 64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.507 ms 64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.388 ms 64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.314 ms 64 bytes from 192.168.101.10: icmp_req=4 ttl=64 time=0.315 ms 64 bytes from 192.168.101.10: icmp_req=5 ttl=64 time=0.335 ms ^C --- 192.168.101.10 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3998ms rtt min/avg/max/mdev = 0.314/0.371/0.507/0.076 ms # readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver /sys/bus/pci/drivers/pci-stub | Above proves that detach-attach cycle was successful. | # virsh detach-device new-installed /root/nic.xml Device detached successfully | | Please note that guest is being shutdown with nic detached. | | # virsh shutdown new-installed Domain new-installed is being shutdown | Guest being started again : | | # virsh start new-installed Domain new-installed started # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data. 64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.583 ms 64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.402 ms 64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.308 ms 64 bytes from 192.168.101.10: icmp_req=4 ttl=64 time=0.316 ms 64 bytes from 192.168.101.10: icmp_req=5 ttl=64 time=0.318 ms 64 bytes from 192.168.101.10: icmp_req=6 ttl=64 time=0.247 ms ^C --- 192.168.101.10 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 4997ms rtt min/avg/max/mdev = 0.247/0.362/0.583/0.109 ms | Please note on starting the guest the PCI device came up | and connectivity was established. The device was also defined | inside the guest XML. | | | Now shutting down the Guest without detaching. | | # virsh shutdown new-installed Domain new-installed is being shutdown # virsh start new-installed Domain new-installed started | | # ping 192.168.101.10 PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.>From 192.168.100.6 icmp_seq=9 Destination Host Unreachable >From 192.168.100.6 icmp_seq=10 Destination Host Unreachable >From 192.168.100.6 icmp_seq=11 Destination Host Unreachable^C --- 192.168.101.10 ping statistics --- 14 packets transmitted, 0 received, +3 errors, 100% packet loss, time 13095ms pipe 3 CONNECTION IS LOST | | We are using latest libvirtd (0.9.13) , lastest ixgbe/ixgbevf (from sf) , lastest stable kernel (3.5.1) | | Regards, | Rajesh Kumar Mallah | Tel. 91-11-46710500 (Ext:303) | Cell +919811255597 | Website: http://www.tradeindia.com/ | \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | | start planting | | | | | | ----- "Alex Jia" <ajia at redhat.com> wrote: | | From: "Alex Jia" <ajia at redhat.com> | | To: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | | Cc: libvirt-users at redhat.com | | Sent: Tuesday, August 21, 2012 9:34:24 AM | | Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage. | | | | On 08/21/2012 11:22 AM, Rajesh Kumar Mallah wrote: | | | | Dear Alex / List , | | | | Thanks for the response. | | | | The problem is regarding detaching from the guest not host. | | | | I see, if you attach a PCI device into guest with 'managed' mode(managed='yes') then the device is automatically detached from guest and returned to host after guest exiting, for example. | | | | # lspci | grep Eth | | 00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02) | | | | # readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f | | /sys/bus/pci/drivers/e1000e | | | | # virsh start foo | | Domain foo started | | | | # ping -c 3 xx.xx.xx.xx | | PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data. | | 64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=1 ttl=49 time=87.8 ms | | 64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=2 ttl=49 time=85.2 ms | | 64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=3 ttl=49 time=91.8 ms | | --- xx.xx.xx.xx ping statistics --- | | 3 packets transmitted, 3 received, 0% packet loss, time 2093ms | | rtt min/avg/max/mdev = 85.240/88.327/91.881/2.731 ms | | | | # cat hostdev.xml | | <hostdev mode='subsystem' type='pci' managed='yes'> | | <source> | | <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> | | </source> | | </hostdev> | | | | # virsh attach-device foo hostdev.xml | | Device attached successfully | | | | # readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f | | /sys/bus/pci/drivers/pci-stub | | | | # ping -c 3 xx.xx.xx.xx | | connect: Network is unreachable | | | | # virsh destroy foo | | Domain foo destroyed | | | | # readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f | | /sys/bus/pci/drivers/e1000e | | | | # ping -c 3 xx.xx.xx.xx | | PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data. | | 64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=49 time=79.7 ms | | 64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=49 time=83.2 ms | | 64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=49 time=84.5 ms | | | | --- xx.xx.xx.xx ping statistics --- | | 3 packets transmitted, 3 received, 0% packet loss, time 2087ms | | rtt min/avg/max/mdev = 79.714/82.480/84.513/2.040 ms | | | | Notes, need to wait for local network recovery again then check network connectivity. | | | | If your network still is disconnect, it may be a bug IMHO, please show your libvirt | | and qemu-kvm version, or my example is not what you want, please let me know. | | | | Thanks, | | Alex | | | | | | In our case since ixgbevf driver is NOT present in the host hence the virtual | | interfaces never really attach themselves to the host. | | | | Regards, | | Rajesh Kumar Mallah | | Tel. 91-11-46710500 (Ext:303) | | Cell +919811255597 | | Website: http://www.tradeindia.com/ | | \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | | | start planting | | | | | | | | | ----- "Alex Jia" <ajia at redhat.com> wrote: | | | From: "Alex Jia" <ajia at redhat.com> | | | To: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | | | Cc: libvirt-users at redhat.com | | | Sent: Tuesday, August 21, 2012 8:31:21 AM | | | Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage. | | | | | | On 08/20/2012 08:34 PM, Rajesh Kumar Mallah wrote: Dear List , We are using intel 82599EB based SRIOV capable 10GbE cards on the host machine.The VFs are being assigned to the guest in pass through mode and are providing the network connectivity as expected. However when we shutdown the guest the connectivity does not comes back via the VFs on the guest in next boot unless the VFs are detached from the *guest* *before* initiating a shutdown of the guests. Dear Rajesh, | | | | | | Please see the following comments. | | | | | | ( the detach is explicitly done via virsh command /usr/local/libvirt/bin/virsh detach-device guest01 ~/nic.xml nic.xml contained <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> ) The question is anyone else facing similar problem and if so what solution was used? Secondly , is there anyway to automate the detaching of the VFs from the guest when the guest is put in the shutdown sequence ? (or for that matter destroyed). | | | When managed is "yes" for a PCI device, it is automatically detached from the host before being passed on to the guest, and automatically reattached to the host after the guest exits such as shutdown a guest. | | | | | | Regards, | | | Alex | | | | | | Regards, Rajesh Kumar Mallah Tel. 91-11-46710500 (Ext:303) Cell +919811255597 Website: http://www.tradeindia.com/ \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | start planting | | | _______________________________________________ libvirt-users mailing list libvirt-users at redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users | | | | | -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20120821/a6b520f1/attachment.htm>
Rajesh Kumar Mallah
2012-Aug-23 03:41 UTC
[libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
| Dear Alex ,
|
| What logging level is useful for you ?
|
| Dear Rajesh,
|
| Level 1 (Debug) should be enough, but I guess it probably hasn't more
| help for us, you had better to login the guest then check network when you
were
| meeting this issue again, for example, to check if network configuration is
valid and try
| to restart relevant network services to see if the guest network can work
again
| or something like that, In addition, you shut down the guest via running
'halt' in
| the guest,| Does it work for you if you use libvirt relevant interface such as
| 'virsh shutdown guest'
| or 'virsh destroy guest'?
Dear Alex ,
The Guest is a minimal debian squeeze install without any GUI desktop
in it . Below is the relevant /etc/network/interfaces file.
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
auto eth0
iface eth0 inet static
address 192.168.101.11
netmask 255.255.0.0
gateway 192.168.1.1
------------------------- end of interfaces file ----------------
the allow-hotplug on eth0 allow network to be automatically
configured once the attach-device is done. similarly detach-device
is handled gracefully.
Before posting we have done multiple permutations and combinations
of many config parameters and activities. This includes shutting down the
guest via 'halt' and 'destroy|shutdown' from the virsh
interface. It does
not help or makes difference.
Most surprisingly there is no alarming messages that are shown
in any of the logs , be it the syslog of host or syslog of guest,
I have however not seen the libvirtd.log in debug mode which i
shall do in a while.
I would also like to tell that the issues was discussed in the
intel forum but no conclusive solution could be found. The link
is:
http://communities.intel.com/message/164211#164211
I am sorry for the bit late response but i assure my continued
feedback regarding this issue. We are working on a test (spare)
hardware and it is easy for us to test any configuration or patches
The platform is : SuperMicro AMD Opteron Hardware , with IOMMU
and SRIOV enabled , intel 82599EB based dual SFP+ port , linux
kernel 3.5.1 from kernel.org.
Regds
mallah
|
|
| Regards,
| Alex
|
|
|
| Regards,
| Rajesh Kumar Mallah
| Tel. 91-11-46710500 (Ext:303)
| Cell +919811255597
| Website: http://www.tradeindia.com/
| \|/ \|/ \|/ stop printing \|/ \|/ \|/
| | | | start planting | | |
Rajesh Kumar Mallah
2012-Aug-27 08:18 UTC
[libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
Dear Alex , What do you have in the Host ? Does the guests behave as we are expecting them to behave in our setup ? <offtopic> can i get rhel6.2 for trying? </offtopic> Also i have attached the files this time. Regards, Rajesh Kumar Mallah Tel. 91-11-46710500 (Ext:303) Cell +919811255597 Website: http://www.tradeindia.com/ \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | start planting | | | ----- "Alex Jia" <ajia at redhat.com> wrote: | From: "Alex Jia" <ajia at redhat.com> | To: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | Cc: libvirt-users at redhat.com, "vikas pandey" <vikas_pandey at tradeindia.com> | Sent: Monday, August 27, 2012 12:55:14 PM | Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage. | | Hi Rajesh, | I guess you forgot to attach your attachments in here. | As usual, I tried to directly assign a PF or VF of the Intel82576 | to guest on RHEL6.2, my box is Dell Optiplex 755. | | -- | Regards, | Alex | | | ----- Original Message ----- | From: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | To: "Alex Jia" <ajia at redhat.com> | Cc: libvirt-users at redhat.com, "vikas pandey" | <vikas_pandey at tradeindia.com> | Sent: Saturday, August 25, 2012 2:31:20 PM | Subject: Re: [libvirt-users] automatically detaching PCI host devices | from guest for SRIOV usage. | | Dear Alex/List , | | I am attaching a zip file that contains a set of files. | The files that start with case-1 is the case when the PCI | device is detached prior to shutdown. case-2 is otherwise. | | Kindly have a look when you have time and see if there | is any hints. | | BTW can you suggest in what free platform does SRIOV and direct | PCI assignment *really* works so that we could try on that as well. | | Regards, | Rajesh Kumar Mallah | Tel. 91-11-46710500 (Ext:303) | Cell +919811255597 | Website: http://www.tradeindia.com/ | \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | | start planting | | | | | | ----- "Alex Jia" <ajia at redhat.com> wrote: | | | From: "Alex Jia" <ajia at redhat.com> | | To: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | | Cc: libvirt-users at redhat.com, "vikas pandey" | <vikas_pandey at tradeindia.com> | | Sent: Thursday, August 23, 2012 8:21:58 AM | | Subject: Re: [libvirt-users] automatically detaching PCI host | devices from guest for SRIOV usage. | | | | ----- Original Message ----- | | From: "Rajesh Kumar Mallah" <mallah at tradeindia.com> | | To: "Alex Jia" <ajia at redhat.com> | | Cc: libvirt-users at redhat.com, "vikas pandey" | | <vikas_pandey at tradeindia.com> | | Sent: Wednesday, August 22, 2012 7:37:05 PM | | Subject: Re: [libvirt-users] automatically detaching PCI host | devices | | from guest for SRIOV usage. | | | | | | | | | | | Could you attach your libvirtd.log? which's version your | libvirt? | | | in addition, if your guest has a desktop, please login and check | if | | | | | the network configuration exists issues. | | | | | | Thanks, | | | Alex | | | | Dear Alex , | | | | What logging level is useful for you ? | | | | Dear Rajesh, | | | | Level 1 (Debug) should be enough, but I guess it probably hasn't | more | | help for us, | | you had better to login the guest then check network when you were | | meeting this issue | | again, for example, to check if network configuration is valid and | try | | to restart | | relevant network services to see if the guest network can work | again | | or something | | like that, In addition, you shut down the guest via running 'halt' | in | | the guest, | | Does it work for you if you use libvirt relevant interface such as | | 'virsh shutdown guest' | | or 'virsh destroy guest'? | | | | | | Regards, | | Alex | | | | | | | | Regards, | | Rajesh Kumar Mallah | | Tel. 91-11-46710500 (Ext:303) | | Cell +919811255597 | | Website: http://www.tradeindia.com/ | | \|/ \|/ \|/ stop printing \|/ \|/ \|/ | | | | | start planting | | | -------------- next part -------------- A non-text attachment was scrubbed... Name: libvirtlogs.zip Type: application/zip Size: 103024 bytes Desc: not available URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20120827/a83c9066/attachment.zip>
Seemingly Similar Threads
- Re-sizing qcow2 images
- Booting virtual machines automatically
- Failed to bind to uuid (GUID)._msdcs.DOMAIN NT_STATUS_NO_LOGON_SERVERS & IRPC callback failed for DsReplicaSync - NT_STATUS_IO_TIMEOUT
- rstudio doesn't work in LMDE without internet conexion.
- Bug#679533: Traffic forwarding issue between Xen domU/dom0 and ovs