lejeczek
2014-Sep-19 14:35 UTC
[libvirt-users] sr-iov on Intel 82576 and rhel 7 - would not work
hi everybody
a windows kvm guest would not start, process gets killed with:
Out of memory: Kill process 21984 (qemu-kvm) score 44 or
sacrifice child
I really don't know where/what I might be missing, config
seems fine, everything looks ok - I only am not sure, do I
need to first stub a SR-IOV device like regular passthrough?
I'm trying sr-iov, having one NIC left to the host and the
second give to a pool.
Then I have
in a guest:
<interface type='network'>
<mac address='52:54:00:99:df:43'/>
<source network='passthrough-pool'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07' function='0x0'/>
</interface>
and the net:
<network>
<name>passthrough-pool</name>
<uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid>
<forward mode='hostdev' managed='yes'>
<pf dev='enp2s0f1'/>
</forward>
</network>
----
02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
Network Connection (rev 01)
Subsystem: Super Micro Computer Inc Device 10c9
Flags: bus master, fast devsel, latency 0, IRQ 40
Memory at dfce0000 (32-bit, non-prefetchable) [size=128K]
Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K]
I/O ports at a800 [size=32]
Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at dfca0000 [disabled] [size=128K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number
00-25-90-ff-ff-72-bb-e2
Capabilities: [150] Alternative Routing-ID
Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: igb
02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit
Network Connection (rev 01)
Subsystem: Super Micro Computer Inc Device 10c9
Flags: bus master, fast devsel, latency 0, IRQ 41
Memory at dfc20000 (32-bit, non-prefetchable) [size=128K]
Memory at dfc00000 (32-bit, non-prefetchable) [size=128K]
I/O ports at a400 [size=32]
Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at dfbe0000 [disabled] [size=128K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number
00-25-90-ff-ff-72-bb-e2
Capabilities: [150] Alternative Routing-ID
Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: igb
Martin Kletzander
2014-Sep-19 18:42 UTC
Re: [libvirt-users] sr-iov on Intel 82576 and rhel 7 - would not work
On Fri, Sep 19, 2014 at 03:35:07PM +0100, lejeczek wrote:>hi everybody > >a windows kvm guest would not start, process gets killed with: > >Out of memory: Kill process 21984 (qemu-kvm) score 44 or >sacrifice child >Not paying attention to the rest of the mail, I'd just say that kernel had to kill something due to not having enough memory. What are the memory statistics when starting the guest? What's your overcommit configuration?>I really don't know where/what I might be missing, config >seems fine, everything looks ok - I only am not sure, do I >need to first stub a SR-IOV device like regular passthrough? > >I'm trying sr-iov, having one NIC left to the host and the >second give to a pool. >Then I have > >in a guest: > > <interface type='network'> > <mac address='52:54:00:99:df:43'/> > <source network='passthrough-pool'/> > <model type='rtl8139'/> > <address type='pci' domain='0x0000' bus='0x00' >slot='0x07' function='0x0'/> > </interface> > >and the net: > ><network> > <name>passthrough-pool</name> > <uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid> > <forward mode='hostdev' managed='yes'> > <pf dev='enp2s0f1'/> > </forward> ></network> > >---- >02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >Network Connection (rev 01) > Subsystem: Super Micro Computer Inc Device 10c9 > Flags: bus master, fast devsel, latency 0, IRQ 40 > Memory at dfce0000 (32-bit, non-prefetchable) [size=128K] > Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K] > I/O ports at a800 [size=32] > Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K] > Expansion ROM at dfca0000 [disabled] [size=128K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=10 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number >00-25-90-ff-ff-72-bb-e2 > Capabilities: [150] Alternative Routing-ID >Interpretation (ARI) > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > Kernel driver in use: igb >02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit >Network Connection (rev 01) > Subsystem: Super Micro Computer Inc Device 10c9 > Flags: bus master, fast devsel, latency 0, IRQ 41 > Memory at dfc20000 (32-bit, non-prefetchable) [size=128K] > Memory at dfc00000 (32-bit, non-prefetchable) [size=128K] > I/O ports at a400 [size=32] > Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K] > Expansion ROM at dfbe0000 [disabled] [size=128K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=10 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number >00-25-90-ff-ff-72-bb-e2 > Capabilities: [150] Alternative Routing-ID >Interpretation (ARI) > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > Kernel driver in use: igb > >_______________________________________________ >libvirt-users mailing list >libvirt-users@redhat.com >https://www.redhat.com/mailman/listinfo/libvirt-users
Laine Stump
2014-Sep-20 21:59 UTC
Re: [libvirt-users] sr-iov on Intel 82576 and rhel 7 - would not work
On 09/19/2014 10:35 AM, lejeczek wrote:> hi everybody > > a windows kvm guest would not start, process gets killed with: > > Out of memory: Kill process 21984 (qemu-kvm) score 44 or sacrifice child > > I really don't know where/what I might be missing, config seems fine, > everything looks ok - I only am not sure, do I need to first stub a > SR-IOV device like regular passthrough? > > I'm trying sr-iov, having one NIC left to the host and the second give > to a pool.I'm not aware of any way that error message might be related to device passthrough. Your config looks fine, as long as enp2s0f1 is the SRIOV card's PF (Physical Function) and the output of ip link show dev enp2s0f1 shows a list of VFs. In the case that you have no VFs created for that device, yoou can make sure this happens by adding the following to /etc/modprobe.d/local.conf: options igb max_vfs=7 BTW, unless you're concerned about the amount of bandwidth available on a single port of your switch, you can use the PF for host networking, and still use the VFs of that same PF for passthrough assignment to guests.> Then I have > > in a guest: > > <interface type='network'> > <mac address='52:54:00:99:df:43'/> > <source network='passthrough-pool'/> > <model type='rtl8139'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07' > function='0x0'/> > </interface> > > and the net: > > <network> > <name>passthrough-pool</name> > <uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid> > <forward mode='hostdev' managed='yes'> > <pf dev='enp2s0f1'/> > </forward> > </network> > > ---- > 02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > Subsystem: Super Micro Computer Inc Device 10c9 > Flags: bus master, fast devsel, latency 0, IRQ 40 > Memory at dfce0000 (32-bit, non-prefetchable) [size=128K] > Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K] > I/O ports at a800 [size=32] > Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K] > Expansion ROM at dfca0000 [disabled] [size=128K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=10 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 > Capabilities: [150] Alternative Routing-ID Interpretation (ARI) > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > Kernel driver in use: igb > 02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > Subsystem: Super Micro Computer Inc Device 10c9 > Flags: bus master, fast devsel, latency 0, IRQ 41 > Memory at dfc20000 (32-bit, non-prefetchable) [size=128K] > Memory at dfc00000 (32-bit, non-prefetchable) [size=128K] > I/O ports at a400 [size=32] > Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K] > Expansion ROM at dfbe0000 [disabled] [size=128K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=10 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 > Capabilities: [150] Alternative Routing-ID Interpretation (ARI) > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > Kernel driver in use: igb > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users > >
lejeczek
2014-Sep-23 08:40 UTC
Re: [libvirt-users] sr-iov on Intel 82576 and rhel 7 - would not work
yes, I've tried both PFs' VFs - no difference if I only take SR-IOV out and use virtio that guest works perfectly fine. Hardware is a Supermicro H8DG6/H8DGi + Opterons 6366 HE and an Intel 82576 Gigabit Network Connection (rev 01). It seems very weird, I think if I am not missing anything config-wise then either the hardware platform as a whole or/and libvirt-qemu-kernel is buggy. any suggestion on how to troubleshoot it? many thanks ps. would you know how to make sure abrtd auto-reports to bugzilla in a non-gui target? On 20/09/14 22:59, Laine Stump wrote:> On 09/19/2014 10:35 AM, lejeczek wrote: >> hi everybody >> >> a windows kvm guest would not start, process gets killed with: >> >> Out of memory: Kill process 21984 (qemu-kvm) score 44 or sacrifice child >> >> I really don't know where/what I might be missing, config seems fine, >> everything looks ok - I only am not sure, do I need to first stub a >> SR-IOV device like regular passthrough? >> >> I'm trying sr-iov, having one NIC left to the host and the second give >> to a pool. > I'm not aware of any way that error message might be related to device > passthrough. > > Your config looks fine, as long as enp2s0f1 is the SRIOV card's PF > (Physical Function) and the output of ip link show dev enp2s0f1 shows a > list of VFs. In the case that you have no VFs created for that device, > yoou can make sure this happens by adding the following to > /etc/modprobe.d/local.conf: > > options igb max_vfs=7 > > BTW, unless you're concerned about the amount of bandwidth available on > a single port of your switch, you can use the PF for host networking, > and still use the VFs of that same PF for passthrough assignment to guests. > >> Then I have >> >> in a guest: >> >> <interface type='network'> >> <mac address='52:54:00:99:df:43'/> >> <source network='passthrough-pool'/> >> <model type='rtl8139'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' >> function='0x0'/> >> </interface> >> >> and the net: >> >> <network> >> <name>passthrough-pool</name> >> <uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid> >> <forward mode='hostdev' managed='yes'> >> <pf dev='enp2s0f1'/> >> </forward> >> </network> >> >> ---- >> 02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network >> Connection (rev 01) >> Subsystem: Super Micro Computer Inc Device 10c9 >> Flags: bus master, fast devsel, latency 0, IRQ 40 >> Memory at dfce0000 (32-bit, non-prefetchable) [size=128K] >> Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K] >> I/O ports at a800 [size=32] >> Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K] >> Expansion ROM at dfca0000 [disabled] [size=128K] >> Capabilities: [40] Power Management version 3 >> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ >> Capabilities: [70] MSI-X: Enable+ Count=10 Masked- >> Capabilities: [a0] Express Endpoint, MSI 00 >> Capabilities: [100] Advanced Error Reporting >> Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 >> Capabilities: [150] Alternative Routing-ID Interpretation (ARI) >> Capabilities: [160] Single Root I/O Virtualization (SR-IOV) >> Kernel driver in use: igb >> 02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network >> Connection (rev 01) >> Subsystem: Super Micro Computer Inc Device 10c9 >> Flags: bus master, fast devsel, latency 0, IRQ 41 >> Memory at dfc20000 (32-bit, non-prefetchable) [size=128K] >> Memory at dfc00000 (32-bit, non-prefetchable) [size=128K] >> I/O ports at a400 [size=32] >> Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K] >> Expansion ROM at dfbe0000 [disabled] [size=128K] >> Capabilities: [40] Power Management version 3 >> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ >> Capabilities: [70] MSI-X: Enable+ Count=10 Masked- >> Capabilities: [a0] Express Endpoint, MSI 00 >> Capabilities: [100] Advanced Error Reporting >> Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 >> Capabilities: [150] Alternative Routing-ID Interpretation (ARI) >> Capabilities: [160] Single Root I/O Virtualization (SR-IOV) >> Kernel driver in use: igb >> >> _______________________________________________ >> libvirt-users mailing list >> libvirt-users@redhat.com >> https://www.redhat.com/mailman/listinfo/libvirt-users >> >>