Dmitriy Novikov
2010-Apr-28 13:08 UTC
[Xen-devel] Conntrack checksumming problem with pv_ops dom0
Hello.
I have 2.6.32.11 pv_ops dom0 kernel, xen 3.4.3.
dom0 has 2 x Broadcom NICs:
02:05.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 10)
02:05.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 10)
On second interface 4,6 tagged vlan configured and connected to
corresponding bridges:
br4 8000.0030485a308f no eth1.4
vif1.1
br6 8000.0030485a308f no eth1.6
vif1.0
domU configured with 2 interfeces (WAN,LAN) which connected to
corresponding bridges:
vif =
[''mac=00:16:3e:00:06:11,bridge=br6'',''mac=00:16:3e:00:04:11,bridge=br4'']
domU used for OpenVPN. OpenVPN runs over tun1 interface. OpenVPN
traffic comes from eth0
and forwarded to local network via eth1 interface. All checksumming
disabled on xen-netfront interfaces:
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
All checksumming disabled on real and vif/dom0 interfaces:
# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
# ethtool -k vif1.1
Offload parameters for vif1.1:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
# ethtool -k vif1.0
Offload parameters for vif1.0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
Broadcom offloading firmware removed from /lib/firmware:
mv /lib/firmware/2.6.32.11-xen-dom0/tigon ~/fw_backup
Apr 27 18:26:09 xenon kernel: [ 0.562660] tg3.c:v3.102 (September 1, 2009)
Apr 27 18:26:09 xenon kernel: [ 0.562800] xen: registering gsi 26
triggering 0 polarity 1
Apr 27 18:26:09 xenon kernel: [ 0.562817] alloc irq_desc for 26 on node 0
Apr 27 18:26:09 xenon kernel: [ 0.562821] alloc kstat_irqs on node 0
Apr 27 18:26:09 xenon kernel: [ 0.562831] xen: --> irq=26
Apr 27 18:26:09 xenon kernel: [ 0.562841] tg3 0000:02:05.0: PCI INT
A -> GSI 26 (level, low) -> IRQ 26
...
Apr 27 18:26:09 xenon kernel: [ 0.607550] eth0: Tigon3
[partno(BCM95704A6) rev 2100] (PCIX:100MHz:64-bit) MAC address
00:30:48:5a:30:8e
Apr 27 18:26:09 xenon kernel: [ 0.607708] eth0: attached PHY is
5704 (10/100/1000Base-T Ethernet) (WireSpeed[1])
Apr 27 18:26:09 xenon kernel: [ 0.607856] eth0: RXcsums[1]
LinkChgREG[0] MIirq[0] ASF[1] TSOcap[0] <---------------RXcsums[1]
???
Apr 27 18:26:09 xenon kernel: [ 0.607965] eth0:
dma_rwctrl[769f4000] dma_mask[64-bit]
Apr 27 18:26:09 xenon kernel: [ 0.608111] xen: registering gsi 19
triggering 0 polarity 1
Apr 27 18:26:09 xenon kernel: [ 0.608128] alloc irq_desc for 19 on node 0
Apr 27 18:26:09 xenon kernel: [ 0.608132] alloc kstat_irqs on node 0
Apr 27 18:26:09 xenon kernel: [ 0.608142] xen: --> irq=19
...
Apr 27 18:26:09 xenon kernel: [ 0.615909] tg3 0000:02:05.1: PCI INT
B -> GSI 27 (level, low) -> IRQ 27
Apr 27 18:26:09 xenon kernel: [ 0.636546] eth1: Tigon3
[partno(BCM95704A6) rev 2100] (PCIX:100MHz:64-bit) MAC address
00:30:48:5a:30:8f
Apr 27 18:26:09 xenon kernel: [ 0.636704] eth1: attached PHY is
5704 (10/100/1000Base-T Ethernet) (WireSpeed[1])
Apr 27 18:26:09 xenon kernel: [ 0.636851] eth1: RXcsums[1]
LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
Apr 27 18:26:09 xenon kernel: [ 0.636961] eth1:
dma_rwctrl[769f4000] dma_mask[64-bit]
...
Apr 27 18:26:09 xenon kernel: [ 7.340398] tg3 0000:02:05.1:
firmware: requesting tigon/tg3_tso.bin
Apr 27 18:26:09 xenon kernel: [ 7.369999] eth1: Failed to load
firmware "tigon/tg3_tso.bin"
Apr 27 18:26:09 xenon kernel: [ 7.370123] eth1: TSO capability disabled.
Before modprobing nf_conntrack_ipv4 forwarding works fine:
# tcpdump -ni tun1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes
15:13:19.525889 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 17, length 64
15:13:19.540332 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 17, length 64
15:13:20.526379 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 18, length 64
15:13:20.540641 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 18, length 64
# tcpdump -ni eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
15:13:41.550283 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 39, length 64
15:13:41.563788 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 39, length 64
15:13:42.551962 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 40, length 64
15:13:42.565110 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 40, length 64
# tcpdump -ni vif1.1 icmp
tcpdump: WARNING: vif1.1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes
15:14:23.823167 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 81, length 64
15:14:23.836664 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 81, length 64
15:14:24.823795 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 82, length 64
15:14:24.836965 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 82, length 64
But after modprobing nf_conntrack_ipv4 traffic destinated to local
network disappears on vif1.1 while traffic is visible on domU''s lan
interface.
A lot of "Attempting to checksum a non-TCP/UDP packet, dropping a
protocol 1 packet" messages seen on dom0.
# tcpdump -ni eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
15:15:25.603809 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 143, length 64
15:15:25.618697 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 143, length 64
15:15:26.610258 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 144, length 64
15:15:26.626101 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 144, length 64
# tcpdump -ni tun1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes
15:15:41.680256 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 159, length 64
15:15:41.694291 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 159, length 64
15:15:42.681677 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 160, length 64
15:15:42.697395 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 160, length 64
# tcpdump -ni vif1.1 icmp
tcpdump: WARNING: vif1.1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes
15:15:58.036979 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 175, length 64
15:15:59.043942 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 176, length 64
15:16:00.052912 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 177, length 64
15:16:01.053850 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 178, length 64
BUT when I set ''sysctl -w
net.netfilter.nf_conntrack_checksum=0''
traffic starts to run again.
Any ideas?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Dmitry Novikov
2010-May-11 07:42 UTC
Re: [Xen-devel] Conntrack checksumming problem with pv_ops dom0
2010/4/28 Dmitriy Novikov <dimetrios@gmail.com>:> Hello. > I have 2.6.32.11 pv_ops dom0 kernel, xen 3.4.3. > > dom0 has 2 x Broadcom NICs: > 02:05.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) > 02:05.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)I had tested on other server with two Intel 82574L nics and have same problem. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-May-22 11:35 UTC
Re: [Xen-devel] Conntrack checksumming problem with pv_ops dom0
On Tue, May 11, 2010 at 10:42:52AM +0300, Dmitry Novikov wrote:> 2010/4/28 Dmitriy Novikov <dimetrios@gmail.com>: > > Hello. > > I have 2.6.32.11 pv_ops dom0 kernel, xen 3.4.3. > > > > dom0 has 2 x Broadcom NICs: > > 02:05.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) > > 02:05.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) > > I had tested on other server with two Intel 82574L nics and have same problem. >The checksumming problems should be fixed now in the latest xen/stable-2.6.32.x tree. It was a bug in netback driver. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel