Ulrich Spörlein
2010-Nov-06 09:37 UTC
Abysmal re(4) performance under 8.1-STABLE (mid-August)
Hello Pyun,
On this new server, I cannot get more than ~280kByte/s up/downstream out of
re(4) without any tweaking.
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
ether 00:21:85:63:74:34
inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1
inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet autoselect (100baseTX <half-duplex>)
status: active
re0@pci0:2:0:0: class=0x020000 card=0x368c1462 chip=0x816810ec rev=0x01 hdr=0x00
vendor = 'Realtek Semiconductor'
device = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)'
class = network
subclass = ethernet
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port
0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2
re0: Using 1 MSI messages
re0: Chip rev. 0x38000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
re0: Ethernet address: 00:21:85:63:74:34
re0: [FILTER]
It's interesting to note, that re0 only negotiates half-duplex, where
linux will negotiate full-duplex (and gets ~10MByte/s as expected).
Next, I disabled almost all options, except that I cannot disable
VLAN_MTU, VLAN_HWCSUM. I also forced the adapter into full-duplex.
# ifconfig re0 -vlanmtu
# ifconfig re0 -vlanhwcsum
ifconfig: -vlanhwcsum: bad value
# ifconfig re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=88<VLAN_MTU,VLAN_HWCSUM>
ether 00:21:85:63:74:34
inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1
inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet 100baseTX <full-duplex> (100baseTX
<half-duplex>)
status: active
If I then immediately start the test-download, I get a ~2MByte/s spike,
which quickly returns to something around 250kByte/s.
Booting with
hw.pci.enable_msix=0
hw.pci.enable_msi=0
I can get almost up to 400kByte/s, but this may be coincidence.
So this is usually as far as it gets:
re0 in 190.504 KB/s 246.136 KB/s 66.709 MB
out 10.290 KB/s 12.985 KB/s 6.076 MB
But then I ran tcpdump in another session, and it looks like the ssh traffic on
the upstream of the interface will make the downloads running in another window
go faster:
re0 in 805.961 KB/s 1.577 MB/s 116.523 MB
out 222.940 KB/s 502.045 KB/s 19.267 MB
Any ideas?
Uli
Ulrich Spörlein
2010-Nov-06 12:09 UTC
Abysmal re(4) performance under 8.1-STABLE (mid-August)
On Sat, 06.11.2010 at 10:37:00 +0100, Ulrich Sp?rlein wrote:> Hello Pyun, > > On this new server, I cannot get more than ~280kByte/s up/downstream out of > re(4) without any tweaking. > > re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC> > ether 00:21:85:63:74:34 > inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1 > inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > media: Ethernet autoselect (100baseTX <half-duplex>) > status: active > > re0@pci0:2:0:0: class=0x020000 card=0x368c1462 chip=0x816810ec rev=0x01 hdr=0x00 > vendor = 'Realtek Semiconductor' > device = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)' > class = network > subclass = ethernetOne more datapoint, I know that this (newer) revision re0@pci0:6:0:0: class=0x020000 card=0x75221462 chip=0x816810ec rev=0x02 hdr=0x00 is working just fine in another system/board.> re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 > re0: Using 1 MSI messages > re0: Chip rev. 0x38000000 > re0: MAC rev. 0x00000000 > miibus0: <MII bus> on re0 > rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0 > rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto > re0: Ethernet address: 00:21:85:63:74:34 > re0: [FILTER] > > > It's interesting to note, that re0 only negotiates half-duplex, where > linux will negotiate full-duplex (and gets ~10MByte/s as expected). > > Next, I disabled almost all options, except that I cannot disable > VLAN_MTU, VLAN_HWCSUM. I also forced the adapter into full-duplex. > > # ifconfig re0 -vlanmtu > # ifconfig re0 -vlanhwcsum > ifconfig: -vlanhwcsum: bad value > # ifconfig re0 > re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=88<VLAN_MTU,VLAN_HWCSUM> > ether 00:21:85:63:74:34 > inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1 > inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>) > status: active > > If I then immediately start the test-download, I get a ~2MByte/s spike, > which quickly returns to something around 250kByte/s. > > Booting with > hw.pci.enable_msix=0 > hw.pci.enable_msi=0 > > I can get almost up to 400kByte/s, but this may be coincidence. > > So this is usually as far as it gets: > > re0 in 190.504 KB/s 246.136 KB/s 66.709 MB > out 10.290 KB/s 12.985 KB/s 6.076 MB > > But then I ran tcpdump in another session, and it looks like the ssh traffic on > the upstream of the interface will make the downloads running in another window > go faster: > > re0 in 805.961 KB/s 1.577 MB/s 116.523 MB > out 222.940 KB/s 502.045 KB/s 19.267 MB > > Any ideas? > > Uli > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Rick Macklem
2010-Nov-06 12:21 UTC
Abysmal re(4) performance under 8.1-STABLE (mid-August)
> Hello Pyun, > > On this new server, I cannot get more than ~280kByte/s up/downstream >I have been working on a similar problem with Pyun's help. Not yet resolved and no idea if it is the same problem, but see below...> > Any ideas?I have been chasing a similar problem w.r.t. re(4) { slow NFS reads on -current }. You could try building a kernel with "options DEVICE_POLLING" and see if that helps. If that makes it work faster, then it might be the same issue. rick
Stefan Bethke
2010-Nov-06 12:25 UTC
Abysmal re(4) performance under 8.1-STABLE (mid-August)
Am 06.11.2010 um 10:37 schrieb Ulrich Sp?rlein:> On this new server, I cannot get more than ~280kByte/s up/downstream out of > re(4) without any tweaking. > > re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC> > ether 00:21:85:63:74:34 > inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1 > inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > media: Ethernet autoselect (100baseTX <half-duplex>) > status: activeAOL: re0@pci0:1:0:0: class=0x020000 card=0x82c61043 chip=0x816810ec rev=0x02 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)' class = network subclass = ethernet re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfdfff000-0xfdffffff,0xfdfe0000-0xfdfeffff irq 18 at device 0.0 on pci1 re0: Using 1 MSI messages re0: Chip rev. 0x3c000000 re0: MAC rev. 0x00400000 miibus0: <MII bus> on re0 rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto re0: Ethernet address: 00:26:18:d5:2c:23 re0: [FILTER] I believe that it was working properly some months ago, but reading Rick's thread over on -current I checked and transfer over NFS seems to be limited to a couple hundred KB as well. Stefan -- Stefan Bethke <stb@lassitu.de> Fon +49 151 14070811
Pyun YongHyeon
2010-Nov-07 06:20 UTC
Abysmal re(4) performance under 8.1-STABLE (mid-August)
On Sat, Nov 6, 2010 at 2:37 AM, Ulrich Sp?rlein <uqs@spoerlein.net> wrote:> Hello Pyun, > > On this new server, I cannot get more than ~280kByte/s up/downstream out of > re(4) without any tweaking. > > re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > ? ? ? ?options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC> > ? ? ? ?ether 00:21:85:63:74:34 > ? ? ? ?inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1 > ? ? ? ?inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191 > ? ? ? ?nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > ? ? ? ?media: Ethernet autoselect (100baseTX <half-duplex>) > ? ? ? ?status: active >It seems the link was resolved to half-duplex. Does link partner also agree on the resolved speed/duplex?> re0@pci0:2:0:0: class=0x020000 card=0x368c1462 chip=0x816810ec rev=0x01 hdr=0x00 > ? ?vendor ? ? = 'Realtek Semiconductor' > ? ?device ? ? = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)' > ? ?class ? ? ?= network > ? ?subclass ? = ethernet > > re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 > re0: Using 1 MSI messages > re0: Chip rev. 0x38000000 > re0: MAC rev. 0x00000000 > miibus0: <MII bus> on re0 > rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0 > rgephy0: ?10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto > re0: Ethernet address: 00:21:85:63:74:34 > re0: [FILTER] > > > It's interesting to note, that re0 only negotiates half-duplex, where > linux will negotiate full-duplex (and gets ~10MByte/s as expected). > > Next, I disabled almost all options, except that I cannot disable > VLAN_MTU, VLAN_HWCSUM. I also forced the adapter into full-duplex. > > # ifconfig re0 -vlanmtu > # ifconfig re0 -vlanhwcsum > ifconfig: -vlanhwcsum: bad valueI'm sure this has nothing to do that this issue. If you want to disable checksum offloading of VLAN interface, use vlan interface instead of parent interface of the VLAN interface(i.e. ifconfig vlan0 -txcsum -rxcsum). And you can't disable VLAN_MTU on re(4). There is no reason to disable supporting VLAN oversized frames.> # ifconfig re0 > re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > ? ? ? ?options=88<VLAN_MTU,VLAN_HWCSUM> > ? ? ? ?ether 00:21:85:63:74:34 > ? ? ? ?inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1 > ? ? ? ?inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191 > ? ? ? ?nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > ? ? ? ?media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>) > ? ? ? ?status: active >This time, it seems you used forced media configuration instead of auto. It still shows duplex mismatch so it's normal to see poor performance. What makes me wonder is why you have duplex mismatch? Did you use forced media configuration on link partner? What happens when you use different switch?> If I then immediately start the test-download, I get a ~2MByte/s spike, > which quickly returns to something around 250kByte/s. > > Booting with > hw.pci.enable_msix=0 > hw.pci.enable_msi=0 > > I can get almost up to 400kByte/s, but this may be coincidence. > > So this is usually as far as it gets: > > re0 ?in ? ?190.504 KB/s ? ? ? ?246.136 KB/s ? ? ? ? ? 66.709 MB > ? ? out ? ?10.290 KB/s ? ? ? ? 12.985 KB/s ? ? ? ? ? ?6.076 MB > > But then I ran tcpdump in another session, and it looks like the ssh traffic on > the upstream of the interface will make the downloads running in another window > go faster: > > re0 ?in ? ?805.961 KB/s ? ? ? ? ?1.577 MB/s ? ? ? ? ?116.523 MB > ? ? out ? 222.940 KB/s ? ? ? ?502.045 KB/s ? ? ? ? ? 19.267 MB > > Any ideas? > > Uli >