I havent started the process of narrowing down where the issue might be
or even if there is an issue, but has anyone else seen a situation where
RELENG_8 i386 nfs client write speed have all of a sudden nose dived ?
Doing a simple cat /dev/zero > /mnt/junk used to yield 500Mb and above
easily. Now, its down to 25Mb. A RELENG_9 client is able to write to
the same NFS server at 900Mb. I tested another i386 RELENG_8 server
with a similar build date and it gets the same slow speed, where as I am
pretty sure the same test gave around 400Mb with an older kernel. Both
problem boxes are i386 at r267114 (~ June 5th). Both have em nics, but
they are different em NICs. But, I have some other boxes from around the
same time that are just as fast, but with different em nics / chipsets
as well so I am not sure if its related to the em driver or not.
The problematic em nics are
em1 at pci0:4:0:0: class=0x020000 card=0x35788086 chip=0x10d38086 rev=0x00
hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
speed 2.5(2.5) ASPM disabled(L0s/L1)
cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
and
em1 at pci0:6:5:0: class=0x020000 card=0x348d8086 chip=0x10768086 rev=0x05
hdr=0x00
vendor = 'Intel Corporation'
device = 'Gigabit Ethernet Controller (82541EI)'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xea020000, size 131072,
enabled
bar [14] = type Memory, range 32, base 0xea000000, size 131072,
enabled
bar [18] = type I/O Port, range 32, base 0x1100, size 64, enabled
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction
And ones that are working as expected
em1 at pci0:14:0:0: class=0x020000 card=0x109a15d9 chip=0x109a8086
rev=0x00 hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel PRO/1000 PL Network Adaptor (82573L)'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xe8b00000, size 131072,
enabled
bar [18] = type I/O Port, range 32, base 0x6000, size 32, enabled
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
speed 2.5(2.5)
ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected
ecap 0003[140] = Serial 1 003048ffff9c59f1
and
em1 at pci0:14:0:0: class=0x020000 card=0x109a15d9 chip=0x109a8086
rev=0x00 hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel PRO/1000 PL Network Adaptor (82573L)'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xe1100000, size 131072,
enabled
bar [18] = type I/O Port, range 32, base 0x6000, size 32, enabled
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
speed 2.5(2.5)
ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected
ecap 0003[140] = Serial 1 003048ffff905219
It might not be an issue with the em driver. Not sure at this point, but
wondering if anyone has noticed any other slow downs with NFS on RELENG_8 ?
---Mike
--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
On 6/23/2014 3:50 PM, Mike Tancsa wrote:> I havent started the process of narrowing down where the issue might be > or even if there is an issue, but has anyone else seen a situation where > RELENG_8 i386 nfs client write speed have all of a sudden nose dived ? > Doing a simple cat /dev/zero > /mnt/junk used to yield 500Mb and above > easily. Now, its down to 25Mb. A RELENG_9 client is able to write toReads are plenty fast. On the initial "problem" server where the writes are all of a sudden super slow, the read rate on a uncached file on the server comes in at 1592524800 bytes transferred in 16.746340 secs (95096887 bytes/sec) or ~ 780Mb/s ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike at sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Mike Tancsa wrote:> I havent started the process of narrowing down where the issue might > be > or even if there is an issue, but has anyone else seen a situation > where > RELENG_8 i386 nfs client write speed have all of a sudden nose dived > ? > Doing a simple cat /dev/zero > /mnt/junk used to yield 500Mb and > above > easily. Now, its down to 25Mb. A RELENG_9 client is able to write to > the same NFS server at 900Mb. I tested another i386 RELENG_8 server > with a similar build date and it gets the same slow speed, where as I > am > pretty sure the same test gave around 400Mb with an older kernel. > Both > problem boxes are i386 at r267114 (~ June 5th). Both have em nics, > but > they are different em NICs. But, I have some other boxes from around > the > same time that are just as fast, but with different em nics / > chipsets > as well so I am not sure if its related to the em driver or not. >I suggest you try disabling TSO on the nics (or setting wsize=32786 on all mounts). Usually such slowdowns are related to the underlying network fabric and there are known TSO issues for nics/drivers that only handle 32 transmit buffers when 64K writes are being used. rick> The problematic em nics are > > em1 at pci0:4:0:0: class=0x020000 card=0x35788086 chip=0x10d38086 > rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > class = network > subclass = ethernet > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link > x1(x1) > speed 2.5(2.5) ASPM disabled(L0s/L1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > and > > em1 at pci0:6:5:0: class=0x020000 card=0x348d8086 chip=0x10768086 > rev=0x05 > hdr=0x00 > vendor = 'Intel Corporation' > device = 'Gigabit Ethernet Controller (82541EI)' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xea020000, size > 131072, > enabled > bar [14] = type Memory, range 32, base 0xea000000, size > 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x1100, size 64, > enabled > cap 01[dc] = powerspec 2 supports D0 D3 current D0 > cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction > > > And ones that are working as expected > > em1 at pci0:14:0:0: class=0x020000 card=0x109a15d9 > chip=0x109a8086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel PRO/1000 PL Network Adaptor (82573L)' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xe8b00000, size > 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x6000, size 32, > enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 > message > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link > x1(x1) > speed 2.5(2.5) > ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected > ecap 0003[140] = Serial 1 003048ffff9c59f1 > > and > > em1 at pci0:14:0:0: class=0x020000 card=0x109a15d9 > chip=0x109a8086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel PRO/1000 PL Network Adaptor (82573L)' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xe1100000, size > 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x6000, size 32, > enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 > message > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link > x1(x1) > speed 2.5(2.5) > ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected > ecap 0003[140] = Serial 1 003048ffff905219 > > It might not be an issue with the em driver. Not sure at this point, > but > wondering if anyone has noticed any other slow downs with NFS on > RELENG_8 ? > > > ---Mike > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike at sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe at freebsd.org" >