> On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote: > > Daniel Braniss wrote: >> >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote: >>> >>> Daniel Braniss wrote: >>>> >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron at gmail.com> >>>>> wrote: >>>>> >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD >>>>> 10.1. Before 10.1 it was less. >>>>> >>>> >>>> this is NOT iperf/3 where i do get close to wire speed, >>>> it?s NFS writes, i.e., almost real work :-) >>>> >>>>> I used to tweak the card settings, but now it's just stock. You may want >>>>> to >>>>> check your settings, the Mellanox may just have better defaults for your >>>>> switch. >>>>> >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be >>> copying >>> every transmitted mbuf chain to a new chain of mbuf clusters via. >>> m_defrag() when >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the >>> 82599 chip >>> these days?) >>> >> >> hi Rick >> >> how can i check the chip? >> > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO helped, > I'll bet you don't have a 82598.) > >>> This has been fixed in the driver very recently, but those fixes won't be >>> in 10.1. >>> >>> rick >>> ps: If you could test with 10.2, it would be interesting to see how the ix >>> does with >>> the current driver fixes in it? >> >> I new TSO was involved! >> ok, firstly, it?s 10.2 stable. >> with TSO enabled, ix is bad, around 64MGB/s. >> disabling TSO it?s better, around 130 >> > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at around > line#2500? > /* TSO parameters */ > 2572 ifp->if_hw_tsomax = 65518; > 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; > 2574 ifp->if_hw_tsomaxsegsize = 2048; > > They are in stable/10. I didn't look at releng/10.2. (And if they're in a #ifdef > for FreeBSD11, take the #ifdef away.) > If they are there and not ifdef'd, I can't explain why disabling TSO would help. > Once TSO is fixed so that it handles the 64K transmit segments without copying all > the mbufs, I suspect you might get better perf. with it enabled? >this is 10.2 : they are on lines 2509-2511 and I don?t see any #ifdefs around it. the plot thickens :-) danny> Good luck with it, rick > >> still, mlxen0 is about 250! with and without TSO >> >> >>> >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru >>>>> <mailto:slw at zxy.spb.ru>> wrote: >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: >>>>> >>>>>> hi, >>>>>> I have a host (Dell R730) with both cards, connected to an HP8200 >>>>>> switch at 10Gb. >>>>>> when writing to the same storage (netapp) this is what I get: >>>>>> ix0: ~130MGB/s >>>>>> mlxen0 ~330MGB/s >>>>>> this is via nfs/tcpv3 >>>>>> >>>>>> I can get similar (bad) performance with the mellanox if I increase >>>>>> the file size >>>>>> to 512MGB. >>>>> >>>>> Look like mellanox have internal beffer for caching and do ACK >>>>> acclerating. >>>>> >>>>>> so at face value, it seems the mlxen does a better use of resources >>>>>> than the intel. >>>>>> Any ideas how to improve ix/intel's performance? >>>>> >>>>> Are you sure about netapp performance? >>>>> _______________________________________________ >>>>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net> >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org >>>>> <mailto:freebsd-net-unsubscribe at freebsd.org>" >>>>> >>>> >>>> _______________________________________________ >>>> freebsd-stable at freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" >> >> _______________________________________________ >> freebsd-stable at freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
Daniel Braniss wrote:> > > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote: > > > > Daniel Braniss wrote: > >> > >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote: > >>> > >>> Daniel Braniss wrote: > >>>> > >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron > >>>>> <csforgeron at gmail.com> > >>>>> wrote: > >>>>> > >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD > >>>>> 10.1. Before 10.1 it was less. > >>>>> > >>>> > >>>> this is NOT iperf/3 where i do get close to wire speed, > >>>> it?s NFS writes, i.e., almost real work :-) > >>>> > >>>>> I used to tweak the card settings, but now it's just stock. You may > >>>>> want > >>>>> to > >>>>> check your settings, the Mellanox may just have better defaults for > >>>>> your > >>>>> switch. > >>>>> > >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be > >>> copying > >>> every transmitted mbuf chain to a new chain of mbuf clusters via. > >>> m_defrag() when > >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the > >>> 82599 chip > >>> these days?) > >>> > >> > >> hi Rick > >> > >> how can i check the chip? > >> > > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO > > helped, > > I'll bet you don't have a 82598.) > > > >>> This has been fixed in the driver very recently, but those fixes won't be > >>> in 10.1. > >>> > >>> rick > >>> ps: If you could test with 10.2, it would be interesting to see how the > >>> ix > >>> does with > >>> the current driver fixes in it? > >> > >> I new TSO was involved! > >> ok, firstly, it?s 10.2 stable. > >> with TSO enabled, ix is bad, around 64MGB/s. > >> disabling TSO it?s better, around 130 > >> > > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at > > around > > line#2500? > > /* TSO parameters */ > > 2572 ifp->if_hw_tsomax = 65518; > > 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; > > 2574 ifp->if_hw_tsomaxsegsize = 2048; > > > > They are in stable/10. I didn't look at releng/10.2. (And if they're in a > > #ifdef > > for FreeBSD11, take the #ifdef away.) > > If they are there and not ifdef'd, I can't explain why disabling TSO would > > help. > > Once TSO is fixed so that it handles the 64K transmit segments without > > copying all > > the mbufs, I suspect you might get better perf. with it enabled? > > > > this is 10.2 : > they are on lines 2509-2511 and I don?t see any #ifdefs around it. > > the plot thickens :-) >If this is just a test machine, maybe you could test with these lines (at about #880) in sys/netinet/tcp_output.c commented out? (It looks to me like this will disable TSO for almost all the NFS writes.) - around line #880 in sys/netinet/tcp_output.c: /* * In case there are too many small fragments * don't use TSO: */ if (len <= max_len) { len = max_len; sendalot = 1; tso = 0; } This was added along with the other stuff that did the if_hw_tsomaxsegcount, etc and I never noticed it until now (not my patch). rick> danny > > > Good luck with it, rick > > > >> still, mlxen0 is about 250! with and without TSO > >> > >> > >>> > >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru > >>>>> <mailto:slw at zxy.spb.ru>> wrote: > >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: > >>>>> > >>>>>> hi, > >>>>>> I have a host (Dell R730) with both cards, connected to an HP8200 > >>>>>> switch at 10Gb. > >>>>>> when writing to the same storage (netapp) this is what I get: > >>>>>> ix0: ~130MGB/s > >>>>>> mlxen0 ~330MGB/s > >>>>>> this is via nfs/tcpv3 > >>>>>> > >>>>>> I can get similar (bad) performance with the mellanox if I > >>>>>> increase > >>>>>> the file size > >>>>>> to 512MGB. > >>>>> > >>>>> Look like mellanox have internal beffer for caching and do ACK > >>>>> acclerating. > >>>>> > >>>>>> so at face value, it seems the mlxen does a better use of > >>>>>> resources > >>>>>> than the intel. > >>>>>> Any ideas how to improve ix/intel's performance? > >>>>> > >>>>> Are you sure about netapp performance? > >>>>> _______________________________________________ > >>>>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list > >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net> > >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org > >>>>> <mailto:freebsd-net-unsubscribe at freebsd.org>" > >>>>> > >>>> > >>>> _______________________________________________ > >>>> freebsd-stable at freebsd.org mailing list > >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >>>> To unsubscribe, send any mail to > >>>> "freebsd-stable-unsubscribe at freebsd.org" > >> > >> _______________________________________________ > >> freebsd-stable at freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" > > _______________________________________________ > freebsd-net at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
Daniel Braniss wrote:> > > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote: > > > > Daniel Braniss wrote: > >> > >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote: > >>> > >>> Daniel Braniss wrote: > >>>> > >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron > >>>>> <csforgeron at gmail.com> > >>>>> wrote: > >>>>> > >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD > >>>>> 10.1. Before 10.1 it was less. > >>>>> > >>>> > >>>> this is NOT iperf/3 where i do get close to wire speed, > >>>> it?s NFS writes, i.e., almost real work :-) > >>>> > >>>>> I used to tweak the card settings, but now it's just stock. You may > >>>>> want > >>>>> to > >>>>> check your settings, the Mellanox may just have better defaults for > >>>>> your > >>>>> switch. > >>>>> > >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be > >>> copying > >>> every transmitted mbuf chain to a new chain of mbuf clusters via. > >>> m_defrag() when > >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the > >>> 82599 chip > >>> these days?) > >>>Oops, I think I screwed up. It looks like t_maxopd is limited to somewhat less than the mtu. If that is the case, the code block wouldn't do what I thought it would do. However, if_hw_tsomaxsegcount does need to be one less than the limit for the driver, since the tcp/ip header isn't yet prepended when it is counted. I think the code in tcp_output() should subtract 1, but you can change it in the driver to test this. Thanks for doing this, rick> >> > >> hi Rick > >> > >> how can i check the chip? > >> > > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO > > helped, > > I'll bet you don't have a 82598.) > > > >>> This has been fixed in the driver very recently, but those fixes won't be > >>> in 10.1. > >>> > >>> rick > >>> ps: If you could test with 10.2, it would be interesting to see how the > >>> ix > >>> does with > >>> the current driver fixes in it? > >> > >> I new TSO was involved! > >> ok, firstly, it?s 10.2 stable. > >> with TSO enabled, ix is bad, around 64MGB/s. > >> disabling TSO it?s better, around 130 > >> > > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at > > around > > line#2500? > > /* TSO parameters */ > > 2572 ifp->if_hw_tsomax = 65518; > > 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; > > 2574 ifp->if_hw_tsomaxsegsize = 2048; > > > > They are in stable/10. I didn't look at releng/10.2. (And if they're in a > > #ifdef > > for FreeBSD11, take the #ifdef away.) > > If they are there and not ifdef'd, I can't explain why disabling TSO would > > help. > > Once TSO is fixed so that it handles the 64K transmit segments without > > copying all > > the mbufs, I suspect you might get better perf. with it enabled? > > > > this is 10.2 : > they are on lines 2509-2511 and I don?t see any #ifdefs around it. > > the plot thickens :-) > > danny > > > Good luck with it, rick > > > >> still, mlxen0 is about 250! with and without TSO > >> > >> > >>> > >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru > >>>>> <mailto:slw at zxy.spb.ru>> wrote: > >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: > >>>>> > >>>>>> hi, > >>>>>> I have a host (Dell R730) with both cards, connected to an HP8200 > >>>>>> switch at 10Gb. > >>>>>> when writing to the same storage (netapp) this is what I get: > >>>>>> ix0: ~130MGB/s > >>>>>> mlxen0 ~330MGB/s > >>>>>> this is via nfs/tcpv3 > >>>>>> > >>>>>> I can get similar (bad) performance with the mellanox if I > >>>>>> increase > >>>>>> the file size > >>>>>> to 512MGB. > >>>>> > >>>>> Look like mellanox have internal beffer for caching and do ACK > >>>>> acclerating. > >>>>> > >>>>>> so at face value, it seems the mlxen does a better use of > >>>>>> resources > >>>>>> than the intel. > >>>>>> Any ideas how to improve ix/intel's performance? > >>>>> > >>>>> Are you sure about netapp performance? > >>>>> _______________________________________________ > >>>>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list > >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net> > >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org > >>>>> <mailto:freebsd-net-unsubscribe at freebsd.org>" > >>>>> > >>>> > >>>> _______________________________________________ > >>>> freebsd-stable at freebsd.org mailing list > >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >>>> To unsubscribe, send any mail to > >>>> "freebsd-stable-unsubscribe at freebsd.org" > >> > >> _______________________________________________ > >> freebsd-stable at freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" > > _______________________________________________ > freebsd-net at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"