Daniel Braniss wrote:>
> > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem at
uoguelph.ca> wrote:
> >
> > Daniel Braniss wrote:
> >>
> >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at
uoguelph.ca> wrote:
> >>>
> >>> Daniel Braniss wrote:
> >>>>
> >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
> >>>>> <csforgeron at gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel
X520-DA2's and FreeBSD
> >>>>> 10.1. Before 10.1 it was less.
> >>>>>
> >>>>
> >>>> this is NOT iperf/3 where i do get close to wire speed,
> >>>> it?s NFS writes, i.e., almost real work :-)
> >>>>
> >>>>> I used to tweak the card settings, but now it's
just stock. You may
> >>>>> want
> >>>>> to
> >>>>> check your settings, the Mellanox may just have better
defaults for
> >>>>> your
> >>>>> switch.
> >>>>>
> >>> Have you tried disabling TSO for the Intel? With TSO enabled,
it will be
> >>> copying
> >>> every transmitted mbuf chain to a new chain of mbuf clusters
via.
> >>> m_defrag() when
> >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most
seem to be the
> >>> 82599 chip
> >>> these days?)
> >>>
> >>
> >> hi Rick
> >>
> >> how can i check the chip?
> >>
> > Haven't a clue. Does "dmesg" tell you? (To be honest,
since disabling TSO
> > helped,
> > I'll bet you don't have a 82598.)
> >
> >>> This has been fixed in the driver very recently, but those
fixes won't be
> >>> in 10.1.
> >>>
> >>> rick
> >>> ps: If you could test with 10.2, it would be interesting to
see how the
> >>> ix
> >>> does with
> >>> the current driver fixes in it?
> >>
> >> I new TSO was involved!
> >> ok, firstly, it?s 10.2 stable.
> >> with TSO enabled, ix is bad, around 64MGB/s.
> >> disabling TSO it?s better, around 130
> >>
> > Hmm, could you check to see of these lines are in
sys/dev/ixgbe/if_ix.c at
> > around
> > line#2500?
> > /* TSO parameters */
> > 2572 ifp->if_hw_tsomax = 65518;
> > 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER;
> > 2574 ifp->if_hw_tsomaxsegsize = 2048;
> >
> > They are in stable/10. I didn't look at releng/10.2. (And if
they're in a
> > #ifdef
> > for FreeBSD11, take the #ifdef away.)
> > If they are there and not ifdef'd, I can't explain why
disabling TSO would
> > help.
> > Once TSO is fixed so that it handles the 64K transmit segments without
> > copying all
> > the mbufs, I suspect you might get better perf. with it enabled?
> >
>
> this is 10.2 :
> they are on lines 2509-2511 and I don?t see any #ifdefs around it.
>
> the plot thickens :-)
>
If this is just a test machine, maybe you could test with these lines (at about
#880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will
disable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
/*
* In case there are too many small fragments
* don't use TSO:
*/
if (len <= max_len) {
len = max_len;
sendalot = 1;
tso = 0;
}
This was added along with the other stuff that did the if_hw_tsomaxsegcount, etc
and I
never noticed it until now (not my patch).
rick
> danny
>
> > Good luck with it, rick
> >
> >> still, mlxen0 is about 250! with and without TSO
> >>
> >>
> >>>
> >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov
<slw at zxy.spb.ru
> >>>>> <mailto:slw at zxy.spb.ru>> wrote:
> >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel
Braniss wrote:
> >>>>>
> >>>>>> hi,
> >>>>>> I have a host (Dell R730) with both cards,
connected to an HP8200
> >>>>>> switch at 10Gb.
> >>>>>> when writing to the same storage (netapp) this
is what I get:
> >>>>>> ix0: ~130MGB/s
> >>>>>> mlxen0 ~330MGB/s
> >>>>>> this is via nfs/tcpv3
> >>>>>>
> >>>>>> I can get similar (bad) performance with the
mellanox if I
> >>>>>> increase
> >>>>>> the file size
> >>>>>> to 512MGB.
> >>>>>
> >>>>> Look like mellanox have internal beffer for caching
and do ACK
> >>>>> acclerating.
> >>>>>
> >>>>>> so at face value, it seems the mlxen does a
better use of
> >>>>>> resources
> >>>>>> than the intel.
> >>>>>> Any ideas how to improve ix/intel's
performance?
> >>>>>
> >>>>> Are you sure about netapp performance?
> >>>>> _______________________________________________
> >>>>> freebsd-net at freebsd.org <mailto:freebsd-net at
freebsd.org> mailing list
> >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>>>>
<https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> >>>>> To unsubscribe, send any mail to
"freebsd-net-unsubscribe at freebsd.org
> >>>>> <mailto:freebsd-net-unsubscribe at
freebsd.org>"
> >>>>>
> >>>>
> >>>> _______________________________________________
> >>>> freebsd-stable at freebsd.org mailing list
> >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-stable-unsubscribe at freebsd.org"
> >>
> >> _______________________________________________
> >> freebsd-stable at freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe
at freebsd.org"
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at
freebsd.org"