Daniel Braniss wrote:>
> > On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at
uoguelph.ca> wrote:
> >
> > Daniel Braniss wrote:
> >>
> >>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
<csforgeron at gmail.com>
> >>> wrote:
> >>>
> >>> FYI, I can regularly hit 9.3 Gib/s with my Intel
X520-DA2's and FreeBSD
> >>> 10.1. Before 10.1 it was less.
> >>>
> >>
> >> this is NOT iperf/3 where i do get close to wire speed,
> >> it?s NFS writes, i.e., almost real work :-)
> >>
> >>> I used to tweak the card settings, but now it's just
stock. You may want
> >>> to
> >>> check your settings, the Mellanox may just have better
defaults for your
> >>> switch.
> >>>
> > Have you tried disabling TSO for the Intel? With TSO enabled, it will
be
> > copying
> > every transmitted mbuf chain to a new chain of mbuf clusters via.
> > m_defrag() when
> > TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to
be the
> > 82599 chip
> > these days?)
> >
>
> hi Rick
>
> how can i check the chip?
>
Haven't a clue. Does "dmesg" tell you? (To be honest, since
disabling TSO helped,
I'll bet you don't have a 82598.)
> > This has been fixed in the driver very recently, but those fixes
won't be
> > in 10.1.
> >
> > rick
> > ps: If you could test with 10.2, it would be interesting to see how
the ix
> > does with
> > the current driver fixes in it?
>
> I new TSO was involved!
> ok, firstly, it?s 10.2 stable.
> with TSO enabled, ix is bad, around 64MGB/s.
> disabling TSO it?s better, around 130
>
Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at
around
line#2500?
/* TSO parameters */
2572 ifp->if_hw_tsomax = 65518;
2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER;
2574 ifp->if_hw_tsomaxsegsize = 2048;
They are in stable/10. I didn't look at releng/10.2. (And if they're in
a #ifdef
for FreeBSD11, take the #ifdef away.)
If they are there and not ifdef'd, I can't explain why disabling TSO
would help.
Once TSO is fixed so that it handles the 64K transmit segments without copying
all
the mbufs, I suspect you might get better perf. with it enabled?
Good luck with it, rick
> still, mlxen0 is about 250! with and without TSO
>
>
> >
> >>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at
zxy.spb.ru
> >>> <mailto:slw at zxy.spb.ru>> wrote:
> >>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss
wrote:
> >>>
> >>>> hi,
> >>>> I have a host (Dell R730) with both cards, connected
to an HP8200
> >>>> switch at 10Gb.
> >>>> when writing to the same storage (netapp) this is
what I get:
> >>>> ix0: ~130MGB/s
> >>>> mlxen0 ~330MGB/s
> >>>> this is via nfs/tcpv3
> >>>>
> >>>> I can get similar (bad) performance with the mellanox
if I increase
> >>>> the file size
> >>>> to 512MGB.
> >>>
> >>> Look like mellanox have internal beffer for caching and do ACK
> >>> acclerating.
> >>>
> >>>> so at face value, it seems the mlxen does a better
use of resources
> >>>> than the intel.
> >>>> Any ideas how to improve ix/intel's performance?
> >>>
> >>> Are you sure about netapp performance?
> >>> _______________________________________________
> >>> freebsd-net at freebsd.org <mailto:freebsd-net at
freebsd.org> mailing list
> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe
at freebsd.org
> >>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
> >>>
> >>
> >> _______________________________________________
> >> freebsd-stable at freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe
at freebsd.org"
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"