freebsd stable - Aug 2015 - ix(intel) vs mlxen(mellanox) 10Gb performance

> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron at
gmail.com> wrote:
> 
> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
10.1. Before 10.1 it was less.
> 
this is NOT iperf/3 where i do get close to wire speed,
it?s NFS writes, i.e., almost real work :-)
> I used to tweak the card settings, but now it's just stock. You may
want to check your settings, the Mellanox may just have better defaults for your
switch.
> 
> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru
<mailto:slw at zxy.spb.ru>> wrote:
> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> 
> > hi,
> >       I have a host (Dell R730) with both cards, connected to an
HP8200 switch at 10Gb.
> >       when writing to the same storage (netapp) this is what I get:
> >               ix0:            ~130MGB/s
> >               mlxen0  ~330MGB/s
> >       this is via nfs/tcpv3
> >
> >       I can get similar (bad) performance with the mellanox if I
increase the file size
> >       to 512MGB.
> 
> Look like mellanox have internal beffer for caching and do ACK acclerating.
> 
> >       so at face value, it seems the mlxen does a better use of
resources than the intel.
> >       Any ideas how to improve ix/intel's performance?
> 
> Are you sure about netapp performance?
> _______________________________________________
> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org>
mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
<https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at
freebsd.org <mailto:freebsd-net-unsubscribe at freebsd.org>"
>

Daniel Braniss wrote:
> 
> > On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron at
gmail.com>
> > wrote:
> > 
> > FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and
FreeBSD
> > 10.1. Before 10.1 it was less.
> > 
> 
> this is NOT iperf/3 where i do get close to wire speed,
> it?s NFS writes, i.e., almost real work :-)
> 
> > I used to tweak the card settings, but now it's just stock. You
may want to
> > check your settings, the Mellanox may just have better defaults for
your
> > switch.
> > 
Have you tried disabling TSO for the Intel? With TSO enabled, it will be copying
every transmitted mbuf chain to a new chain of mbuf clusters via. m_defrag()
when
TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the
82599 chip
these days?)

This has been fixed in the driver very recently, but those fixes won't be in
10.1.

rick
ps: If you could test with 10.2, it would be interesting to see how the ix does
with
    the current driver fixes in it?
> > On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at
zxy.spb.ru
> > <mailto:slw at zxy.spb.ru>> wrote:
> > On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> > 
> > > hi,
> > >       I have a host (Dell R730) with both cards, connected to an
HP8200
> > >       switch at 10Gb.
> > >       when writing to the same storage (netapp) this is what I
get:
> > >               ix0:            ~130MGB/s
> > >               mlxen0  ~330MGB/s
> > >       this is via nfs/tcpv3
> > >
> > >       I can get similar (bad) performance with the mellanox if I
increase
> > >       the file size
> > >       to 512MGB.
> > 
> > Look like mellanox have internal beffer for caching and do ACK
acclerating.
> > 
> > >       so at face value, it seems the mlxen does a better use of
resources
> > >       than the intel.
> > >       Any ideas how to improve ix/intel's performance?
> > 
> > Are you sure about netapp performance?
> > _______________________________________________
> > freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org>
mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at
freebsd.org
> > <mailto:freebsd-net-unsubscribe at freebsd.org>"
> > 
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"