> On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem at uoguelph.ca>
wrote:
>
> Daniel Braniss wrote:
>>
>>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at
uoguelph.ca> wrote:
>>>
>>> Daniel Braniss wrote:
>>>>
>>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
<csforgeron at gmail.com>
>>>>> wrote:
>>>>>
>>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel
X520-DA2's and FreeBSD
>>>>> 10.1. Before 10.1 it was less.
>>>>>
>>>>
>>>> this is NOT iperf/3 where i do get close to wire speed,
>>>> it?s NFS writes, i.e., almost real work :-)
>>>>
>>>>> I used to tweak the card settings, but now it's just
stock. You may want
>>>>> to
>>>>> check your settings, the Mellanox may just have better
defaults for your
>>>>> switch.
>>>>>
>>> Have you tried disabling TSO for the Intel? With TSO enabled, it
will be
>>> copying
>>> every transmitted mbuf chain to a new chain of mbuf clusters via.
>>> m_defrag() when
>>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem
to be the
>>> 82599 chip
>>> these days?)
>>>
>>
>> hi Rick
>>
>> how can i check the chip?
>>
> Haven't a clue. Does "dmesg" tell you? (To be honest, since
disabling TSO helped,
> I'll bet you don't have a 82598.)
>
>>> This has been fixed in the driver very recently, but those fixes
won't be
>>> in 10.1.
>>>
>>> rick
>>> ps: If you could test with 10.2, it would be interesting to see how
the ix
>>> does with
>>> the current driver fixes in it?
>>
>> I new TSO was involved!
>> ok, firstly, it?s 10.2 stable.
>> with TSO enabled, ix is bad, around 64MGB/s.
>> disabling TSO it?s better, around 130
>>
> Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at
around
> line#2500?
> /* TSO parameters */
> 2572 ifp->if_hw_tsomax = 65518;
> 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER;
> 2574 ifp->if_hw_tsomaxsegsize = 2048;
>
> They are in stable/10. I didn't look at releng/10.2. (And if
they're in a #ifdef
> for FreeBSD11, take the #ifdef away.)
> If they are there and not ifdef'd, I can't explain why disabling
TSO would help.
> Once TSO is fixed so that it handles the 64K transmit segments without
copying all
> the mbufs, I suspect you might get better perf. with it enabled?
>
this is 10.2 :
they are on lines 2509-2511 and I don?t see any #ifdefs around it.
the plot thickens :-)
danny
> Good luck with it, rick
>
>> still, mlxen0 is about 250! with and without TSO
>>
>>
>>>
>>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw
at zxy.spb.ru
>>>>> <mailto:slw at zxy.spb.ru>> wrote:
>>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss
wrote:
>>>>>
>>>>>> hi,
>>>>>> I have a host (Dell R730) with both cards,
connected to an HP8200
>>>>>> switch at 10Gb.
>>>>>> when writing to the same storage (netapp) this is
what I get:
>>>>>> ix0: ~130MGB/s
>>>>>> mlxen0 ~330MGB/s
>>>>>> this is via nfs/tcpv3
>>>>>>
>>>>>> I can get similar (bad) performance with the
mellanox if I increase
>>>>>> the file size
>>>>>> to 512MGB.
>>>>>
>>>>> Look like mellanox have internal beffer for caching and do
ACK
>>>>> acclerating.
>>>>>
>>>>>> so at face value, it seems the mlxen does a better
use of resources
>>>>>> than the intel.
>>>>>> Any ideas how to improve ix/intel's
performance?
>>>>>
>>>>> Are you sure about netapp performance?
>>>>> _______________________________________________
>>>>> freebsd-net at freebsd.org <mailto:freebsd-net at
freebsd.org> mailing list
>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>
<https://lists.freebsd.org/mailman/listinfo/freebsd-net>
>>>>> To unsubscribe, send any mail to
"freebsd-net-unsubscribe at freebsd.org
>>>>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>>>>
>>>>
>>>> _______________________________________________
>>>> freebsd-stable at freebsd.org mailing list
>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>>> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe at freebsd.org"
>>
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"