freebsd stable - Aug 2015 - ix(intel) vs mlxen(mellanox) 10Gb performance

On 08/19/15 09:42, Yonghyeon PYUN wrote:
> On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wrote:
>> On 08/18/15 23:54, Rick Macklem wrote:
>>> Ouch! Yes, I now see that the code that counts the # of mbufs is
before the
>>> code that adds the tcp/ip header mbuf.
>>>
>>> In my opinion, this should be fixed by setting if_hw_tsomaxsegcount
to
>>> whatever
>>> the driver provides - 1. It is not the driver's responsibility
to know if
>>> a tcp/ip
>>> header mbuf will be added and is a lot less confusing that
expecting the
>>> driver
>>> author to know to subtract one. (I had mistakenly thought that
>>> tcp_output() had
>>> added the tc/ip header mbuf before the loop that counts mbufs in
the list.
>>> Btw,
>>> this tcp/ip header mbuf also has leading space for the MAC layer
header.)
>>>
>>
>> Hi Rick,
>>
>> Your question is good. With the Mellanox hardware we have separate
>> so-called inline data space for the TCP/IP headers, so if the TCP stack
>> subtracts something, then we would need to add something to the limit,
>> because then the scatter gather list is only used for the data part.
>>
>
> I think all drivers in tree don't subtract 1 for
> if_hw_tsomaxsegcount.  Probably touching Mellanox driver would be
> simpler than fixing all other drivers in tree.
Hi,

If you change the behaviour don't forget to update and/or add comments 
describing it. Maybe the amount of subtraction could be defined by some 
macro? Then drivers which inline the headers can subtract it?

Your suggestion is fine by me.

The initial TSO limits were tried to be preserved, and I believe that 
TSO limits never accounted for IP/TCP/ETHERNET/VLAN headers!
>
>> Maybe it can be controlled by some kind of flag, if all the three TSO
>> limits should include the TCP/IP/ethernet headers too. I'm pretty
sure
>> we want both versions.
>>
>
> Hmm, I'm afraid it's already complex.  Drivers have to tell almost
> the same information to both bus_dma(9) and network stack.
You're right it's complicated. Not sure if bus_dma can provide an API 
for this though.

--HPS

On Wed, Aug 19, 2015 at 09:51:44AM +0200, Hans Petter Selasky
wrote:
> On 08/19/15 09:42, Yonghyeon PYUN wrote:
> >On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wrote:
> >>On 08/18/15 23:54, Rick Macklem wrote:
> >>>Ouch! Yes, I now see that the code that counts the # of mbufs
is before
> >>>the
> >>>code that adds the tcp/ip header mbuf.
> >>>
> >>>In my opinion, this should be fixed by setting
if_hw_tsomaxsegcount to
> >>>whatever
> >>>the driver provides - 1. It is not the driver's
responsibility to know if
> >>>a tcp/ip
> >>>header mbuf will be added and is a lot less confusing that
expecting the
> >>>driver
> >>>author to know to subtract one. (I had mistakenly thought that
> >>>tcp_output() had
> >>>added the tc/ip header mbuf before the loop that counts mbufs
in the
> >>>list.
> >>>Btw,
> >>>this tcp/ip header mbuf also has leading space for the MAC
layer header.)
> >>>
> >>
> >>Hi Rick,
> >>
> >>Your question is good. With the Mellanox hardware we have separate
> >>so-called inline data space for the TCP/IP headers, so if the TCP
stack
> >>subtracts something, then we would need to add something to the
limit,
> >>because then the scatter gather list is only used for the data
part.
> >>
> >
> >I think all drivers in tree don't subtract 1 for
> >if_hw_tsomaxsegcount.  Probably touching Mellanox driver would be
> >simpler than fixing all other drivers in tree.
> 
> Hi,
> 
> If you change the behaviour don't forget to update and/or add comments 
> describing it. Maybe the amount of subtraction could be defined by some 
> macro? Then drivers which inline the headers can subtract it?
> 
I'm also ok with your suggestion.
> Your suggestion is fine by me.
> 
> The initial TSO limits were tried to be preserved, and I believe that 
> TSO limits never accounted for IP/TCP/ETHERNET/VLAN headers!
> 
I guess FreeBSD used to follow MS LSOv1 specification with minor
exception in pseudo checksum computation. If I recall correctly the
specification says upper stack can generate up to IP_MAXPACKET sized
packet.  Other L2 headers like ethernet/vlan header size is not
included in the packet and it's drivers responsibility to allocate
additional DMA buffers/segments for L2 headers.
> >
> >>Maybe it can be controlled by some kind of flag, if all the three
TSO
> >>limits should include the TCP/IP/ethernet headers too. I'm
pretty sure
> >>we want both versions.
> >>
> >
> >Hmm, I'm afraid it's already complex.  Drivers have to tell
almost
> >the same information to both bus_dma(9) and network stack.
> 
> You're right it's complicated. Not sure if bus_dma can provide an
API
> for this though.
> 
> --HPS