On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wrote:> On 08/18/15 23:54, Rick Macklem wrote: > >Ouch! Yes, I now see that the code that counts the # of mbufs is before the > >code that adds the tcp/ip header mbuf. > > > >In my opinion, this should be fixed by setting if_hw_tsomaxsegcount to > >whatever > >the driver provides - 1. It is not the driver's responsibility to know if > >a tcp/ip > >header mbuf will be added and is a lot less confusing that expecting the > >driver > >author to know to subtract one. (I had mistakenly thought that > >tcp_output() had > >added the tc/ip header mbuf before the loop that counts mbufs in the list. > >Btw, > >this tcp/ip header mbuf also has leading space for the MAC layer header.) > > > > Hi Rick, > > Your question is good. With the Mellanox hardware we have separate > so-called inline data space for the TCP/IP headers, so if the TCP stack > subtracts something, then we would need to add something to the limit, > because then the scatter gather list is only used for the data part. >I think all drivers in tree don't subtract 1 for if_hw_tsomaxsegcount. Probably touching Mellanox driver would be simpler than fixing all other drivers in tree.> Maybe it can be controlled by some kind of flag, if all the three TSO > limits should include the TCP/IP/ethernet headers too. I'm pretty sure > we want both versions. >Hmm, I'm afraid it's already complex. Drivers have to tell almost the same information to both bus_dma(9) and network stack.
On 08/19/15 09:42, Yonghyeon PYUN wrote:> On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wrote: >> On 08/18/15 23:54, Rick Macklem wrote: >>> Ouch! Yes, I now see that the code that counts the # of mbufs is before the >>> code that adds the tcp/ip header mbuf. >>> >>> In my opinion, this should be fixed by setting if_hw_tsomaxsegcount to >>> whatever >>> the driver provides - 1. It is not the driver's responsibility to know if >>> a tcp/ip >>> header mbuf will be added and is a lot less confusing that expecting the >>> driver >>> author to know to subtract one. (I had mistakenly thought that >>> tcp_output() had >>> added the tc/ip header mbuf before the loop that counts mbufs in the list. >>> Btw, >>> this tcp/ip header mbuf also has leading space for the MAC layer header.) >>> >> >> Hi Rick, >> >> Your question is good. With the Mellanox hardware we have separate >> so-called inline data space for the TCP/IP headers, so if the TCP stack >> subtracts something, then we would need to add something to the limit, >> because then the scatter gather list is only used for the data part. >> > > I think all drivers in tree don't subtract 1 for > if_hw_tsomaxsegcount. Probably touching Mellanox driver would be > simpler than fixing all other drivers in tree.Hi, If you change the behaviour don't forget to update and/or add comments describing it. Maybe the amount of subtraction could be defined by some macro? Then drivers which inline the headers can subtract it? Your suggestion is fine by me. The initial TSO limits were tried to be preserved, and I believe that TSO limits never accounted for IP/TCP/ETHERNET/VLAN headers!> >> Maybe it can be controlled by some kind of flag, if all the three TSO >> limits should include the TCP/IP/ethernet headers too. I'm pretty sure >> we want both versions. >> > > Hmm, I'm afraid it's already complex. Drivers have to tell almost > the same information to both bus_dma(9) and network stack.You're right it's complicated. Not sure if bus_dma can provide an API for this though. --HPS
On 08/19/15 09:42, Yonghyeon PYUN wrote:> On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wrote: >> On 08/18/15 23:54, Rick Macklem wrote: >>> Ouch! Yes, I now see that the code that counts the # of mbufs is before the >>> code that adds the tcp/ip header mbuf. >>> >>> In my opinion, this should be fixed by setting if_hw_tsomaxsegcount to >>> whatever >>> the driver provides - 1. It is not the driver's responsibility to know if >>> a tcp/ip >>> header mbuf will be added and is a lot less confusing that expecting the >>> driver >>> author to know to subtract one. (I had mistakenly thought that >>> tcp_output() had >>> added the tc/ip header mbuf before the loop that counts mbufs in the list. >>> Btw, >>> this tcp/ip header mbuf also has leading space for the MAC layer header.) >>> >> >> Hi Rick, >> >> Your question is good. With the Mellanox hardware we have separate >> so-called inline data space for the TCP/IP headers, so if the TCP stack >> subtracts something, then we would need to add something to the limit, >> because then the scatter gather list is only used for the data part. >> > > I think all drivers in tree don't subtract 1 for > if_hw_tsomaxsegcount. Probably touching Mellanox driver would be > simpler than fixing all other drivers in tree. > >> Maybe it can be controlled by some kind of flag, if all the three TSO >> limits should include the TCP/IP/ethernet headers too. I'm pretty sure >> we want both versions. >> > > Hmm, I'm afraid it's already complex. Drivers have to tell almost > the same information to both bus_dma(9) and network stack.Don't forget that not all drivers in the tree set the TSO limits before if_attach(), so possibly the subtraction of one TSO fragment needs to go into ip_output() .... --HPS