Am 21.07.2016 um 20:56 schrieb Russell R Poyner:> Jeremy, > > I think this is exactly a complex interaction between FreeBSD and > Samba. Best guess would be some system call that is fast in linux but > slow in FreeBSD holding things back. > > Russ > > On 07/21/2016 01:00 PM, Jeremy Allison wrote: >> On Thu, Jul 21, 2016 at 12:23:01PM -0500, Russell R Poyner wrote: >>> One more data point for comparison >>> >>> I installed the stock samba 4.2 rpm on a centos 7 machine and ran >>> the same diskspd tests against a share configured with: >>> vfs objects = aio_pthread >>> aio read size = 1024 >>> aio read size = 1024 >>> >>> smb2 leases = yes >>> >>> I get 27MB/s with 4k blocks and 145MB/s with 64k blocks. Disabling >>> cacheing by passing the -h switch to diskspd lowered these to 72MB/s >>> and 11MB/s. Which I view as 'close enough' to wire speed. Thus it >>> seems that the dismal performance I see is associated with the >>> FreeBSD implementation somehow. >> That's interesting, but I'm afraid I don't know FreeBSD well >> enough to help here. This does imply the problem isn't Samba >> specific though (unless it's a complex interaction between >> Samba+FreeBSD). > >On my debian jessie server with zfs these settings seem to work. max xmit = 65536 socket options = TCP_NODELAY Copying an file from server to a windows 7 client increases from 50% to 80% network utilisation on an 1GB link without jumbo frames when i add these settings.
On Thu, Jul 21, 2016 at 10:50:16PM +0200, Achim Gottinger wrote:> > > Am 21.07.2016 um 20:56 schrieb Russell R Poyner: > >Jeremy, > > > >I think this is exactly a complex interaction between FreeBSD and > >Samba. Best guess would be some system call that is fast in linux > >but slow in FreeBSD holding things back. > > > >Russ > > > >On 07/21/2016 01:00 PM, Jeremy Allison wrote: > >>On Thu, Jul 21, 2016 at 12:23:01PM -0500, Russell R Poyner wrote: > >>>One more data point for comparison > >>> > >>>I installed the stock samba 4.2 rpm on a centos 7 machine and ran > >>>the same diskspd tests against a share configured with: > >>> vfs objects = aio_pthread > >>> aio read size = 1024 > >>> aio read size = 1024 > >>> > >>> smb2 leases = yes > >>> > >>>I get 27MB/s with 4k blocks and 145MB/s with 64k blocks. Disabling > >>>cacheing by passing the -h switch to diskspd lowered these to 72MB/s > >>>and 11MB/s. Which I view as 'close enough' to wire speed. Thus it > >>>seems that the dismal performance I see is associated with the > >>>FreeBSD implementation somehow. > >>That's interesting, but I'm afraid I don't know FreeBSD well > >>enough to help here. This does imply the problem isn't Samba > >>specific though (unless it's a complex interaction between > >>Samba+FreeBSD). > > > > > On my debian jessie server with zfs these settings seem to work. > > max xmit = 65536 > socket options = TCP_NODELAY > > Copying an file from server to a windows 7 client increases from 50% > to 80% network utilisation on an 1GB link without jumbo frames when > i add these settings.That's kind of voodoo I'm afraid. By default Samba sets TCP_NODELAY, and max xmit is only used on SMB1, and modern Samba and Windows 7 should only be using SMB2.
Am 21.07.2016 um 22:59 schrieb Jeremy Allison:> On Thu, Jul 21, 2016 at 10:50:16PM +0200, Achim Gottinger wrote: >> >> Am 21.07.2016 um 20:56 schrieb Russell R Poyner: >>> Jeremy, >>> >>> I think this is exactly a complex interaction between FreeBSD and >>> Samba. Best guess would be some system call that is fast in linux >>> but slow in FreeBSD holding things back. >>> >>> Russ >>> >>> On 07/21/2016 01:00 PM, Jeremy Allison wrote: >>>> On Thu, Jul 21, 2016 at 12:23:01PM -0500, Russell R Poyner wrote: >>>>> One more data point for comparison >>>>> >>>>> I installed the stock samba 4.2 rpm on a centos 7 machine and ran >>>>> the same diskspd tests against a share configured with: >>>>> vfs objects = aio_pthread >>>>> aio read size = 1024 >>>>> aio read size = 1024 >>>>> >>>>> smb2 leases = yes >>>>> >>>>> I get 27MB/s with 4k blocks and 145MB/s with 64k blocks. Disabling >>>>> cacheing by passing the -h switch to diskspd lowered these to 72MB/s >>>>> and 11MB/s. Which I view as 'close enough' to wire speed. Thus it >>>>> seems that the dismal performance I see is associated with the >>>>> FreeBSD implementation somehow. >>>> That's interesting, but I'm afraid I don't know FreeBSD well >>>> enough to help here. This does imply the problem isn't Samba >>>> specific though (unless it's a complex interaction between >>>> Samba+FreeBSD). >>> >> On my debian jessie server with zfs these settings seem to work. >> >> max xmit = 65536 >> socket options = TCP_NODELAY >> >> Copying an file from server to a windows 7 client increases from 50% >> to 80% network utilisation on an 1GB link without jumbo frames when >> i add these settings. > That's kind of voodoo I'm afraid. By default Samba sets > TCP_NODELAY, and max xmit is only used on SMB1, and modern > Samba and Windows 7 should only be using SMB2.Thank you for the clarification, finaly gotta drop TCP_NODELAY from configs now. Indeed the connection is using smb2_10. I assume some intermediate caching influenced the copy test. Tested it a few times with these settings enabled/disabled before i posted but still... Now im at 80% without these settings. It is mentioned here (together with the abandonen IPTOS_LOWDELAY) http://unicolet.blogspot.de/2013/03/a-not-so-short-guide-to-zfs-on-linux.html I assume zfs set xattr=sa tank/fish zfs set atime=off tank/fish are an good idea on freebsd as well.