I'm tuning a samba 4.3 install on freebsd and I'm confused about aio
settings.
I've loaded the freebsd aio kernel module and tried various values or
aio read size and aio write size, but it seems to make no difference in
the speed.
Using MS diskspd against a samba share from a fast zfs pool I get
something like 25MB/s tops. That's well below the capacity of my Gb
network and my disk system. FWIW iperf shows >900Mbits/sec in both
directions on the link.
# smbd -b|grep aio
vfs_aio_fork_init
vfs_aio_posix_init
vfs_aio_pthread_init
As always google finds lots of tuning advice, but it's not clear what if
any of it applies to 4.3 on FreeBSD.
Thanks
Russ Poyner
On Tue, Jul 19, 2016 at 12:49:09PM -0500, Russell R Poyner wrote:> I'm tuning a samba 4.3 install on freebsd and I'm confused about aio > settings. > > I've loaded the freebsd aio kernel module and tried various values > or aio read size and aio write size, but it seems to make no > difference in the speed. > > Using MS diskspd against a samba share from a fast zfs pool I get > something like 25MB/s tops. That's well below the capacity of my Gb > network and my disk system. FWIW iperf shows >900Mbits/sec in both > directions on the link. > > # smbd -b|grep aio > vfs_aio_fork_init > vfs_aio_posix_init > vfs_aio_pthread_initYou don't need these, modern Samba includes a pthread pool implementation that will parallelize SMB io requests. Ensure the clients are using SMB2 and SMB2 leases are enabled. You should be able to get close to wirespeed.
On Tue, Jul 19, 2016 at 12:42:53PM -0700, Jeremy Allison wrote:> On Tue, Jul 19, 2016 at 12:49:09PM -0500, Russell R Poyner wrote: > > I'm tuning a samba 4.3 install on freebsd and I'm confused about aio > > settings. > > > > I've loaded the freebsd aio kernel module and tried various values > > or aio read size and aio write size, but it seems to make no > > difference in the speed. > > > > Using MS diskspd against a samba share from a fast zfs pool I get > > something like 25MB/s tops. That's well below the capacity of my Gb > > network and my disk system. FWIW iperf shows >900Mbits/sec in both > > directions on the link. > > > > # smbd -b|grep aio > > vfs_aio_fork_init > > vfs_aio_posix_init > > vfs_aio_pthread_init > > You don't need these, modern Samba includes a pthread pool > implementation that will parallelize SMB io requests.The main reason for our user-space threaded approach is lack of aio in Linux. Proper kernel support for posix AIO might be faster than our implementation. "vfs objects = aio_posix" will give you that. This needs very thorough performance testing. If it turns out to be faster than our threaded aio on FreeBSD, we might have to revive the aio_posix module, it went away last year. Volker
Jeremy, I re-built the samba43 port with aio_support unset and pthreadpool set. smbd -b|grep aio vfs_aio_fork_init vfs_aio_pthread_init from smb4.conf: oplocks = yes kernel oplocks = no smb2 leases = yes server min protocol = smb2 aio read size = 1024 aio write size = 1024 vfs objects = aio_pthread The new build gives pretty much the same speeds as before using MS diskspd under windows 7. Around 15MB/s for a random workload with 4k blocks. Up to about 25MB/s with 64k blocks. For comparison I ran diskspd on the windows 7 box using a windows 8 machine as the server. The results were 49MB/s for 4k blocks and 1787MB/s for 64k blocks. Clearly 1787 MB/s is more than wire speed and reflects a cache effect that samba doesn't access. With the new build I do see smbd spawning extra threads under load. They just don't seem to add any performance benefit. This is a freebsd 10.2 system with samba running inside a jail. Thanks again Russ poyner On 07/19/2016 02:42 PM, Jeremy Allison wrote:> On Tue, Jul 19, 2016 at 12:49:09PM -0500, Russell R Poyner wrote: >> I'm tuning a samba 4.3 install on freebsd and I'm confused about aio >> settings. >> >> I've loaded the freebsd aio kernel module and tried various values >> or aio read size and aio write size, but it seems to make no >> difference in the speed. >> >> Using MS diskspd against a samba share from a fast zfs pool I get >> something like 25MB/s tops. That's well below the capacity of my Gb >> network and my disk system. FWIW iperf shows >900Mbits/sec in both >> directions on the link. >> >> # smbd -b|grep aio >> vfs_aio_fork_init >> vfs_aio_posix_init >> vfs_aio_pthread_init > You don't need these, modern Samba includes a pthread pool > implementation that will parallelize SMB io requests. > > Ensure the clients are using SMB2 and SMB2 leases are > enabled. You should be able to get close to wirespeed.