On Tue, Jul 19, 2016 at 12:42:53PM -0700, Jeremy Allison wrote:> On Tue, Jul 19, 2016 at 12:49:09PM -0500, Russell R Poyner wrote: > > I'm tuning a samba 4.3 install on freebsd and I'm confused about aio > > settings. > > > > I've loaded the freebsd aio kernel module and tried various values > > or aio read size and aio write size, but it seems to make no > > difference in the speed. > > > > Using MS diskspd against a samba share from a fast zfs pool I get > > something like 25MB/s tops. That's well below the capacity of my Gb > > network and my disk system. FWIW iperf shows >900Mbits/sec in both > > directions on the link. > > > > # smbd -b|grep aio > > vfs_aio_fork_init > > vfs_aio_posix_init > > vfs_aio_pthread_init > > You don't need these, modern Samba includes a pthread pool > implementation that will parallelize SMB io requests.The main reason for our user-space threaded approach is lack of aio in Linux. Proper kernel support for posix AIO might be faster than our implementation. "vfs objects = aio_posix" will give you that. This needs very thorough performance testing. If it turns out to be faster than our threaded aio on FreeBSD, we might have to revive the aio_posix module, it went away last year. Volker
Volker, Today I built samba43 with support for both posix and pthread aio. I then ran the diskspd tests using either vfs objects = aio_pthread or vfs objects = aio_posix. There appears to be a very small advantage to the aio_pthread implementation. Hard to say for sure given the run to run variation in the numbers. I'm left to conclude that something else in samba is limiting performance. The best I've been able to measure was 48MB/s using 64k blocks against a memdisk on the FreeBSD server. Still only around 1/2 of wire speed as measured by iperf, and much less than I see running the same test against a windows 8.1 server sharing from a single 7200rpm disk. Any suggestions on to where to look next are welcome. Thanks again Russ Poyner On 07/20/2016 03:58 AM, Volker Lendecke wrote:> On Tue, Jul 19, 2016 at 12:42:53PM -0700, Jeremy Allison wrote: >> On Tue, Jul 19, 2016 at 12:49:09PM -0500, Russell R Poyner wrote: >>> I'm tuning a samba 4.3 install on freebsd and I'm confused about aio >>> settings. >>> >>> I've loaded the freebsd aio kernel module and tried various values >>> or aio read size and aio write size, but it seems to make no >>> difference in the speed. >>> >>> Using MS diskspd against a samba share from a fast zfs pool I get >>> something like 25MB/s tops. That's well below the capacity of my Gb >>> network and my disk system. FWIW iperf shows >900Mbits/sec in both >>> directions on the link. >>> >>> # smbd -b|grep aio >>> vfs_aio_fork_init >>> vfs_aio_posix_init >>> vfs_aio_pthread_init >> You don't need these, modern Samba includes a pthread pool >> implementation that will parallelize SMB io requests. > The main reason for our user-space threaded approach is lack of aio in > Linux. Proper kernel support for posix AIO might be faster than our > implementation. "vfs objects = aio_posix" will give you that. > > This needs very thorough performance testing. If it turns out to be > faster than our threaded aio on FreeBSD, we might have to revive > the aio_posix module, it went away last year. > > Volker
One more data point for comparison
I installed the stock samba 4.2 rpm on a centos 7 machine and ran the
same diskspd tests against a share configured with:
vfs objects = aio_pthread
aio read size = 1024
aio read size = 1024
smb2 leases = yes
I get 27MB/s with 4k blocks and 145MB/s with 64k blocks. Disabling
cacheing by passing the -h switch to diskspd lowered these to 72MB/s and
11MB/s. Which I view as 'close enough' to wire speed. Thus it seems that
the dismal performance I see is associated with the FreeBSD
implementation somehow.
Thanks again
RP
On 07/20/2016 03:58 AM, Volker Lendecke wrote:> On Tue, Jul 19, 2016 at 12:42:53PM -0700, Jeremy Allison wrote:
>> On Tue, Jul 19, 2016 at 12:49:09PM -0500, Russell R Poyner wrote:
>>> I'm tuning a samba 4.3 install on freebsd and I'm confused
about aio
>>> settings.
>>>
>>> I've loaded the freebsd aio kernel module and tried various
values
>>> or aio read size and aio write size, but it seems to make no
>>> difference in the speed.
>>>
>>> Using MS diskspd against a samba share from a fast zfs pool I get
>>> something like 25MB/s tops. That's well below the capacity of
my Gb
>>> network and my disk system. FWIW iperf shows >900Mbits/sec in
both
>>> directions on the link.
>>>
>>> # smbd -b|grep aio
>>> vfs_aio_fork_init
>>> vfs_aio_posix_init
>>> vfs_aio_pthread_init
>> You don't need these, modern Samba includes a pthread pool
>> implementation that will parallelize SMB io requests.
> The main reason for our user-space threaded approach is lack of aio in
> Linux. Proper kernel support for posix AIO might be faster than our
> implementation. "vfs objects = aio_posix" will give you that.
>
> This needs very thorough performance testing. If it turns out to be
> faster than our threaded aio on FreeBSD, we might have to revive
> the aio_posix module, it went away last year.
>
> Volker
Am 21.07.2016 um 19:23 schrieb Russell R Poyner:> One more data point for comparison > > I installed the stock samba 4.2 rpm on a centos 7 machine and ran the > same diskspd tests against a share configured with: > vfs objects = aio_pthread > aio read size = 1024 > aio read size = 1024 > > smb2 leases = yes > > I get 27MB/s with 4k blocks and 145MB/s with 64k blocks. Disabling > cacheing by passing the -h switch to diskspd lowered these to 72MB/s > and 11MB/s. Which I view as 'close enough' to wire speed. Thus it > seems that the dismal performance I see is associated with the FreeBSD > implementation somehow. > > Thanks again > RP > > On 07/20/2016 03:58 AM, Volker Lendecke wrote: >> On Tue, Jul 19, 2016 at 12:42:53PM -0700, Jeremy Allison wrote: >>> On Tue, Jul 19, 2016 at 12:49:09PM -0500, Russell R Poyner wrote: >>>> I'm tuning a samba 4.3 install on freebsd and I'm confused about aio >>>> settings. >>>> >>>> I've loaded the freebsd aio kernel module and tried various values >>>> or aio read size and aio write size, but it seems to make no >>>> difference in the speed. >>>> >>>> Using MS diskspd against a samba share from a fast zfs pool I get >>>> something like 25MB/s tops. That's well below the capacity of my Gb >>>> network and my disk system. FWIW iperf shows >900Mbits/sec in both >>>> directions on the link. >>>> >>>> # smbd -b|grep aio >>>> vfs_aio_fork_init >>>> vfs_aio_posix_init >>>> vfs_aio_pthread_init >>> You don't need these, modern Samba includes a pthread pool >>> implementation that will parallelize SMB io requests. >> The main reason for our user-space threaded approach is lack of aio in >> Linux. Proper kernel support for posix AIO might be faster than our >> implementation. "vfs objects = aio_posix" will give you that. >> >> This needs very thorough performance testing. If it turns out to be >> faster than our threaded aio on FreeBSD, we might have to revive >> the aio_posix module, it went away last year. >> >> Volker > >I think you must tune zfs on freebsd and assume you used an different fs for the centos test.
On Thu, Jul 21, 2016 at 12:23:01PM -0500, Russell R Poyner wrote:> One more data point for comparison > > I installed the stock samba 4.2 rpm on a centos 7 machine and ran > the same diskspd tests against a share configured with: > vfs objects = aio_pthread > aio read size = 1024 > aio read size = 1024 > > smb2 leases = yes > > I get 27MB/s with 4k blocks and 145MB/s with 64k blocks. Disabling > cacheing by passing the -h switch to diskspd lowered these to 72MB/s > and 11MB/s. Which I view as 'close enough' to wire speed. Thus it > seems that the dismal performance I see is associated with the > FreeBSD implementation somehow.That's interesting, but I'm afraid I don't know FreeBSD well enough to help here. This does imply the problem isn't Samba specific though (unless it's a complex interaction between Samba+FreeBSD).