samba - Nov 2019 - samba performance when writing lots of small files

hi there,

running a simple test like

time for s in $(seq 0 9999); do echo hello > world-$s; done

takes a little more than a minute on a locally mounted test share on our
samba server. the same test finishes in about 5 seconds when it is run
on the server's file system, without samba.

i understand that dealing with many small files isn't exactly the sweet
spot of any (networked) file system. however, more than one order of
magnitude seems a little much ?

the samba version is 4.7 from ubuntu bionic running in a clustered
environment with two servers. the underlying file system is a cephfs, so
server-side os caches _should_ not intervene in both test cases - fwiw,
running sync immediately after the file system only-test returned at once.

googling around, we already tuned the server like this:

        kernel change notify = no
        change notify = no
        read raw = Yes
        write raw = Yes
        aio read size = 1
        aio write size = 1

and also enabled the aio_pthread vfs module. we found that seemingly
relevant client-side options (cache=loose,fsc) made hardly any
difference. during both tests, the server was mostly idling.

do you have any recommendations on how to further improve sambas write
performance for this use case?

thank you very much & with kind regards,
thoralf.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190925/6e842029/signature.sig>

Hi Thoralf,

try to use a newer version of Samba 4.7 is already outdated. The Problem
the smb-protocol and not Samba it self. As Steve French mentioned on the
last SambaXP it will be much better with Linux 5.x. So maybe you try a
new kernel

Stefan

Am 25.09.19 um 16:37 schrieb thoralf schulze via samba:
> hi there,
>
> running a simple test like
>
> time for s in $(seq 0 9999); do echo hello > world-$s; done
>
> takes a little more than a minute on a locally mounted test share on our
> samba server. the same test finishes in about 5 seconds when it is run
> on the server's file system, without samba.
>
> i understand that dealing with many small files isn't exactly the sweet
> spot of any (networked) file system. however, more than one order of
> magnitude seems a little much ?
>
> the samba version is 4.7 from ubuntu bionic running in a clustered
> environment with two servers. the underlying file system is a cephfs, so
> server-side os caches _should_ not intervene in both test cases - fwiw,
> running sync immediately after the file system only-test returned at once.
>
> googling around, we already tuned the server like this:
>
>         kernel change notify = no
>         change notify = no
>         read raw = Yes
>         write raw = Yes
>         aio read size = 1
>         aio write size = 1
>
> and also enabled the aio_pthread vfs module. we found that seemingly
> relevant client-side options (cache=loose,fsc) made hardly any
> difference. during both tests, the server was mostly idling.
>
> do you have any recommendations on how to further improve sambas write
> performance for this use case?
>
> thank you very much & with kind regards,
> thoralf.
>
>
-- 
Stefan Kania

hkp://subkeys.pgp.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190926/bb428fb7/signature.sig>

hi Stefan / all,

resurrecting this thread ?

On 9/26/19 9:19 PM, Stefan Kania via samba wrote:
> try to use a newer version of Samba 4.7 is already outdated. The Problem
> the smb-protocol and not Samba it self. As Steve French mentioned on the
> last SambaXP it will be much better with Linux 5.x. So maybe you try a
> new kernel
root at plattentest:~# uname -a
Linux plattentest 5.3.0-18-generic #19-Ubuntu SMP Tue Oct 8 20:14:06 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
root at plattentest:~# dpkg -l samba
[?]
ii  samba          2:4.10.7+dfsg-0ubuntu3 amd64        SMB/CIFS file,
print, and login server for Unix
root at plattentest:~# time for s in $(seq 0 9999); do echo $s >
/tubfs/test/10.000/test-$s; done # this directory is a cephfs mount

real	0m5.921s
user	0m0.515s
sys	0m0.861s
root at plattentest:~# time for s in $(seq 0 9999); do echo $s >
/mnt/10.000-cifs/test-$s; done # this is the former re-exported via samba

real    13m27.317s
user    0m0.881s
sys     0m6.326s
root at plattentest:~# cat /etc/samba/smb.conf
[global]
        kernel change notify = no
        change notify = no
        read raw = Yes
        write raw = Yes
[test]
        path=/tubfs/test/
        guest ok = yes
        force user = root
        writeable = yes
        aio read size = 1
        aio write size = 1
root at plattentest:~# mount | grep test
//localhost/test on /mnt type cifs
(rw,relatime,vers=3.1.1,sec=none,cache=strict,uid=0,noforceuid,gid=0,noforcegid,addr=127.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
root at plattentest:~#

mounting the share with cache=loose, nostrictsync and noblocksend adds
another minute, adding "aio write behind = *" to the share definition
does next to nothing. neither cpu nor ram are bottlenecks.

thank you very much & with kind regards,
t.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20191106/e83f37c4/signature.sig>