samba - Nov 2019 - samba performance when writing lots of small files

hi Stefan / all,

resurrecting this thread ?

On 9/26/19 9:19 PM, Stefan Kania via samba wrote:
> try to use a newer version of Samba 4.7 is already outdated. The Problem
> the smb-protocol and not Samba it self. As Steve French mentioned on the
> last SambaXP it will be much better with Linux 5.x. So maybe you try a
> new kernel
root at plattentest:~# uname -a
Linux plattentest 5.3.0-18-generic #19-Ubuntu SMP Tue Oct 8 20:14:06 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
root at plattentest:~# dpkg -l samba
[?]
ii  samba          2:4.10.7+dfsg-0ubuntu3 amd64        SMB/CIFS file,
print, and login server for Unix
root at plattentest:~# time for s in $(seq 0 9999); do echo $s >
/tubfs/test/10.000/test-$s; done # this directory is a cephfs mount

real	0m5.921s
user	0m0.515s
sys	0m0.861s
root at plattentest:~# time for s in $(seq 0 9999); do echo $s >
/mnt/10.000-cifs/test-$s; done # this is the former re-exported via samba

real    13m27.317s
user    0m0.881s
sys     0m6.326s
root at plattentest:~# cat /etc/samba/smb.conf
[global]
        kernel change notify = no
        change notify = no
        read raw = Yes
        write raw = Yes
[test]
        path=/tubfs/test/
        guest ok = yes
        force user = root
        writeable = yes
        aio read size = 1
        aio write size = 1
root at plattentest:~# mount | grep test
//localhost/test on /mnt type cifs
(rw,relatime,vers=3.1.1,sec=none,cache=strict,uid=0,noforceuid,gid=0,noforcegid,addr=127.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
root at plattentest:~#

mounting the share with cache=loose, nostrictsync and noblocksend adds
another minute, adding "aio write behind = *" to the share definition
does next to nothing. neither cpu nor ram are bottlenecks.

thank you very much & with kind regards,
t.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20191106/e83f37c4/signature.sig>

On Wed, Nov 06, 2019 at 03:18:24PM +0100, thoralf schulze via samba
wrote:
> hi Stefan / all,
> 
> resurrecting this thread ?
> 
> On 9/26/19 9:19 PM, Stefan Kania via samba wrote:
> > try to use a newer version of Samba 4.7 is already outdated. The
Problem
> > the smb-protocol and not Samba it self. As Steve French mentioned on
the
> > last SambaXP it will be much better with Linux 5.x. So maybe you try a
> > new kernel
> 
> root at plattentest:~# uname -a
> Linux plattentest 5.3.0-18-generic #19-Ubuntu SMP Tue Oct 8 20:14:06 UTC
> 2019 x86_64 x86_64 x86_64 GNU/Linux
> root at plattentest:~# dpkg -l samba
> [?]
> ii  samba          2:4.10.7+dfsg-0ubuntu3 amd64        SMB/CIFS file,
> print, and login server for Unix
> root at plattentest:~# time for s in $(seq 0 9999); do echo $s >
> /tubfs/test/10.000/test-$s; done # this directory is a cephfs mount
> 
> real	0m5.921s
> user	0m0.515s
> sys	0m0.861s
> root at plattentest:~# time for s in $(seq 0 9999); do echo $s >
> /mnt/10.000-cifs/test-$s; done # this is the former re-exported via samba
> 
> real    13m27.317s
> user    0m0.881s
> sys     0m6.326s
> root at plattentest:~# cat /etc/samba/smb.conf
> [global]
>         kernel change notify = no
>         change notify = no
>         read raw = Yes
>         write raw = Yes
> [test]
>         path=/tubfs/test/
>         guest ok = yes
>         force user = root
>         writeable = yes
>         aio read size = 1
>         aio write size = 1
> root at plattentest:~# mount | grep test
> //localhost/test on /mnt type cifs
>
(rw,relatime,vers=3.1.1,sec=none,cache=strict,uid=0,noforceuid,gid=0,noforcegid,addr=127.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
> root at plattentest:~#
> 
> mounting the share with cache=loose, nostrictsync and noblocksend adds
> another minute, adding "aio write behind = *" to the share
definition
> does next to nothing. neither cpu nor ram are bottlenecks.
This is re-exporting via ceph whilst creating 1000 files,
yes ? What timings do you get when doing this via Samba
onto a local ext4/xfs/btrfs/zfs filesystem ?

hi jeremy / all,

On 11/6/19 10:39 PM, Jeremy Allison wrote:
> This is re-exporting via ceph whilst creating 1000 files,
> yes ? What timings do you get when doing this via Samba
> onto a local ext4/xfs/btrfs/zfs filesystem ?
yes, creating 10k small files. doing the same on a local ssd, formatted
with an ext4 fs without any special options:

root at plattentest:/mnt-ssd/os# time for s in $(seq 0 9999); do echo $s >
test-$s; done

real	0m0.376s
user	0m0.130s
sys	0m0.246s
root at plattentest:/mnt-ssd/os#

and on the very same ssd, exported via cifs and re-mounted locally,
samba and share config are identical to the ceph test:

root at plattentest:/mnt-ssd-cifs/cifs# time for s in $(seq 0 9999); do
echo $s > test-$s; done

real	0m43.228s
user	0m0.445s
sys	0m2.692s
root at plattentest:/mnt-ssd-cifs/cifs#

that's more in line with what one would expect :-)
do you have any thoughts on why the same test with the ceph share is
more than an order of magnitude slower? afaik the ceph client should
block until all data has been written, no matter if it is being fed by
echo or samba ? the cephfs is mounted via the kernel driver, if that
matters.

thank you very much & with kind regards,
t.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20191107/b0925c40/signature.sig>