samba - Feb 2023 - long delays with file enumeration & listing in large data storage environment

On 2/8/23 09:29, Matthias K?hne | Ellerhold Aktiengesellschaft via samba 
wrote:
> Directory listings from our MacOS clients took a big hit when we
> switched from AFP shares and SMB shares from our old Mac OS Server to
> the new debian fileserver with samba shares.
> 
> Especially opening big directories with AFP is blazingly fast, while in
> samba it takes forever. E. g. we have a directory with 37k
> subdirectories and 245k files in one level. Opening this dir on a Mac
> via a samba share take approx. 9 minutes. Our advice was to reduce the
> amount of files in a single directory and create cronjobs that
> automatically moves files into subdirectories.
Apple designed a few SMB2 protocol extensions that improve directory 
listing performance and Samba implements those in vfs_fruit. 
Additionally latest macos since a few releases has additinoal 
improvements that should improve listing regardless of these extensions.


-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20230208/8b652d06/OpenPGP_signature.sig>

Hello Ralph B?hme, Hello samba-people,

I *think* I have enabled all possible optimizations, but if you want to 
have a look if I forgot something I'd be thankful. Thanks! (Sorry for 
highjacking this thread - should I open a new one?)

Ive asked the users and on-prem IT-people for more details and this is 
what they tell me:

  * MacOS 10.15 Catalina with Intel chipset -> fast directory listing
  * MacOS 12+ with Intel or M1 chipset -> slow directory listing

Server side is the same Debian linux box with samba 4.16.

The smb.conf for this fileserver s:

[global]
 ??? workgroup = AD-ELLERHOLD
 ??? realm???? = AD.ELLERHOLD.LAN
 ??? dns proxy = no

 ??? load printers?????????? = no
 ??? printing??????????????? = bsd
 ??? printcap name?????????? = /dev/null
 ??? disable spoolss???????? = Yes
 ??? show add printer wizard = no

 ??? panic action = /usr/share/samba/panic-action %d
 ??? server role = member server
 ??? netbios name = FILESERVER

 ??? security = ADS

 ??? winbind offline logon = Yes

 ??? idmap config * : backend = tdb
 ??? idmap config * : range = 3000-7999

 ??? idmap config AD-ELLERHOLD:backend = ad
 ??? idmap config AD-ELLERHOLD:range = 10000-99999
 ??? idmap config AD-ELLERHOLD:schema_mode = rfc2307
 ??? idmap config AD-ELLERHOLD:unix_nss_info = yes
 ??? idmap config AD-ELLERHOLD:unix_primary_group = yes

 ??? username map??????????? = /etc/samba/user-map.conf

 ??? winbind refresh tickets = Yes
 ??? dedicated keytab file?? = /etc/krb5.keytab
 ??? kerberos method???????? = secrets and keytab

 ??? map acl inherit???????? = Yes
 ??? template shell????????? = /bin/bash
 ??? template homedir??????? = /home/AD-ELLERHOLD/%U

 ??? spotlight backend???? = elasticsearch
 ??? elasticsearch:address = localhost
 ??? elasticsearch:port??? = 9200
 ??? spotlight???????????? = Yes

 ??? vfs objects = catia fruit streams_xattr acl_xattr recycle
 ??? fruit:metadata = stream
 ??? fruit:veto_appledouble = no

 ??? veto files = /.DS_Store/
 ??? delete veto files = yes

 ??? recycle:repository = Papierkorb
 ??? recycle:keeptree = Yes
 ??? recycle:touch_mtime = Yes
 ??? recycle:directory_mode = 0770

 ??? server min protocol = NT1
 ??? ntlm auth?????????? = yes
 ??? lanman auth???????? = yes

 ??? client min protocol = NT1
 ??? client lanman auth? = yes

 ??? unix extensions = Yes

 ??? store dos attributes = No
 ??? map archive = No
 ??? map hidden = No
 ??? map readonly = No
 ??? map system = No


[homes]
 ??? browseable = no
 ??? path = /storage/homes/%U
 ??? read only = no

[My Share1]
 ??? force group = +AD-ELLERHOLD\fs_fileserver_my_share_1
 ??? inherit permissions = yes
 ??? path = /storage/my-share1
 ??? read only = no

Am 08.02.23 um 09:47 schrieb Ralph Boehme via samba:
> On 2/8/23 09:29, Matthias K?hne | Ellerhold Aktiengesellschaft via 
> samba wrote:
>> Directory listings from our MacOS clients took a big hit when we
>> switched from AFP shares and SMB shares from our old Mac OS Server to
>> the new debian fileserver with samba shares.
>>
>> Especially opening big directories with AFP is blazingly fast, while in
>> samba it takes forever. E. g. we have a directory with 37k
>> subdirectories and 245k files in one level. Opening this dir on a Mac
>> via a samba share take approx. 9 minutes. Our advice was to reduce the
>> amount of files in a single directory and create cronjobs that
>> automatically moves files into subdirectories.
>
> Apple designed a few SMB2 protocol extensions that improve directory 
> listing performance and Samba implements those in vfs_fruit. 
> Additionally latest macos since a few releases has additinoal 
> improvements that should improve listing regardless of these extensions.
>
>
> -slow
>
>
-- 
Senior Webentwickler
Datenschutzbeauftragter

Ellerhold Aktiengesellschaft
Friedrich-List-Str. 4
01445 Radebeul

Telefon: +49 (0) 351 83933-61
Web:www.ellerhold.de
Facebook:www.facebook.com/ellerhold.gruppe
Instagram:www.instagram.com/ellerhold.gruppe
Twitter:https://twitter.com/EllerholdGruppe

Amtsgericht Dresden / HRB 23769
Vorstand: Stephan Ellerhold, Maximilian Ellerhold
Vorsitzender des Aufsichtsrates: Frank Ellerhold


---Diese E-Mail und Ihre Anlagen enthalten vertrauliche Mitteilungen. Sollten
Sie nicht der beabsichtigte Adressat sein, so bitten wir Sie um Mitteilung und
um sofortiges l?schen dieser E-Mail und der Anlagen.

Unsere Hinweise zum Datenschutz finden Sie hier:
http://www.ellerhold.de/datenschutz/

This e-mail and its attachments are privileged and confidential. If you are not
the intended recipient, please notify us and immediately delete this e-mail and
its attachments.

You can find our privacy policy here: http://www.ellerhold.de/datenschutz/