samba - Feb 2023 - long delays with file enumeration & listing in large data storage environment

On Tue, Feb 07, 2023 at 08:07:46PM -0500, CaptainTrips28 via samba
wrote:
>
>- Samba versions 4.13.3 (rhel 8.4) and 4.15.5 (rhel 8.6) have both been
>tried with identical outcome, all pulled from RH satellite repos)
>
>- We've tried with stigs both applied and unapplied, as well as
folder/file
>encryption both on and off (no difference in performance).
>
>- PBIS/ADBridge was tried with both versions 22.2.x and 22.3.x (latest)
>
>- Testing share access from Windows 2019 Server Datacenter (Build 17763).
>Our user VDA sessions would also being accessing the shares from the same.
>
>- Production environment is enterprise scale; hundreds of users/thousands
>of folders/millions of files
>
>- Share enumeration speed is the same if accessing by either mapped drive,
>file explorer // access, or symbolically linking to the share
>
>Any suggestions/recommendations on how to reduce or eliminate enumeration
>and listing times for these shares is certainly appreciated.
You may be running into this (from the Samba 4.17.x release notes).

NEW FEATURES/CHANGES
===================
SMB Server performance improvements
-----------------------------------

The security improvements in recent releases
(4.13, 4.14, 4.15, 4.16), mainly as protection against symlink races,
caused performance regressions for meta data heavy workloads.

With 4.17 the situation improved a lot again:

- Pathnames given by a client are devided into dirname and basename.
   The amount of syscalls to validate dirnames is reduced to 2 syscalls
   (openat, close) per component. On modern Linux kernels (>= 5.6) smbd
   makes use of the openat2() syscall with RESOLVE_NO_SYMLINKS,
   in order to just use 2 syscalls (openat2, close) for the whole dirname.

- Contended path based operations used to generate a lot of unsolicited
   wakeup events causing thundering herd problems, which lead to masive
   latencies for some clients. These events are now avoided in order
   to provide stable latencies and much higher throughput of open/close
   operations.

I would suggest you try with Samba 4.17.latest to see if you
find an improvement.

Thanks Jeremy, will give it a shot in the morning and update asap.

On Tue, Feb 7, 2023, 8:42 PM Jeremy Allison <jra at samba.org> wrote:
> On Tue, Feb 07, 2023 at 08:07:46PM -0500, CaptainTrips28 via samba wrote:
> >
> >- Samba versions 4.13.3 (rhel 8.4) and 4.15.5 (rhel 8.6) have both been
> >tried with identical outcome, all pulled from RH satellite repos)
> >
> >- We've tried with stigs both applied and unapplied, as well as
> folder/file
> >encryption both on and off (no difference in performance).
> >
> >- PBIS/ADBridge was tried with both versions 22.2.x and 22.3.x (latest)
> >
> >- Testing share access from Windows 2019 Server Datacenter (Build
17763).
> >Our user VDA sessions would also being accessing the shares from the
same.
> >
> >- Production environment is enterprise scale; hundreds of
users/thousands
> >of folders/millions of files
> >
> >- Share enumeration speed is the same if accessing by either mapped
drive,
> >file explorer // access, or symbolically linking to the share
> >
> >Any suggestions/recommendations on how to reduce or eliminate
enumeration
> >and listing times for these shares is certainly appreciated.
>
> You may be running into this (from the Samba 4.17.x release notes).
>
> NEW FEATURES/CHANGES
> ===================>
> SMB Server performance improvements
> -----------------------------------
>
> The security improvements in recent releases
> (4.13, 4.14, 4.15, 4.16), mainly as protection against symlink races,
> caused performance regressions for meta data heavy workloads.
>
> With 4.17 the situation improved a lot again:
>
> - Pathnames given by a client are devided into dirname and basename.
>    The amount of syscalls to validate dirnames is reduced to 2 syscalls
>    (openat, close) per component. On modern Linux kernels (>= 5.6) smbd
>    makes use of the openat2() syscall with RESOLVE_NO_SYMLINKS,
>    in order to just use 2 syscalls (openat2, close) for the whole dirname.
>
> - Contended path based operations used to generate a lot of unsolicited
>    wakeup events causing thundering herd problems, which lead to masive
>    latencies for some clients. These events are now avoided in order
>    to provide stable latencies and much higher throughput of open/close
>    operations.
>
> I would suggest you try with Samba 4.17.latest to see if you
> find an improvement.
>

On 2/8/23 02:42, Jeremy Allison via samba wrote:
> I would suggest you try with Samba 4.17.latest to see if you
> find an improvement.
while this is generally a good advice :) I don't think this will help in 
this case. Directory enumeration didn't really see any changes and the 
OP mentions that enumeration was fast before joining to the domain. This 
is quite unexpected and this is the point where I would start investigating.

-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20230208/f8e7f9f2/OpenPGP_signature.sig>