samba - Mar 2023 - CIFS client mounts meta ops 30 times slow than server

Hi Samba users,

We would like to use samba for re-exporting our other distributed file
system (BeeGFS). However, we found that the meta data operations are very
slow (dir stat, removal etc.). On SMB client the performance is more than
30x slow than on the SMB server. We would expect some slowness but this is
too big a drop.

This is the result on CIFS client

$ sudo mdtest -b 10 -I 10 -L -z 2

mdtest-3.3.0 was launched with 1 total task(s) on 1 node(s)
Command line used: mdtest '-b' '10' '-I' '10'
'-L' '-z' '2'
FS: 1341.0 TiB   Used FS: 53.8%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1
1 tasks, 1000 files/directories

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean
    Std Dev
   ---------                      ---            ---           ----
    -------
   Directory creation        :          8.431          8.431
8.431          0.000
   Directory stat            :         34.470         34.470
34.470          0.000
   Directory removal         :         11.401         11.401
11.401          0.000
   File creation             :         15.151         15.151
15.151          0.000
   File stat                 :         26.313         26.313
26.313          0.000
   File read                 :         11.206         11.206
11.206          0.000
   File removal              :         10.526         10.526
10.526          0.000
   Tree creation             :          9.121          9.121
9.121          0.000
   Tree removal              :         15.190         15.190
15.190          0.000
-- finished at 02/17/2023 10:11:41 --

And this is the result of the same folder on SMB server.

$ mdtest -b 10 -I 10 -L -z 2 -- started at 02/17/2023 10:26:11 --
mdtest-3.3.0 was launched with 1 total task(s) on 1 node(s) Command line
used: mdtest '-b' '10' '-I' '10' '-L'
'-z' '2' FS: 1341.0 TiB Used FS:
54.0% Inodes: 0.0 Mi Used Inodes: -nan% Nodemap: 1 1 tasks, 1000
files/directories SUMMARY rate: (of 1 iterations) Operation Max Min Mean
Std Dev --------- --- --- ---- ------- Directory creation : 290.328 290.328
290.328 0.000 Directory stat : 702.193 702.193 702.193 0.000 Directory
removal : 343.934 343.934 343.934 0.000 File creation : 330.556 330.556
330.556 0.000 File stat : 1368.871 1368.871 1368.871 0.000 File read :
519.276 519.276 519.276 0.000 File removal : 454.345 454.345 454.345 0.000
Tree creation : 314.333 314.333 314.333 0.000 Tree removal : 382.234
382.234 382.234 0.000 -- finished at 02/17/2023 10:26:28 --

Additionally, it seems our client does not have any kind of cache
available. I've tried many options:

   - cache=loose
   - actimeo=3600
   - nostrictsync
   - noperm
   - fsc
   - etc....

But none of the seems to have an impact.

Our work load consists of mainly large number of small files.

I know there many possible factors going on here. So I'm wondering if there
is any way we could profile/trace this to tune the performance.


Thanks!

Info:

Client linux kernel version: Ubuntu 22.04 5.15.0-60-generic
Server SMB version: 4.15.13+dfsg-0ubuntu1
Mount options: <path> type cifs
(rw,relatime,vers=3.1.1,cache=strict,username=elvis_chen,uid=0,noforceuid,gid=0,noforcegid,addr=<ip>,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

On Fri, Mar 10, 2023 at 01:15:41AM +0800, Shenyue Chen via samba
wrote:
>Hi Samba users,
>
>We would like to use samba for re-exporting our other distributed file
>system (BeeGFS). However, we found that the meta data operations are very
>slow (dir stat, removal etc.). On SMB client the performance is more than
>30x slow than on the SMB server. We would expect some slowness but this is
>too big a drop.
>
>This is the result on CIFS client
>
>$ sudo mdtest -b 10 -I 10 -L -z 2
>
>mdtest-3.3.0 was launched with 1 total task(s) on 1 node(s)
>Command line used: mdtest '-b' '10' '-I'
'10' '-L' '-z' '2'
>FS: 1341.0 TiB   Used FS: 53.8%   Inodes: 0.0 Mi   Used Inodes: -nan%
Can you be a little clearer on your setup please ?

I'm assuming you are mounting the BeeGFS filesystem
on the Samba server, and then re-exporting that via
Samba to the Linux cifs client. Is that the case ?

On 3/9/23 18:15, Shenyue Chen via samba wrote:
> Server SMB version: 4.15.13+dfsg-0ubuntu1
- 30 times slower is actually not that bad for metadata heavy workloads
- please use Samba 4.18, that has considerable improved metadata performance
- research whether beegfs supports file case lookup that could be used 
to implement a beegfs specific VFS module (there isn't one yet, but we 
have VFS modules for other filesystems that support this eg gpfs and 
glusterfs)

Cheers!
-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba
SAMBA+ Samba packages                   https://samba.plus/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20230309/0cc682a7/OpenPGP_signature.sig>

Also from my observation, based on network packets and time measurements,
it seems the CIFS clients haven't done any sort of cache.

For example, doing a "ls" for 5 times on the same folder within a
short period of time. The network packet sizes are almost equal every time
and the time it takes is similar.

Isn't there supposed to be some page cache or dentry cache that could
accelerate this, under the loose cache policy? Or is this the expected
behavior?

Shenyue Chen <chensy1996 at gmail.com> ?2023?3?10??? 01:15???
> Hi Samba users,
>
> We would like to use samba for re-exporting our other distributed file
> system (BeeGFS). However, we found that the meta data operations are very
> slow (dir stat, removal etc.). On SMB client the performance is more than
> 30x slow than on the SMB server. We would expect some slowness but this is
> too big a drop.
>
> This is the result on CIFS client
>
> $ sudo mdtest -b 10 -I 10 -L -z 2
>
> mdtest-3.3.0 was launched with 1 total task(s) on 1 node(s)
> Command line used: mdtest '-b' '10' '-I'
'10' '-L' '-z' '2'
> FS: 1341.0 TiB   Used FS: 53.8%   Inodes: 0.0 Mi   Used Inodes: -nan%
>
> Nodemap: 1
> 1 tasks, 1000 files/directories
>
> SUMMARY rate: (of 1 iterations)
>    Operation                      Max            Min           Mean       
Std Dev
>    ---------                      ---            ---           ----       
-------
>    Directory creation        :          8.431          8.431          8.431
0.000
>    Directory stat            :         34.470         34.470         34.470
0.000
>    Directory removal         :         11.401         11.401         11.401
0.000
>    File creation             :         15.151         15.151         15.151
0.000
>    File stat                 :         26.313         26.313         26.313
0.000
>    File read                 :         11.206         11.206         11.206
0.000
>    File removal              :         10.526         10.526         10.526
0.000
>    Tree creation             :          9.121          9.121          9.121
0.000
>    Tree removal              :         15.190         15.190         15.190
0.000
> -- finished at 02/17/2023 10:11:41 --
>
> And this is the result of the same folder on SMB server.
>
> $ mdtest -b 10 -I 10 -L -z 2 -- started at 02/17/2023 10:26:11 --
> mdtest-3.3.0 was launched with 1 total task(s) on 1 node(s) Command line
> used: mdtest '-b' '10' '-I' '10'
'-L' '-z' '2' FS: 1341.0 TiB Used FS:
> 54.0% Inodes: 0.0 Mi Used Inodes: -nan% Nodemap: 1 1 tasks, 1000
> files/directories SUMMARY rate: (of 1 iterations) Operation Max Min Mean
> Std Dev --------- --- --- ---- ------- Directory creation : 290.328 290.328
> 290.328 0.000 Directory stat : 702.193 702.193 702.193 0.000 Directory
> removal : 343.934 343.934 343.934 0.000 File creation : 330.556 330.556
> 330.556 0.000 File stat : 1368.871 1368.871 1368.871 0.000 File read :
> 519.276 519.276 519.276 0.000 File removal : 454.345 454.345 454.345 0.000
> Tree creation : 314.333 314.333 314.333 0.000 Tree removal : 382.234
> 382.234 382.234 0.000 -- finished at 02/17/2023 10:26:28 --
>
> Additionally, it seems our client does not have any kind of cache
> available. I've tried many options:
>
>    - cache=loose
>    - actimeo=3600
>    - nostrictsync
>    - noperm
>    - fsc
>    - etc....
>
> But none of the seems to have an impact.
>
> Our work load consists of mainly large number of small files.
>
> I know there many possible factors going on here. So I'm wondering if
> there is any way we could profile/trace this to tune the performance.
>
>
> Thanks!
>
> Info:
>
> Client linux kernel version: Ubuntu 22.04 5.15.0-60-generic
> Server SMB version: 4.15.13+dfsg-0ubuntu1
> Mount options: <path> type cifs
>
(rw,relatime,vers=3.1.1,cache=strict,username=elvis_chen,uid=0,noforceuid,gid=0,noforcegid,addr=<ip>,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
>