Shenyue Chen
2023-Mar-09 18:16 UTC
[Samba] CIFS client mounts meta ops 30 times slow than server
Also from my observation, based on network packets and time measurements, it seems the CIFS clients haven't done any sort of cache. For example, doing a "ls" for 5 times on the same folder within a short period of time. The network packet sizes are almost equal every time and the time it takes is similar. Isn't there supposed to be some page cache or dentry cache that could accelerate this, under the loose cache policy? Or is this the expected behavior? Shenyue Chen <chensy1996 at gmail.com> ?2023?3?10??? 01:15???> Hi Samba users, > > We would like to use samba for re-exporting our other distributed file > system (BeeGFS). However, we found that the meta data operations are very > slow (dir stat, removal etc.). On SMB client the performance is more than > 30x slow than on the SMB server. We would expect some slowness but this is > too big a drop. > > This is the result on CIFS client > > $ sudo mdtest -b 10 -I 10 -L -z 2 > > mdtest-3.3.0 was launched with 1 total task(s) on 1 node(s) > Command line used: mdtest '-b' '10' '-I' '10' '-L' '-z' '2' > FS: 1341.0 TiB Used FS: 53.8% Inodes: 0.0 Mi Used Inodes: -nan% > > Nodemap: 1 > 1 tasks, 1000 files/directories > > SUMMARY rate: (of 1 iterations) > Operation Max Min Mean Std Dev > --------- --- --- ---- ------- > Directory creation : 8.431 8.431 8.431 0.000 > Directory stat : 34.470 34.470 34.470 0.000 > Directory removal : 11.401 11.401 11.401 0.000 > File creation : 15.151 15.151 15.151 0.000 > File stat : 26.313 26.313 26.313 0.000 > File read : 11.206 11.206 11.206 0.000 > File removal : 10.526 10.526 10.526 0.000 > Tree creation : 9.121 9.121 9.121 0.000 > Tree removal : 15.190 15.190 15.190 0.000 > -- finished at 02/17/2023 10:11:41 -- > > And this is the result of the same folder on SMB server. > > $ mdtest -b 10 -I 10 -L -z 2 -- started at 02/17/2023 10:26:11 -- > mdtest-3.3.0 was launched with 1 total task(s) on 1 node(s) Command line > used: mdtest '-b' '10' '-I' '10' '-L' '-z' '2' FS: 1341.0 TiB Used FS: > 54.0% Inodes: 0.0 Mi Used Inodes: -nan% Nodemap: 1 1 tasks, 1000 > files/directories SUMMARY rate: (of 1 iterations) Operation Max Min Mean > Std Dev --------- --- --- ---- ------- Directory creation : 290.328 290.328 > 290.328 0.000 Directory stat : 702.193 702.193 702.193 0.000 Directory > removal : 343.934 343.934 343.934 0.000 File creation : 330.556 330.556 > 330.556 0.000 File stat : 1368.871 1368.871 1368.871 0.000 File read : > 519.276 519.276 519.276 0.000 File removal : 454.345 454.345 454.345 0.000 > Tree creation : 314.333 314.333 314.333 0.000 Tree removal : 382.234 > 382.234 382.234 0.000 -- finished at 02/17/2023 10:26:28 -- > > Additionally, it seems our client does not have any kind of cache > available. I've tried many options: > > - cache=loose > - actimeo=3600 > - nostrictsync > - noperm > - fsc > - etc.... > > But none of the seems to have an impact. > > Our work load consists of mainly large number of small files. > > I know there many possible factors going on here. So I'm wondering if > there is any way we could profile/trace this to tune the performance. > > > Thanks! > > Info: > > Client linux kernel version: Ubuntu 22.04 5.15.0-60-generic > Server SMB version: 4.15.13+dfsg-0ubuntu1 > Mount options: <path> type cifs > (rw,relatime,vers=3.1.1,cache=strict,username=elvis_chen,uid=0,noforceuid,gid=0,noforcegid,addr=<ip>,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1) >
Ralph Boehme
2023-Mar-09 18:29 UTC
[Samba] CIFS client mounts meta ops 30 times slow than server
On 3/9/23 19:16, Shenyue Chen via samba wrote:> Also from my observation, based on network packets and time measurements, > it seems the CIFS clients haven't done any sort of cache. > > For example, doing a "ls" for 5 times on the same folder within a > short period of time. The network packet sizes are almost equal every time > and the time it takes is similar. > > Isn't there supposed to be some page cache or dentry cache that could > accelerate this, under the loose cache policy? Or is this the expected > behavior?not sure about how the client may cache directories on SMB mounts. There's a protocol feature that deals with this, Directory Leases, but unfortunately we don't implement this yet due to lack of funding. -slow -- Ralph Boehme, Samba Team https://samba.org/ SerNet Samba Team Lead https://sernet.de/en/team-samba SAMBA+ Samba packages https://samba.plus/ -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/samba/attachments/20230309/b6c0ed5d/OpenPGP_signature.sig>