Dāvis Mosāns
2018-Mar-17 20:34 UTC
[Samba] Terrible share access performance (v.4.8 and current master branch)
Hello! When I'm using qBittorrent [1] on Windows 10 with download location set to remote Samba share on Arch Linux then it severely affects all shares on that host, unrelated to disk where qBittorrent is actually writing. On Arch Linux that smbd process is using 100% of one CPU core time and seems it's blocking because of lseek calls. strace shows full of lseek(47, 1420820480, SEEK_DATA) = 1421344768 lseek(47, 1421344768, SEEK_HOLE) = 1423441920 lseek(47, 1423441920, SEEK_DATA) = 1423966208 lseek(47, 1423966208, SEEK_HOLE) = 1425014784 lseek(47, 1425014784, SEEK_DATA) = 1425539072 lseek(47, 1425539072, SEEK_HOLE) = 1426063360 lseek(47, 1426063360, SEEK_DATA) = 1426587648 lseek(47, 1426587648, SEEK_HOLE) = 1427111936 lseek(47, 1427111936, SEEK_DATA) = 1427636224 lseek(47, 1427636224, SEEK_HOLE) = 1429733376 lseek(47, 1429733376, SEEK_DATA) = 1430781952 lseek(47, 1430781952, SEEK_HOLE) = 1431191552 lseek(47, 1431191552, SEEK_DATA) = 1432879104 lseek(47, 1432879104, SEEK_HOLE) = 1433403392 lseek(47, 1433403392, SEEK_DATA) = 1433927680 lseek(47, 1433927680, SEEK_HOLE) = 1434402816 lseek(47, 1434402816, SEEK_DATA) = 1434451968 lseek(47, 1434451968, SEEK_HOLE) = 1434976256 lseek(47, 1434976256, SEEK_DATA) = 1435500544 lseek(47, 1435500544, SEEK_HOLE) = 1436024832 lseek(47, 1436024832, SEEK_DATA) = 1438121984 lseek(47, 1438121984, SEEK_HOLE) = 1438646272 lseek(47, 1438646272, SEEK_DATA) = 1439170560 lseek(47, 1439170560, SEEK_HOLE) = 1439694848 lseek(47, 1439694848, SEEK_DATA) = 1440743424 lseek(47, 1440743424, SEEK_HOLE) = 1442316288 lseek(47, 1442316288, SEEK_DATA) = 1443889152 ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 94.42 29.247198 449 65142 lseek 1.13 0.350504 82 4285 writev 0.60 0.185996 10 19194 geteuid 0.60 0.185819 10 19194 getegid 0.43 0.134625 15 8912 readv 0.40 0.124905 12 10353 24 stat 0.36 0.111773 17 6773 4183 getxattr 0.33 0.102632 12 8922 setresuid 0.31 0.097227 11 8922 setresgid 0.27 0.083860 13 6659 epoll_pwait 0.24 0.073222 15 4803 3 futex 0.23 0.071162 11 6671 getpid 0.21 0.066586 13 5136 setgroups 0.14 0.044679 11 4057 read 0.11 0.035451 14 2476 chdir 0.08 0.026077 10 2700 getgroups 0.07 0.021106 12 1768 fcntl 0.01 0.003112 33 93 clone 0.01 0.002433 10 233 fstat 0.01 0.001825 10 186 rt_sigprocmask 0.00 0.001099 15 75 close 0.00 0.000780 16 50 openat 0.00 0.000773 13 60 lstat 0.00 0.000637 12 55 mprotect 0.00 0.000598 11 55 mmap 0.00 0.000474 13 36 newfstatat 0.00 0.000453 21 22 getdents 0.00 0.000377 29 13 sendmsg 0.00 0.000270 14 20 socket 0.00 0.000270 11 24 getcwd 0.00 0.000246 12 20 10 connect 0.00 0.000149 14 11 flock 0.00 0.000142 24 6 utimensat 0.00 0.000137 11 12 pread64 0.00 0.000129 11 12 kill 0.00 0.000117 15 8 open 0.00 0.000019 19 1 brk ------ ----------- ----------- --------- --------- ---------------- 100.00 30.976862 186959 4220 total backtrace 0x00007fe85731114d in lseek64 () from /usr/lib/libpthread.so.0 Thread 1 (Thread 0x7fe8576fd840 (LWP 2894)): #0 0x00007fe85731114d in lseek64 () from /usr/lib/libpthread.so.0 #1 0x00007fe856a0abdc in vfswrap_lseek () from /usr/lib/samba/libsmbd-base-samba4.so #2 0x00007fe856ad656f in smb2_ioctl_filesys () from /usr/lib/samba/libsmbd-base-samba4.so #3 0x00007fe856ad53ec in smbd_smb2_request_process_ioctl () from /usr/lib/samba/libsmbd-base-samba4.so #4 0x00007fe856ac6f73 in smbd_smb2_request_dispatch () from /usr/lib/samba/libsmbd-base-samba4.so #5 0x00007fe856ac7c40 in smbd_smb2_connection_handler () from /usr/lib/samba/libsmbd-base-samba4.so #6 0x00007fe85358241f in ?? () from /usr/lib/libtevent.so.0 #7 0x00007fe853580839 in ?? () from /usr/lib/libtevent.so.0 #8 0x00007fe85357cb9e in _tevent_loop_once () from /usr/lib/libtevent.so.0 #9 0x00007fe85357cdbc in tevent_common_loop_wait () from /usr/lib/libtevent.so.0 #10 0x00007fe8535807d9 in ?? () from /usr/lib/libtevent.so.0 #11 0x00007fe856ab7998 in smbd_process () from /usr/lib/samba/libsmbd-base-samba4.so #12 0x00005560528caf9c in smbd_accept_connection () #13 0x00007fe85358241f in ?? () from /usr/lib/libtevent.so.0 #14 0x00007fe853580839 in ?? () from /usr/lib/libtevent.so.0 #15 0x00007fe85357cb9e in _tevent_loop_once () from /usr/lib/libtevent.so.0 #16 0x00007fe85357cdbc in tevent_common_loop_wait () from /usr/lib/libtevent.so.0 #17 0x00007fe8535807d9 in ?? () from /usr/lib/libtevent.so.0 #18 0x00005560528c60d6 in main () I made a Ruby script (see attachment) to benchmark other share's access time which is on different partition and physical disk than one to which qBittorrent is writing. When qBittorrent is downloading then for that other share on average real time is 0.22 seconds but when it's not then it's 0.033 seconds which is huge difference and very noticeable. shares are hosted on Arch Linux with kernel 4.15.10 and just compiled samba from git master branch (commit a6054c01c29c2507e0d5a6aa110fee4fd5c5eeb9) using BTRFS filesystem for all shares but different partitions for qBittorrent writing and other shares. client is Windows 10 (1709, build 16299.309) qBittorrent v4.0.4 [1] https://www.qbittorrent.org/
Possibly Parallel Threads
Wisdom of the Ancients
