Aurélien Aptel
2019-Feb-15 11:05 UTC
[Samba] 32 seconds vs 72 minutes -- expected performance difference?
Saurabh Nanda <saurabhnanda at gmail.com> writes:> 1) Why is the log saying `SMB2` everywhere? Shouldn't it be saying > `SMB3`?"SMB3" is mostly marketing, it inherits almost everything from SMB2 hence why it's often handled by SMB2 code. You will see this in Samba, Wireshark, Linux, and even Microsoft specification "MS-SMB2" which actually covers both versions.> 2) Is this normal -- fs/cifs/smb2misc.c: Calculated size 157 length 156 > mismatch mid 11907Could be a bug or miscalculated length + non critical warning, I also see this on master kernel. Either way I doubt it's slowing everything down.> 3) Is this normal -- fs/cifs/misc.c: Null buffer passed to > cifs_small_buf_releaseThat can be ignored> 4) Is this normal -- > fs/cifs/readdir.c: index not in buffer - could not findnext into > it > fs/cifs/readdir.c: could not find entry > 5) This definitely seems like some error, right? -- fs/cifs/smb2maperror.c: > Mapping SMB2 status code 0x80000006 to POSIX err -610x80000006 is STATUS_NO_MORE_FILES, returned by the server at the end of a listing. Can be ignored, its not added to the stats as it's the expected "error" for the last listing packet.> There is also another data point which _might_ be relevant. It _seems_ that > "systemd-resolved" is randomly crashing on the server , possibly because of > the mounted samba share. Caveat: I am not completely sure about the > correlation between samba mount and systemd-resolved crashes. Nevertheless, > I had already asked this question on StackExchange at > https://askubuntu.com/questions/1117842/systemd-resolved-crashing-with-failed-to-set-up-mount-namespacing-invalid-argu > if you feel that this might be relevant. Further, I can confirm that > systemd-resolved had NOT crashed when the following logs were obtained: > > ## OPERATION 1 - `ls debug.log` -- simply listing a single file. Does > anything seem wrong with this?This is going to be disapointing but we don't see the error that gets added in the stats in this log extract. Have you checked your samba logs as well? There might be useful information in there (might have to increase "log level" in smb.conf, it can go up to 10 but that will generate a *lot* of data). If you still don't want to share the network capture you should at least try to open it yourself in wireshark, it would save everyone time. You can find out the protocol negotiated by filtering for smb2.cmd == 0 And looking at the Response SMB2 > Negotiate Protocol Response > Dialect in the lower panel. Note that if you re-mounted during the capture or if some error happened the kernel might have tried to reconnect automatically and you will see multiple Req/Resp. You can filter for packets with errors like so: smb2 && smb2.nt_status != 0x80000006 && smb2.nt_status != 0 You can ignore "MORE_PROCESSING_REQUIRED" and ioctl error "FS_DRIVER_REQUIRED" as well. They will show up on every reconnection. Also, please attach files or disable word-wrapping when you send logs in the future. Cheers, -- Aurélien Aptel / SUSE Labs Samba Team GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
Saurabh Nanda
2019-Feb-16 02:43 UTC
[Samba] 32 seconds vs 72 minutes -- expected performance difference?
> > > 2) Is this normal -- fs/cifs/smb2misc.c: Calculated size 157 length 156 > > mismatch mid 11907 > > Could be a bug or miscalculated length + non critical warning, I also > see this on master kernel. Either way I doubt it's slowing everything down. >Should I file a bug for this?> ## OPERATION 1 - `ls debug.log` -- simply listing a single file. Does > > anything seem wrong with this? > > This is going to be disapointing but we don't see the error that gets > added in the stats in this log extract. Have you checked your samba logs > as well? There might be useful information in there (might have to > increase "log level" in smb.conf, it can go up to 10 but that will > generate a *lot* of data). > > If you still don't want to share the network capture you should at least > try to open it yourself in wireshark, it would save everyone time. >I tried looking at the network capture using `tcpdump -A -X` but wasn't able to understand anything. I tried installing wireshark on a throwaway cloud instance, but realised that it's a GUI program. Can you please help be with the network trace at https://www.dropbox.com/s/r8cn0qggrvmrpc3/dump.pcap?dl=0 ? It's capturing the network chatter for ~1min of `ls -lR` on the shared folder. Does this help in getting to the bottom of this problem?> Also, please attach files or disable word-wrapping when you send logs in > the future. >Ah - I thought most mailing lists scrub attachments. -- Saurabh.
Nick Howitt
2019-Feb-16 09:13 UTC
[Samba] 32 seconds vs 72 minutes -- expected performance difference?
On 16/02/2019 02:43, Saurabh Nanda via samba wrote:>>> 2) Is this normal -- fs/cifs/smb2misc.c: Calculated size 157 length 156 >>> mismatch mid 11907 >> Could be a bug or miscalculated length + non critical warning, I also >> see this on master kernel. Either way I doubt it's slowing everything down. >> > Should I file a bug for this? > > >> ## OPERATION 1 - `ls debug.log` -- simply listing a single file. Does >>> anything seem wrong with this? >> This is going to be disapointing but we don't see the error that gets >> added in the stats in this log extract. Have you checked your samba logs >> as well? There might be useful information in there (might have to >> increase "log level" in smb.conf, it can go up to 10 but that will >> generate a *lot* of data). >> >> If you still don't want to share the network capture you should at least >> try to open it yourself in wireshark, it would save everyone time. >> > I tried looking at the network capture using `tcpdump -A -X` but wasn't > able to understand anything. I tried installing wireshark on a throwaway > cloud instance, but realised that it's a GUI program. Can you please help > be with the network trace at > https://www.dropbox.com/s/r8cn0qggrvmrpc3/dump.pcap?dl=0 ? It's capturing > the network chatter for ~1min of `ls -lR` on the shared folder. Does this > help in getting to the bottom of this problem? > > >> Also, please attach files or disable word-wrapping when you send logs in >> the future. >> > Ah - I thought most mailing lists scrub attachments. > > -- Saurabh.This is a complete lateral jump. In the El/Centos line of distros I've seen a lot of issues caused by using the r8169 driver on RTL8111/8168/8411 cards including things like DNS failures and network slowdowns and other more exotic issues. The card performs better (in EL/Centos) with the r8168 driver which does not come built into the kernel, unless Ubuntu has unified the two drivers under the r8169 banner. You can do a quick check with: lspci -k | grep Eth -A 3 Please ignore this post if it is totally irrelevant. Nick
Saurabh Nanda
2019-Feb-18 10:43 UTC
[Samba] 32 seconds vs 72 minutes -- expected performance difference?
> > I tried looking at the network capture using `tcpdump -A -X` but wasn't > able to understand anything. I tried installing wireshark on a throwaway > cloud instance, but realised that it's a GUI program. Can you please help > be with the network trace at > https://www.dropbox.com/s/r8cn0qggrvmrpc3/dump.pcap?dl=0 ? It's capturing > the network chatter for ~1min of `ls -lR` on the shared folder. Does this > help in getting to the bottom of this problem? >Help in analysing the tcpdump to get to the bottom of this perf issue would be appreciated. -- Saurabh.
Possibly Parallel Threads
- 32 seconds vs 72 minutes -- expected performance difference?
- 32 seconds vs 72 minutes -- expected performance difference?
- 32 seconds vs 72 minutes -- expected performance difference?
- 32 seconds vs 72 minutes -- expected performance difference?
- Problem to many connection