samba - Feb 2019 - 32 seconds vs 72 minutes -- expected performance difference?

Saurabh Nanda <saurabhnanda at gmail.com> writes:
> 1) Why is the log saying `SMB2` everywhere? Shouldn't it be saying
> `SMB3`?
"SMB3" is mostly marketing, it inherits almost everything from SMB2
hence
why it's often handled by SMB2 code. You will see this in Samba,
Wireshark, Linux, and even Microsoft specification "MS-SMB2" which
actually covers both versions.
> 2) Is this normal -- fs/cifs/smb2misc.c: Calculated size 157 length 156
> mismatch mid 11907
Could be a bug or miscalculated length + non critical warning, I also
see this on master kernel. Either way I doubt it's slowing everything down.
> 3) Is this normal -- fs/cifs/misc.c: Null buffer passed to
> cifs_small_buf_release
That can be ignored
> 4) Is this normal --
>           fs/cifs/readdir.c: index not in buffer - could not findnext into
> it
>           fs/cifs/readdir.c: could not find entry
> 5) This definitely seems like some error, right? -- fs/cifs/smb2maperror.c:
> Mapping SMB2 status code 0x80000006 to POSIX err -61
0x80000006 is STATUS_NO_MORE_FILES, returned by the server at the end of
a listing. Can be ignored, its not added to the stats as it's the
expected "error" for the last listing packet.
> There is also another data point which _might_ be relevant. It _seems_ that
> "systemd-resolved" is randomly crashing on the server , possibly
because of
> the mounted samba share. Caveat: I am not completely sure about the
> correlation between samba mount and systemd-resolved crashes. Nevertheless,
> I had already asked this question on StackExchange at
>
https://askubuntu.com/questions/1117842/systemd-resolved-crashing-with-failed-to-set-up-mount-namespacing-invalid-argu
> if you feel that this might be relevant. Further, I can confirm that
> systemd-resolved had NOT crashed when the following logs were obtained:
>
> ## OPERATION 1 -  `ls debug.log` -- simply listing a single file. Does
> anything seem wrong with this?
This is going to be disapointing but we don't see the error that gets
added in the stats in this log extract. Have you checked your samba logs
as well? There might be useful information in there (might have to
increase "log level" in smb.conf, it can go up to 10 but that will
generate a *lot* of data).

If you still don't want to share the network capture you should at least
try to open it yourself in wireshark, it would save everyone time.

You can find out the protocol negotiated by filtering for

  smb2.cmd == 0

And looking at the Response SMB2 > Negotiate Protocol Response > Dialect
in the lower panel.

Note that if you re-mounted during the capture or if some error happened
the kernel might have tried to reconnect automatically and you will see
multiple Req/Resp.

You can filter for packets with errors like so:

  smb2 && smb2.nt_status != 0x80000006 && smb2.nt_status != 0

You can ignore "MORE_PROCESSING_REQUIRED" and ioctl error
"FS_DRIVER_REQUIRED" as well. They will show up on every reconnection.

Also, please attach files or disable word-wrapping when you send logs in
the future.

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

>
> > 2) Is this normal -- fs/cifs/smb2misc.c: Calculated size 157 length
156
> > mismatch mid 11907
>
> Could be a bug or miscalculated length + non critical warning, I also
> see this on master kernel. Either way I doubt it's slowing everything
down.
>
Should I file a bug for this?

> ## OPERATION 1 -  `ls debug.log` -- simply listing a single file. Does
> > anything seem wrong with this?
>
> This is going to be disapointing but we don't see the error that gets
> added in the stats in this log extract. Have you checked your samba logs
> as well? There might be useful information in there (might have to
> increase "log level" in smb.conf, it can go up to 10 but that
will
> generate a *lot* of data).
>
> If you still don't want to share the network capture you should at
least
> try to open it yourself in wireshark, it would save everyone time.
>
I tried looking at the network capture using `tcpdump -A -X` but wasn't
able to understand anything. I tried installing wireshark on a throwaway
cloud instance, but realised that it's a GUI program. Can you please help
be with the network trace at
https://www.dropbox.com/s/r8cn0qggrvmrpc3/dump.pcap?dl=0 ? It's capturing
the network chatter for ~1min of `ls -lR` on the shared folder. Does this
help in getting to the bottom of this problem?

> Also, please attach files or disable word-wrapping when you send logs in
> the future.
>
Ah - I thought most mailing lists scrub attachments.

-- Saurabh.

On 16/02/2019 02:43, Saurabh Nanda via samba wrote:
>>> 2) Is this normal -- fs/cifs/smb2misc.c: Calculated size 157 length
156
>>> mismatch mid 11907
>> Could be a bug or miscalculated length + non critical warning, I also
>> see this on master kernel. Either way I doubt it's slowing
everything down.
>>
> Should I file a bug for this?
>
>
>> ## OPERATION 1 -  `ls debug.log` -- simply listing a single file. Does
>>> anything seem wrong with this?
>> This is going to be disapointing but we don't see the error that
gets
>> added in the stats in this log extract. Have you checked your samba
logs
>> as well? There might be useful information in there (might have to
>> increase "log level" in smb.conf, it can go up to 10 but that
will
>> generate a *lot* of data).
>>
>> If you still don't want to share the network capture you should at
least
>> try to open it yourself in wireshark, it would save everyone time.
>>
> I tried looking at the network capture using `tcpdump -A -X` but wasn't
> able to understand anything. I tried installing wireshark on a throwaway
> cloud instance, but realised that it's a GUI program. Can you please
help
> be with the network trace at
> https://www.dropbox.com/s/r8cn0qggrvmrpc3/dump.pcap?dl=0 ? It's
capturing
> the network chatter for ~1min of `ls -lR` on the shared folder. Does this
> help in getting to the bottom of this problem?
>
>
>> Also, please attach files or disable word-wrapping when you send logs
in
>> the future.
>>
> Ah - I thought most mailing lists scrub attachments.
>
> -- Saurabh.
This is a complete lateral jump. In the El/Centos line of distros I've 
seen a lot of issues caused by using the r8169 driver on 
RTL8111/8168/8411 cards including things like DNS failures and network 
slowdowns and other more exotic issues. The card performs better (in 
EL/Centos) with the r8168 driver which does not come built into the 
kernel, unless Ubuntu has unified the two drivers under the r8169 
banner. You can do a quick check with:
lspci -k | grep Eth -A 3

Please ignore this post if it is totally irrelevant.

Nick

>
> I tried looking at the network capture using `tcpdump -A -X` but wasn't
> able to understand anything. I tried installing wireshark on a throwaway
> cloud instance, but realised that it's a GUI program. Can you please
help
> be with the network trace at
> https://www.dropbox.com/s/r8cn0qggrvmrpc3/dump.pcap?dl=0 ? It's
capturing
> the network chatter for ~1min of `ls -lR` on the shared folder. Does this
> help in getting to the bottom of this problem?
>
Help in analysing the tcpdump to get to the bottom of this perf issue would
be appreciated.

-- Saurabh.