samba - Feb 2019 - 32 seconds vs 72 minutes -- expected performance difference?

>
> Unless you upload a network capture of you mounting and doing the ls -lR
> on the client it's hard to say what really goes on. I understand you
> might not want to make it public.. but if you do
>
This is the last thing I'll try after I've exhausted all the other
options.


How are you mounting your share (which mount options)?
>
Something weird is going on with the mount options. I hadn't used vers= in
my fstab earlier, and noticed the following in syslog:

    No dialect specified on mount. Default has changed to a more secure
dialect,
    SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure
SMB1
    dialect to access old servers which do not support SMB3 (or SMB2.1)
specify
    vers=1.0 on mount.

Which implies that the server & client auto-negotiated a protocol version
greater than SMB2.1, right? However, to be sure, I manually specified versin
fstab, but something strange happened. While `man mount.cifs` claims
that the following are allowed -- 1.0, 2.0, 2.1, 3.0, 3.1.1 (or 3.11) --
few of them failed with strange errors:

# mount -t cifs -o
rw,username=myuser,password=[REDACTED],uid=myuser,gid=myuser,vers=3.11
//[REDACTED]/uploaded_files /home/myuser/shared

mount error(11): Resource temporarily unavailable
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

# tail  /var/log/syslog

[...]
Feb 14 12:57:04 prod-backoffice kernel: [105926.746067] CIFS VFS: failed to
connect to IPC (rc=-11)
Feb 14 12:57:04 prod-backoffice kernel: [105926.767443] CIFS VFS: session
0000000044187aeb has no tcon available for a dfs referral request
Feb 14 12:57:04 prod-backoffice kernel: [105926.770039] CIFS VFS:
cifs_mount failed w/return code = -11


# mount -t cifs -o
rw,username=myuser,password=[REDACTED],uid=myuser,gid=myuser,vers=3.1
//[REDACTED]/uploaded_files /home/myuser/shared

mount error(22): Invalid argument
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

# tail  /var/log/syslog

[...]
Feb 14 12:58:54 prod-backoffice kernel: [106036.706942] CIFS VFS: Unknown
vers= option specified: 3.1

# mount -t cifs -o
rw,username=myuser,password=[REDACTED],uid=myuser,gid=myuser,vers=3.0
//[REDACTED]/uploaded_files /home/myuser/shared

[ finally worked! ]

It seems that vers=3.0 worked, but how do I confirm that the mount ACTUALLY
happened with vers=3.0?

Further, `mount` is showing a LOT of options that I did not specify. I do
not fully understand their impact:

//[REDACTED]/uploaded_files on /home/myuser/shared type cifs
(rw,relatime,vers=3.0,cache=strict,username=vl,domain=,uid=1000,forceuid,gid=1000,forcegid,addr=95.216.67.179,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)

Namely: forceuid, forcegid, nouinx, serverino, & mapposix

Could any of these be causing the problems that I'm observing?

PS: vers=3.0 didn't fix the problem!

-- Saurabh.

> Unless you upload a network capture of you mounting and doing the ls -lR
>> on the client it's hard to say what really goes on. I understand
you
>> might not want to make it public.. but if you do
>>
>
> This is the last thing I'll try after I've exhausted all the other
options.
>
Actually, I'm doing this right now. How do I analyse and publicly share the
results?

-- Saurabh.

Saurabh Nanda via samba <samba at lists.samba.org>
writes:
> Which implies that the server & client auto-negotiated a protocol
version
> greater than SMB2.1, right? However, to be sure, I manually specified
vers> in fstab, but something strange happened. While `man mount.cifs` claims
> that the following are allowed -- 1.0, 2.0, 2.1, 3.0, 3.1.1 (or 3.11) --
> few of them failed with strange errors:
>
> # mount -t cifs -o
> rw,username=myuser,password=[REDACTED],uid=myuser,gid=myuser,vers=3.11
> //[REDACTED]/uploaded_files /home/myuser/shared
>
> mount error(11): Resource temporarily unavailable
> Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
That's normal. You need a very recent kernel to be able to use
3.1.1. Note that 3.11 and 3.1.1 are aliased to the same thing, and 3.1
doesn't exist hence why you get invalid parameter.
> # mount -t cifs -o
> rw,username=myuser,password=[REDACTED],uid=myuser,gid=myuser,vers=3.0
> //[REDACTED]/uploaded_files /home/myuser/shared
>
> [ finally worked! ]
>
> It seems that vers=3.0 worked, but how do I confirm that the mount ACTUALLY
> happened with vers=3.0?
When you provide an exact vers= then no auto-negotiation happens (unless
you pass "vers=3" which essentially means 3.x: use 3.0 or above). You
either get the connection or mount fails. None the less, you can dump
the current SMB ressources managed by the kernel by looking at
/proc/fs/cifs/DebugData if your kernel is recent enough it should show
the SMB version.

Alternatively looking at the network trace will also tell you. After
having made the trace you can open it in wireshark [0], filter for
'smb||smb2' and look for Negotiate protocol request/response. The
response will tell you what was picked. More info on how to use
wireshark here [1]. It's a bit old and only covers SMB1 but it's a good
ressource.
> Further, `mount` is showing a LOT of options that I did not specify. I do
> not fully understand their impact:
> [...]
> Namely: forceuid, forcegid, nouinx, serverino, & mapposix
Note that some of this option string is generated at runtime by the
kernel, so what you see is the current running state, which might differ
from what you provided as some options are deduced/tried during connection.

* forceuid/forcegid are sort of internal options to say use mount-provided
  uid in all cases and ignore what the server returns.

* nounix means the server doesn't support unix extensions. These extensions
are only
  supported by samba and for SMB1 only at the moment. Work is in progress
  to add them to SMB2+.

* serverino makes the client use the server-provided unique file id as
  inode numbers. Without it the local inode numbers are generated and not
  persistent and your local apps can run into all sorts of issues.

*  mapposix has been default for a while, it maps reserved characters (: ?
  \ " etc) to reserved unicode characters to/from the servers so you can
  use those characters in file names transparently from linux, without
  needing to rely on protocol extensions.
> Could any of these be causing the problems that I'm observing?
I don't think so.

If you provide a trace we could see if any unecesarry packets are
exchanged or if something goes in loop, or if due to some error, the
kernel close and reopens the connection often and looses time, ...

And from the time between request and response on listing queries we can
see if it's just the server that takes an abnormal amount of time to
respond.

You can look at my instructions in a previous email. The resulting .pcap
file might be quite large.

You could open a bug on the samba bugzilla and upload the trace there,
this will require an account and I don't know the file size limit on
attachements.

Otherwise you could encrypt the trace [2] and upload it on one of many
file sharing websites out there (dropbox, google drive, mega, ...).

0: https://www.wireshark.org/
1:
https://wiki.wireshark.org/Presentations?action=AttachFile&do=view&target=RonnieSahlberg_UsingWireshark.pdf
2: gpg --output mycapture.pcap.enc --symmetric --cipher-algo AES256
mycapture.pcap
   type a passphrase and share it with whoever you feel comfortable

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

>
> When you provide an exact vers= then no auto-negotiation happens (unless
> you pass "vers=3" which essentially means 3.x: use 3.0 or above).
You
> either get the connection or mount fails. None the less, you can dump
> the current SMB ressources managed by the kernel by looking at
> /proc/fs/cifs/DebugData if your kernel is recent enough it should show
> the SMB version.
>
There is something going horribly wrong with my configuration. Notice that
CIFS version in the debug output below --

Display Internal CIFS Data Structures for Debugging
---------------------------------------------------
CIFS Version 2.10
Features: dfs fscache lanman posix spnego xattr acl
Active VFS Requests: 0
Servers:
Number of credits: 510
1) Name: [REDACTED] Uses: 1 Capability: 0x300047     Session Status: 1
 TCP status: 1
        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0
        Shares:
        0) IPC: \\[REDACTED]\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0
        Share Capabilities: None        Share Flags: 0x8000

        1) \\[REDACTED]\uploaded_files Mounts: 1 DevInfo: 0x20 Attributes:
0x1006f
        PathComponentMax: 255 Status: 1 type: DISK
        Share Capabilities: None Aligned, Partition Aligned,    Share
Flags: 0x8000     Optimal sector size: 0x200

        MIDs:


> * nounix means the server doesn't support unix extensions. These
> extensions are only
>   supported by samba and for SMB1 only at the moment. Work is in progress
>   to add them to SMB2+.
>
Will this in any way interfere with "case sensitive = yes" -
https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#NAMEMANGLINGSECT
?

> *  mapposix has been default for a while, it maps reserved characters (: ?
>   \ " etc) to reserved unicode characters to/from the servers so you
can
>   use those characters in file names transparently from linux, without
>   needing to rely on protocol extensions.
>
Again, this sounds like it directly conflicts with `nounix`!

> If you provide a trace we could see if any unecesarry packets are
> exchanged or if something goes in loop, or if due to some error, the
> kernel close and reopens the connection often and looses time, ...
>
I have a 1.3 GB capture.pcap from the command you shared earlier. `ls -lR`
ran only for a few minutes, and ended up generating so much network
chatter!

Btw, I have `smb encrypt = required` on the server config. Ideally, I would
hope, the network capture would be useless!

Apart from network capture, is there any debug logging that I can enable to
give similar insights?

-- Saurabh.