samba - Jan 2022 - making smbclient work with a DFS setup where mount.cifs does work / disabling krb5 for testing?

Hi,

I am having trouble accessing a file share structure with
authentication via MS AD and using several storage servers strung
together using DFS links. Maybe someone here has some ideas on how to
track down the failure in smbclient.

My goal is to enable shell users on some Linux systems that have no
further deal in the AD domain (login handled via SSH keys) to access
their SMB shares using smbclient, without having to configure mount
points in the system or even allowing them adding mounts via FUSE.
Sadly, smbclient just fails to connect into our DFS structure. I can
access any of the actual storage server endpoints when I put in the
resolved server address and share name, with the same AD
authentication behind the scenes (I am told). But entering via the DFS
just fails.

Smbclient seems like the simplest ad-hoc way with the least amount of
complexity. Just a simple program speaking the protocol, no behind the
scenes magic.

From a Linux box, without any domain joining or Kerberos ticketing (as
I can gather), the access works for root via

# mount -t cifs -o vers=3,username=user at domain.suffix //ad.domain.suffix/data
/mnt/aux/
Password for user at domain.suffix@//ad.domain.suffix/data:  ******************
# ls /mnt/aux/link1/link2
[proper directory contents being listed]

(Note: The AD domain is ad.domain.suffix, the users are named with
user at domain.suffix, not user at ad.domain.suffix.)

This works nicely. With some raised verbosity, I get such in dmesg,
some of those chatty messages could relate to the failure to get
anything up with smbclient:

[7478545.444023] Status code returned 0xc0000016 STATUS_MORE_PROCESSING_REQUIRED
[7478545.454679] Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[7478545.454687] CIFS VFS: \\ad.domain.suffix\data error -2 on ioctl to get
interface list
[7478545.467777] Status code returned 0xc0000016 STATUS_MORE_PROCESSING_REQUIRED
[7478545.475884] CIFS VFS: Autodisabling the use of server inode numbers on new
server.
[7478545.483650] CIFS VFS: The server doesn't seem to support them properly
or the files might be on different servers (DFS).
[7478545.494690] CIFS VFS: Hardlinks will not be recognized on this mount.
Consider mounting with the "noserverino" option to silence this
message.
[7478588.975709] Status code returned 0x80000006 STATUS_NO_MORE_FILES
[7478589.896426] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478589.897828] FS-Cache: Duplicate cookie detected
[7478589.902575] FS-Cache: O-cookie c=00000000a07782bc [p=000000003590d94a
fl=222 nc=0 na=1]
[7478589.910757] FS-Cache: O-cookie d=00000000d70a5c7c n=00000000ba7c6d95
[7478589.917288] FS-Cache: O-key=[5] '646174656e'
[7478589.921740] FS-Cache: N-cookie c=0000000094a3fdd9 [p=000000003590d94a fl=2
nc=0 na=1]
[7478589.929740] FS-Cache: N-cookie d=00000000d70a5c7c n=00000000a61e519a
[7478589.936270] FS-Cache: N-key=[5] '646174656e'
[7478589.943311] FS-Cache: Duplicate cookie detected
[7478589.948021] FS-Cache: O-cookie c=00000000a07782bc [p=000000003590d94a
fl=222 nc=0 na=1]
[7478589.956189] FS-Cache: O-cookie d=00000000d70a5c7c n=00000000ba7c6d95
[7478589.962716] FS-Cache: O-key=[5] '646174656e'
[7478589.967163] FS-Cache: N-cookie c=0000000094a3fdd9 [p=000000003590d94a fl=2
nc=0 na=1]
[7478589.975162] FS-Cache: N-cookie d=00000000d70a5c7c n=00000000a61e519a
[7478589.981688] FS-Cache: N-key=[5] '646174656e'
[7478589.988726] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478589.991085] Status code returned 0xc0000016 STATUS_MORE_PROCESSING_REQUIRED
[7478590.006991] Status code returned 0xc00000cc STATUS_BAD_NETWORK_NAME
[7478590.006995] CIFS VFS:  BAD_NETWORK_NAME: \\ad.domain.suffix\link1$
[7478590.015454] Status code returned 0xc0000016 STATUS_MORE_PROCESSING_REQUIRED
[7478590.023452] Status code returned 0xc00000cc STATUS_BAD_NETWORK_NAME
[7478590.023456] CIFS VFS:  BAD_NETWORK_NAME: \\ad.domain.suffix\link1$
[7478590.031603] Status code returned 0xc0000016 STATUS_MORE_PROCESSING_REQUIRED
[7478590.039503] CIFS VFS: Autodisabling the use of server inode numbers on new
server.
[7478590.047255] CIFS VFS: The server doesn't seem to support them properly
or the files might be on different servers (DFS).
[7478590.058291] CIFS VFS: Hardlinks will not be recognized on this mount.
Consider mounting with the "noserverino" option to silence this
message.
[7478590.074730] Status code returned 0x80000006 STATUS_NO_MORE_FILES
[7478590.079865] Status code returned 0x80000006 STATUS_NO_MORE_FILES
[7478590.080971] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478590.081282] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478590.081578] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478590.081886] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478590.082187] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478590.082447] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478590.082731] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478591.220069] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478591.222862] Status code returned 0x80000006 STATUS_NO_MORE_FILES
[7478592.001255] Status code returned 0x80000006 STATUS_NO_MORE_FILES
[7478592.002404] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478592.426108] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478592.427992] FS-Cache: Duplicate cookie detected
[7478592.432731] FS-Cache: O-cookie c=000000004ec79308 [p=000000006acf55c2
fl=222 nc=0 na=1]
[7478592.440956] FS-Cache: O-cookie d=00000000d70a5c7c n=0000000020add7cf
[7478592.447479] FS-Cache: O-key=[4] '72727a24'
[7478592.451753] FS-Cache: N-cookie c=0000000053e9df59 [p=000000006acf55c2 fl=2
nc=0 na=1]
[7478592.459752] FS-Cache: N-cookie d=00000000d70a5c7c n=00000000928a05f7
[7478592.466278] FS-Cache: N-key=[4] '72727a24'
[7478592.473267] Status code returned 0xc0000016 STATUS_MORE_PROCESSING_REQUIRED
[7478592.484641] Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED
[7478592.498916] Status code returned 0xc0000016 STATUS_MORE_PROCESSING_REQUIRED
[7478592.601238] Status code returned 0xc00000bb STATUS_NOT_SUPPORTED
[7478592.601493] Status code returned 0xc0000003 STATUS_INVALID_INFO_CLASS
[7478592.603253] CIFS VFS: Autodisabling the use of server inode numbers on new
server.
[7478592.611013] CIFS VFS: The server doesn't seem to support them properly
or the files might be on different servers (DFS).
[7478592.622091] CIFS VFS: Hardlinks will not be recognized on this mount.
Consider mounting with the "noserverino" option to silence this
message.
[7478592.640939] Status code returned 0x80000006 STATUS_NO_MORE_FILES

Interesting is the repeated line of

[7478590.006995] CIFS VFS:  BAD_NETWORK_NAME: \\ad.domain.suffix\link1$

which is non-fatal for mount.cifs, but it might indicate some possible
trouble.

When I try that with smbclient on the very same box, it always just
looks like authentication failure (but I'm rather sure I am correctly
typing the password some of the times). I also tested a different entry
point earlier where the smbclient connection works, but then the
failure comes in the same way once I try to follow a DFS link.

# smbclient -d 7 -U user at domain.suffix //ad.domain.suffix/daten
INFO: Current debug levels:
[?]
lp_load_ex: refreshing parameters
Initialising global parameters
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
INFO: Current debug levels:
[?]
Processing section "[global]"
doing parameter workgroup = WORKGROUP
doing parameter server string = %h server (Samba, Ubuntu)
doing parameter log file = /var/log/samba/log.%m
doing parameter max log size = 1000
doing parameter logging = file
doing parameter panic action = /usr/share/samba/panic-action %d
doing parameter server role = standalone server
doing parameter obey pam restrictions = yes
doing parameter unix password sync = yes
doing parameter passwd program = /usr/bin/passwd %u
doing parameter passwd chat = *Enter\snew\s*\spassword:* %n\n
*Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
doing parameter pam password change = yes
doing parameter map to guest = bad user
doing parameter usershare allow guests = yes
pm_process() returned Yes
lp_servicenumber: couldn't find homes
added interface [?]
added interface [?]
added interface [?]
Netbios name list:-
my_netbios_names[0]="servername"
Client started (version 4.13.14-Ubuntu).
Opening cache file at /run/samba/gencache.tdb
sitename_fetch: No stored sitename for realm ''
name ad.domain.suffix#20 found.
Connecting to <IP of one of the domain controller nodes> at port 445
Socket options:
	SO_KEEPALIVE = 0
	SO_REUSEADDR = 0
	SO_BROADCAST = 0
	TCP_NODELAY = 1
	TCP_KEEPCNT = 9
	TCP_KEEPIDLE = 7200
	TCP_KEEPINTVL = 75
	IPTOS_LOWDELAY = 0
	IPTOS_THROUGHPUT = 0
	SO_REUSEPORT = 0
	SO_SNDBUF = 46080
	SO_RCVBUF = 131072
	SO_SNDLOWAT = 1
	SO_RCVLOWAT = 1
	SO_SNDTIMEO = 0
	SO_RCVTIMEO = 0
	TCP_QUICKACK = 1
	TCP_DEFER_ACCEPT = 0
	TCP_USER_TIMEOUT = 0
 session request ok
 negotiated dialect[SMB3_11] against server[ad.domain.suffix]
Enter user at domain.suffix's password: 
cli_session_creds_prepare_krb5: Doing kinit for user at domain.suffix to access
ad.domain.suffix
cli_session_setup_spnego_send: Connect to ad.domain.suffix as USER at
DOMAIN.SUFFIX using SPNEGO
GENSEC backend 'gssapi_spnego' registered
GENSEC backend 'gssapi_krb5' registered
GENSEC backend 'gssapi_krb5_sasl' registered
GENSEC backend 'spnego' registered
GENSEC backend 'schannel' registered
GENSEC backend 'naclrpc_as_system' registered
GENSEC backend 'sasl-EXTERNAL' registered
GENSEC backend 'ntlmssp' registered
GENSEC backend 'ntlmssp_resume_ccache' registered
GENSEC backend 'http_basic' registered
GENSEC backend 'http_ntlm' registered
GENSEC backend 'http_negotiate' registered
GENSEC backend 'krb5' registered
GENSEC backend 'fake_gssapi_krb5' registered
Starting GENSEC mechanism spnego
Starting GENSEC submechanism gse_krb5
gse_get_client_auth_token: gss_init_sec_context failed with [ Miscellaneous
failure (see text): Message stream modified](2529638953)
gensec_update_done: gse_krb5[0x559a8ee8c4f0]: NT_STATUS_LOGON_FAILURE
gensec_spnego_client_negTokenTarg_step: SPNEGO(gse_krb5) login failed:
NT_STATUS_LOGON_FAILURE
gensec_update_done: spnego[0x559a8ee80680]: NT_STATUS_LOGON_FAILURE
SPNEGO login failed: The attempted logon is invalid. This is either due to a bad
username or authentication information.
session setup failed: NT_STATUS_LOGON_FAILURE


My question at that point: It very much looks like smbclient is trying
to get things running using krb5 authentication. I'm pretty sure that
mount.cifs is not attempting that. Is there some way to make smbclient
try something else? Or fall back to NTLMSSP? I only found an option to
explicitly _enforce_ krb5, not disable it.

I'd like to debug smbclient not working and any possible path down into
Kerberos realms separately. I do remember trying krb5 explicitly on a
system where kinit/klist worked just fine getting a ticket, but I got
the same ?Message stream modified? error when trying to access DFS
links. On that system, mount.cifs also doesn't do the trick with DFS.
There could be all kinds of fun with network limitations for machines
not in segregated Windows networks, so I am trying to establish a
baseline here on a system that is just fine with the DFS using
mount.cifs.

Any ideas?


Alrighty then,

Thomas

-- 
Dr. Thomas Orgis
HPC @ Universit?t Hamburg

Hello,

On 1/13/22 00:24, Dr. Thomas Orgis via samba wrote:
> # smbclient -d 7 -U user at domain.suffix //ad.domain.suffix/daten
> INFO: Current debug levels:
fwiw, to get useful logs use loglevel 10. Anything below is useless for 
debugging. Anything above is rarely adding anything important.
> [?]
> My question at that point: It very much looks like smbclient is trying
> to get things running using krb5 authentication. I'm pretty sure that
> mount.cifs is not attempting that. Is there some way to make smbclient
> try something else? Or fall back to NTLMSSP? I only found an option to
> explicitly _enforce_ krb5, not disable it.
It depends on the version. Iirc in older versions -k no? In 4.15 
--use-kerberos=off

You could also just use the server IP instead of the DNS name, that will 
implicitly prevent Kerberos from being used.
> I'd like to debug smbclient not working and any possible path down into
> Kerberos realms separately. I do remember trying krb5 explicitly on a
> system where kinit/klist worked just fine getting a ticket, but I got
> the same ?Message stream modified? error when trying to access DFS
> links. On that system, mount.cifs also doesn't do the trick with DFS.
> There could be all kinds of fun with network limitations for machines
> not in segregated Windows networks, so I am trying to establish a
> baseline here on a system that is just fine with the DFS using
> mount.cifs.
Well, for debugging the DFS issue a network trace and loglevel 10 log 
would be helpful and the hopefully someone has some spare time to look 
into those logs.

-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20220113/8608846e/OpenPGP_signature.sig>