samba - Dec 2020 - Cannot contact any KDC error when contacting Active Directory domains with a short domain name length

Hi, I'm developing a service which uses Samba to connect to Active
Directory domains of arbitrary names. For the most part the service works
fine, but I've hit an odd corner case where if I try to connect to a domain
name of a short length, like 'prod.com', I get errors which indicate
that
the KDC cannot be contacted. I've observed this for 'xyz.com',
'blah.com',
and 'abc.com', but upping the character length to 'prod1.com'
and greater,
the issue goes away. The issue is seen after joining the domain, and doing
a 'net ads search -P ....'. I've manually kinited with the machine
keytab,
and retrieved the password from the tdb and that worked for a manual kinit,
so I can confirm that neither of those are an issue.

I've observed, but cannot explain, a mitigation. In my smb.conf file, for
the failed attempts, the netbios name I was generating was 15 characters
long. This works fine for the domains of longer length. For the shorter
domain names, I found the issue goes away if I limit the netbios name
length between 6-11 characters in length.

Here is an example output I was seeing after 'net ads search -P ...'

kerberos_kinit_password 307CA1E6F351C28$@BLAH.COM <http://blah.com/>
failed:
> Cannot contact any KDC for requested realm ads_connect: Cannot contact any
> KDC for requested realm.

There is also some hint that this will fail during the 'net ads join'
call.

A good join will have output like this:
> root at 1a6eb250bf2d376:/# net ads join -s /var/opt/mssql/smb.conf -U
> 'setupadmin%<redacted>' -k
> Kinit for setupadmin to access cifs/dc3a254757566c2.blah.com at BLAH.COM
> failed: Cannot determine realm for host
> Using short domain name -- BLAH
> Joined 'QWERTY123' to dns domain 'blah.com'
> No DNS domain configured for qwerty123. Unable to perform DNS Update.
> DNS update failed: NT_STATUS_INVALID_PARAMETER

and a bad join will have output like this:
> root at bf089a716f8ca86:/# net ads join -s /var/opt/mssql/ad/smb.conf -U
> 'setupadmin%<redacted>' -k
> Kinit for setupadmin to access cifs/dc8aa83259417d0.blah.com at BLAH.COM
> failed: Cannot determine realm for host
> Kinit for 307CA1E6F351C28$ to access cifs/dc8aa83259417d0.blah.com at
BLAH.COM
> failed: Client not found in Kerberos database
> Using short domain name -- BLAH
> Joined '307CA1E6F351C28' to dns domain 'blah.com'
> kerberos_kinit_password 307CA1E6F351C28$@BLAH.COM failed: Client not
> found in Kerberos database
> DNS update failed: kinit failed: Client not found in Kerberos database

The setups above were identical, except the 'good' scenario uses a short
netbios name of 'QWERTY123', while the 'bad' scenario uses
netbios name of
'307CA1E6F351C28', but these longer netbios name work fine for longer
domain name lengths.

I suspect that somewhere in samba it is using some heuristic to lookup or
contact the KDC using a slightly different mechanism dependent on the
domain name length, and that there is some necessary port that is blocked
or some networking issue on my end. When I highly simplified my network
scenario, I was not able to repro this behavior. The current network
scenario involves multiple Google Cloud VPC networks which are peered. When
I simplified to a single network, I could not repro this behavior with
short domain names. This is why I believe there must be something samba
does differently in the network communication for these short domains
(contacts a different port, uses some different dns lookup, etc).

Any suggestions on what could be causing this is highly appreciated, I've
been looking into this issue for several days, and it took quite a bit of
experimenting to find this mitigation that I can't explain why it works.

Thanks,

Douglas Dlutz

I think I've found out the root cause is due to a UDP limit, and now my
question is essentially "How can I force samba to communicate to Active
Directory over TCP"?

I took some packet captures, and noticed that using a long domain resulted
in KRB5KRB_ERR_REPONSE_TOO_BIG response to the AS-REQ, and then the next
AS-REQ was over TCP and all was fine. Then, I noticed that for short domain
with short netbios name, it all fit within a single UDP packet. with a
short domain name and large netbios name, it didn't fit within a single UDP
packet and was getting fragmented, but for some reason isn't sending
back KRB5KRB_ERR_REPONSE_TOO_BIG.

If somebody knows a way to force "TCP always" in either smb.conf for
Active
Directory commands or in 'net ads ...' command flags that would be
great. I
can't find anything easily documented on
https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html or
https://www.samba.org/samba/docs/current/man-html/net.8.html.

I assumed samba would be using krb5 library underneath the hood and would
respect udp_preference_limit in my krb5.conf, but that does not appear to
be happening.