samba - Apr 2020 - BIND9/DNS lookups stopped working after upgrading our Comcast modem/gateway

OK here's our setup, we're running pfSense as our firewall/router. We 
have a dual-WAN with a Comcast Business Gateway and another local ISP. 
We're running Samba 4.7.6 as the AD DC & Bind 9.9.4. Here's the
issue,
we just upgraded our Comcast service to a higher speed & they replaced 
the modem/gateway. Everything was working fine, before the modem swap. 
Once they did that we could no longer browse the internet from inside 
the LAN over the Comcast pipe. If I disable our Comcast pipe in pfSense 
we can get to the internet over our other ISP.
I'm pretty sure the issue is with Bind, but can't figure out why. Here 
is some output when I run nslookup & dig on the DC box. Both nslookup & 
dig work if I specify an external DNS server, but not when I let it use 
localhost/Bind.

 ??? [root at dc etc]# nslookup comcast.com
 ??? Server:???????? 127.0.0.1
 ??? Address:??????? 127.0.0.1#53

 ??? ** server can't find comcast.com: NXDOMAIN


 ??? [root at dc etc]# nslookup comcast.com 8.8.8.8
 ??? Server:???????? 8.8.8.8
 ??? Address:??????? 8.8.8.8#53

 ??? Non-authoritative answer:
 ??? Name:?? comcast.com
 ??? Address: 69.252.80.75


 ??? [root at dc etc]# dig comcast.com
 ??? ; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>>
comcast.com
 ??? ;; global options: +cmd
 ??? ;; Got answer:
 ??? ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 3360
 ??? ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

 ??? ;; OPT PSEUDOSECTION:
 ??? ; EDNS: version: 0, flags:; udp: 4096
 ??? ;; QUESTION SECTION:
 ??? ;comcast.com.?????????????????? IN????? A

 ??? ;; Query time: 1 msec
 ??? ;; SERVER: 127.0.0.1#53(127.0.0.1)
 ??? ;; WHEN: Sat Apr 11 17:08:25 PDT 2020
 ??? ;; MSG SIZE? rcvd: 40


 ??? [root at dc etc]# dig @8.8.8.8 comcast.com
 ??? ; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @8.8.8.8
comcast.com
 ??? ; (1 server found)
 ??? ;; global options: +cmd
 ??? ;; Got answer:
 ??? ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26449
 ??? ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

 ??? ;; OPT PSEUDOSECTION:
 ??? ; EDNS: version: 0, flags:; udp: 4096
 ??? ;; QUESTION SECTION:
 ??? ;comcast.com.?????????????????? IN????? A

 ??? ;; ANSWER SECTION:
 ??? comcast.com.??????????? 32????? IN????? A?????? 69.252.80.75

 ??? ;; Query time: 4 msec
 ??? ;; SERVER: 8.8.8.8#53(8.8.8.8)
 ??? ;; WHEN: Sat Apr 11 17:08:43 PDT 2020
 ??? ;; MSG SIZE? rcvd: 56

Also, if I disable our Comcast pipe in pfSense and do `nslookup 
comcast.com` or `dig comcast.com` over the other ISP, they return 
perfectly fine when using localhost/Bind.

Here is my named.conf file

 ??? acl mynetworks {
 ??? ??? 192.168.254.0/24;
 ??? ??? 192.168.252.0/24;
 ??? ??? 192.168.251.0/24;
 ??? ??? 192.168.250.0/24;
 ???? };

 ??? options {
 ??? ??? listen-on port 53 { localnets; };
 ??? ??? listen-on-v6 { none; };
 ??? ??? directory ??? "/var/named";
 ??? ??? dump-file ??? "/var/named/data/cache_dump.db";
 ??? ??? statistics-file "/var/named/data/named_stats.txt";
 ??? ??? memstatistics-file "/var/named/data/named_mem_stats.txt";
 ??? ??? allow-query???? { localhost; mynetworks; };

 ??? ??? recursion yes;
 ??? ??? dnssec-enable yes;
 ??? ??? dnssec-validation auto;
 ??? ??? dnssec-lookaside auto;

 ??? ??? /* Path to ISC DLV key */
 ??? ??? bindkeys-file "/etc/named.iscdlv.key";

 ??? ??? managed-keys-directory "/var/named/dynamic";

 ??? ??? pid-file "/run/named/named.pid";
 ??? ??? session-keyfile "/run/named/session.key";

 ??? ??? # samba BIND9_DLZ
 ??? ??? tkey-gssapi-keytab "/var/lib/samba/private/dns.keytab";
 ??? };

 ??? logging {
 ??? ??? channel default_debug {
 ??? ??? ??? file "data/named.run";
 ??? ??? ??? severity dynamic;
 ??? ??? };
 ??? ??? channel my_log_file {
 ??? ??? ??? file "/var/log/named/named.log" versions 3 size 3m;
 ??? ??? ??? severity info;
 ??? ??? ??? print-time yes;
 ??? ??? ??? print-severity yes;
 ??? ??? ??? print-category yes;
 ??? ??? };
 ??? ??? channel my_syslog {
 ??? ??? ??? syslog daemon;
 ??? ??? ??? severity info;
 ??? ??? ??? print-time no;
 ??? ??? ??? print-severity no;
 ??? ??? ??? print-category no;
 ??? ??? };
 ??? ??? category default { my_log_file; my_syslog; };
 ??? ??? category dnssec? { my_log_file; };
 ??? ??? category lame-servers { null; };
 ??? };


 ??? // so we can control the running named process with the rndc utility
 ??? include "/etc/rndc.key";

 ??? zone "." IN {
 ??? ??? type hint;
 ??? ??? file "named.ca";
 ??? };

 ??? dlz? "tipping.lan" {
 ???????? database "dlopen /usr/lib64/samba/bind9/dlz_bind9_9.so";
 ??? };

 ??? include "/etc/named.rfc1912.zones";
 ??? include "/etc/named.root.key";

If I connect a laptop directly to the Comcast modem, I can get to the 
internet just fine. Also, if I bypass the DC on a LAN workstation to use 
an external DNS server, I can access the internet over the Comcast pipe.

So why is DNS/Bind working on one pipe, but not the other? Any help 
would be greatly appreciated.

On 12/04/2020 02:08, Barry Ralphs via samba wrote:
> OK here's our setup, we're running pfSense as our firewall/router.
We
> have a dual-WAN with a Comcast Business Gateway and another local ISP. 
> We're running Samba 4.7.6 as the AD DC & Bind 9.9.4. Here's the
issue,
> we just upgraded our Comcast service to a higher speed & they replaced 
> the modem/gateway. Everything was working fine, before the modem swap. 
> Once they did that we could no longer browse the internet from inside 
> the LAN over the Comcast pipe. If I disable our Comcast pipe in 
> pfSense we can get to the internet over our other ISP.
> I'm pretty sure the issue is with Bind, but can't figure out why.
Are you sure it worked before ?

This is my take on your named.conf:

 ??? acl mynetworks {
 ??????? 192.168.254.0/24;
 ??????? 192.168.252.0/24;
 ??????? 192.168.251.0/24;
 ??????? 192.168.250.0/24;
 ???? };

 ??? options {
 ??????? directory???? "/var/named";
 ??????? notify no;
 ??????? empty-zones-enable no;
 ??????? allow-query???? { localhost; mynetworks; };
 ??????? allow-recursion { 127.0.0.1/32; mynetworks; };
 ??????? forwarders { 8.8.8.8; 8.8.4.4; };
 ??????? allow-transfer { none; };
 ??????? dnssec-validation no;
 ??????? dnssec-enable no;
 ??????? dnssec-lookaside no;
 ??????? listen-on-v6 { none; };
 ??????? listen-on port 53 { localnets; };
 ??????? dump-file???? "/var/named/data/cache_dump.db";
 ??????? statistics-file "/var/named/data/named_stats.txt";
 ??????? memstatistics-file "/var/named/data/named_mem_stats.txt";
 ??????? managed-keys-directory "/var/named/dynamic";
 ??????? pid-file "/run/named/named.pid";
 ??????? session-keyfile "/run/named/session.key";

 ??????? # samba BIND9_DLZ
 ??????? tkey-gssapi-keytab "/var/lib/samba/private/dns.keytab";
 ??? };

 ??? logging {
 ??????? channel default_debug {
 ??????????? file "data/named.run";
 ??????????? severity dynamic;
 ??????? };
 ??????? channel my_log_file {
 ??????????? file "/var/log/named/named.log" versions 3 size 3m;
 ??????????? severity info;
 ??????????? print-time yes;
 ??????????? print-severity yes;
 ??????????? print-category yes;
 ??????? };
 ??????? channel my_syslog {
 ??????????? syslog daemon;
 ??????????? severity info;
 ??????????? print-time no;
 ??????????? print-severity no;
 ??????????? print-category no;
 ??????? };
 ??????? category default { my_log_file; my_syslog; };
 ??????? category dnssec? { my_log_file; };
 ??????? category lame-servers { null; };
 ??? };


 ??? zone "." IN {
 ??????? type hint;
 ??????? file "named.ca";
 ??? };

 ??? include "/var/lib/samba/private/named.conf";
 ??? include "/etc/named.rfc1912.zones";

You will notice that I have removed, altered and added lines, the above 
is what I would expect on a Samba AD DC. If the first nameserver in 
/etc/resolv.conf is pointing to 127.0.0.1, then change it to the DC's 
ipaddress.

You might also notice that I have added this:

forwarders { 8.8.8.8; 8.8.4.4; };

Your AD clients should use the DC as their nameserver. The DC is 
authoritative for the AD domain and should return all dns records for 
the AD domain. Anything outside the AD domain will be forwarded to the 
'forwarders' (Google in this case) who will return the internet info to 
the DC, which will then pass this to the client. You don't have this 
line, so the DC doesn't know where to ask for internet data, you might 
have a forwarder line in smb.conf, but this will be ignored when using 
bind9.

So to recap, it should be:

 ? Client --> DC --> return AD domain data
 ?????????? ? ? ?? |---> anything outside the AD domain --> ask forwarder

Rowland

> Are you sure it worked before ?
>
> This is my take on your named.conf:
>
> ??? acl mynetworks {
> ??????? 192.168.254.0/24;
> ??????? 192.168.252.0/24;
> ??????? 192.168.251.0/24;
> ??????? 192.168.250.0/24;
> ???? };
>
> ??? options {
> ??????? directory???? "/var/named";
> ??????? notify no;
> ??????? empty-zones-enable no;
> ??????? allow-query???? { localhost; mynetworks; };
> ??????? allow-recursion { 127.0.0.1/32; mynetworks; };
> ??????? forwarders { 8.8.8.8; 8.8.4.4; };
> ??????? allow-transfer { none; };
> ??????? dnssec-validation no;
> ??????? dnssec-enable no;
> ??????? dnssec-lookaside no;
> ??????? listen-on-v6 { none; };
> ??????? listen-on port 53 { localnets; };
> ??????? dump-file???? "/var/named/data/cache_dump.db";
> ??????? statistics-file "/var/named/data/named_stats.txt";
> ??????? memstatistics-file "/var/named/data/named_mem_stats.txt";
> ??????? managed-keys-directory "/var/named/dynamic";
> ??????? pid-file "/run/named/named.pid";
> ??????? session-keyfile "/run/named/session.key";
>
> ??????? # samba BIND9_DLZ
> ??????? tkey-gssapi-keytab "/var/lib/samba/private/dns.keytab";
> ??? };
>
> ??? logging {
> ??????? channel default_debug {
> ??????????? file "data/named.run";
> ??????????? severity dynamic;
> ??????? };
> ??????? channel my_log_file {
> ??????????? file "/var/log/named/named.log" versions 3 size 3m;
> ??????????? severity info;
> ??????????? print-time yes;
> ??????????? print-severity yes;
> ??????????? print-category yes;
> ??????? };
> ??????? channel my_syslog {
> ??????????? syslog daemon;
> ??????????? severity info;
> ??????????? print-time no;
> ??????????? print-severity no;
> ??????????? print-category no;
> ??????? };
> ??????? category default { my_log_file; my_syslog; };
> ??????? category dnssec? { my_log_file; };
> ??????? category lame-servers { null; };
> ??? };
>
>
> ??? zone "." IN {
> ??????? type hint;
> ??????? file "named.ca";
> ??? };
>
> ??? include "/var/lib/samba/private/named.conf";
> ??? include "/etc/named.rfc1912.zones";
>
> You will notice that I have removed, altered and added lines, the 
> above is what I would expect on a Samba AD DC. If the first nameserver 
> in /etc/resolv.conf is pointing to 127.0.0.1, then change it to the 
> DC's ipaddress.
>
> You might also notice that I have added this:
>
> forwarders { 8.8.8.8; 8.8.4.4; };
>
> Your AD clients should use the DC as their nameserver. The DC is 
> authoritative for the AD domain and should return all dns records for 
> the AD domain. Anything outside the AD domain will be forwarded to the 
> 'forwarders' (Google in this case) who will return the internet
info
> to the DC, which will then pass this to the client. You don't have 
> this line, so the DC doesn't know where to ask for internet data, you 
> might have a forwarder line in smb.conf, but this will be ignored when 
> using bind9.
>
> So to recap, it should be:
>
> ? Client --> DC --> return AD domain data
> ?????????? ? ? ?? |---> anything outside the AD domain --> ask
forwarder
>
> Rowland
>
Thanks for the reply Rowland.
Yes, I'm sure it was working before & now I've figure out what the
real
issue was.
When we upgraded our Comcast service & came with their new SecurityEdge 
service.
A cloud based web filter that's supposed to block malware, phishing, etc..
https://business.comcast.com/learn/internet/security-edge

However, it seems it's also blocking BIND from receiving the DNS replies 
from the root servers.
To fix it I put BIND in forwarding mode to 1.1.1.1.1 & 8.8.8.8 (like 
your recommendation) & commented out the root server lookups.
#zone "." IN {
#?? type hint;
#?? file "named.ca";
#};

And everything started working again.
Hopefully I can get Comcast to unblock the root servers or remove the 
SecurityEdge service & then I can set BIND back to use the root servers...
Here's some other posts from people having similar issues, which lead me 
to this solution.
https://forums.businesshelp.comcast.com/t5/Domain-Names-Static-IP/transparent-dns-proxying-started-after-a-modem-swap/m-p/39845
https://www.reddit.com/r/msp/comments/dikvta/comcast_securityedge/

Do your config charges still apply in forwarding only mode, or are there 
other changes I should make to the config?

My resolv.conf file is:
# Generated by NetworkManager
search tipping.lan
nameserver 127.0.0.1

Still change it to my DC IP?
I think if I change my /etc/sysconfig/network-scripts/ifcfg-eth0
IPADDR=192.168.254.37
DNS1=127.0.0.1
to
IPADDR=192.168.254.37
DNS1=192.168.254.37

It will update the resolv.conf


Thanks,
Barry