Hi?guys. There is a wierd problem with iptables recently, hopes somebody can help me. I have installed Centos 7.2.1511 on a bare metal Dell server these days, disabled firewalld and enabled iptables.services, and setup a group of very simple rules, as the following: # iptables-save # Generated by iptables-save v1.4.21 on Tue Apr 23 09:15:14 2019 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [2449555:327804572] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 49152:49664 -m conntrack --ctstate NEW -j ACCEPT -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 49152:49664 -m conntrack --ctstate NEW -j ACCEPT -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 24007 -m conntrack --ctstate NEW -j ACCEPT -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 24007 -m conntrack --ctstate NEW -j ACCEPT -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT -A INPUT -s 10.3.7.0/25 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Tue Apr 23 09:15:14 2019>From time to time, when this server, say 10.3.3.3, trying to connect to port24007 of another server 10.3.3.4, it will fail sometimes. from tcpdump output, you can see these packages: 22:49:05.992737 IP 10.3.3.3.49149 > 10.3.3.4.24007: Flags [S], seq 2454712274, win 29200, options [mss 1460,sackOK,TS val 24055648 ecr 0,nop,wscale 7], length 0 22:49:05.992847 IP 10.3.3.4.24007 > 10.3.3.3.49149: Flags [S.], seq 3127562073, ack 2454712275, win 28960, options [mss 1460,sackOK,TS val 17803660 ecr 24055648,nop,wscale 7], length 0 22:49:05.992872 IP 10.3.3.3 > 10.3.3.4: ICMP host 10.3.3.3 unreachable - admin prohibited, length 68 The package back from 10.3.3.4 is prohibited by 10.3.3.3, why? Obviously, the package that's been prohibited is the SYNC package that in response of the SYNC sent out by 10.3.3.3, it should match the iptables rule -m state --state RELATED,ESTABLISHED -j ACCEPT, but still be prohibited by the rule -j REJECT --reject-with icmp-host-prohibited. Also, the iptables -nvL output: # iptables -nvL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 8258 615K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 1 84 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT tcp -- * * 172.22.0.0/16 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW 0 0 ACCEPT tcp -- * * 10.3.3.0/25 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW 0 0 ACCEPT tcp -- * * 172.22.0.0/16 0.0.0.0/0 tcp dpt:24007 ctstate NEW 918 55080 ACCEPT tcp -- * * 10.3.3.0/25 0.0.0.0/0 tcp dpt:24007 ctstate NEW 1 60 ACCEPT tcp -- * * 10.3.3.0/25 0.0.0.0/0 tcp dpt:22 ctstate NEW 0 0 ACCEPT tcp -- * * 10.3.7.0/25 0.0.0.0/0 tcp dpt:22 ctstate NEW 244 14640 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT 7378 packets, 957K bytes) pkts bytes target prot opt in out source destination I noticed that the 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited one increase fast. Is this a bug of iptables, or is there something that I have not noticed? Any hints will be appreciated. Likun
Stephen John Smoogen
2019-Apr-24 10:35 UTC
[CentOS] Iptables blocks out going connetion some times
On Wed, 24 Apr 2019 at 06:01, likun <kun.li at ucarinc.com> wrote:> Hi?guys. > > There is a wierd problem with iptables recently, hopes somebody can help > me. > > I have installed Centos 7.2.1511 on a bare metal Dell server these days, > disabled firewalld and enabled iptables.services, and setup a group of very > simple rules, as the following: > >I believe you are saying both 10.3.3.3 and 10.3.3.4 have this same firewall but I am not sure.> # iptables-save > > # Generated by iptables-save v1.4.21 on Tue Apr 23 09:15:14 2019 > > *filter > > :INPUT ACCEPT [0:0] > > :FORWARD ACCEPT [0:0] > > :OUTPUT ACCEPT [2449555:327804572] > > -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT > > -A INPUT -p icmp -j ACCEPT > > -A INPUT -i lo -j ACCEPT > > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 49152:49664 -m conntrack > --ctstate NEW -j ACCEPT > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 49152:49664 -m conntrack > --ctstate NEW -j ACCEPT > > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 24007 -m conntrack > --ctstate > NEW -j ACCEPT > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 24007 -m conntrack --ctstate > NEW -j ACCEPT > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 22 -m conntrack --ctstate > NEW > -j ACCEPT > > -A INPUT -s 10.3.7.0/25 -p tcp -m tcp --dport 22 -m conntrack --ctstate > NEW > -j ACCEPT > > -A INPUT -j REJECT --reject-with icmp-host-prohibited > > -A FORWARD -j REJECT --reject-with icmp-host-prohibited > > COMMIT > > # Completed on Tue Apr 23 09:15:14 2019 > > > > From time to time, when this server, say 10.3.3.3, trying to connect to > port > 24007 of another server 10.3.3.4, it will fail sometimes. from tcpdump > output, you can see these packages: > > 22:49:05.992737 IP 10.3.3.3.49149 > 10.3.3.4.24007: Flags [S], seq > 2454712274, win 29200, options [mss 1460,sackOK,TS val 24055648 ecr > 0,nop,wscale 7], length 0 > > 22:49:05.992847 IP 10.3.3.4.24007 > 10.3.3.3.49149: Flags [S.], seq > 3127562073, ack 2454712275, win 28960, options [mss 1460,sackOK,TS val > 17803660 ecr 24055648,nop,wscale 7], length 0 > > 22:49:05.992872 IP 10.3.3.3 > 10.3.3.4: ICMP host 10.3.3.3 unreachable - > admin prohibited, length 68 > > >So it looks like the packet getting rejected wasn't just a SYN packet but had something extra tagged NONE tagged in it [S.]. Do the packets which don't trigger an ICMP return contain a [SA] or a [S.] ? My guess is that something in that return packet is making the cstate not be NEW or ESTABLISHED and so it is failing down to the drop. If the packets which are accepted look the same then I would look to see if the conntrack table is overflowing.. if the memory is tight and the traffic is high, streams will get pushed out and you will get similar failures. That however usally puts something in dmesg when it happens. Sorry I don't have a better answer and I am looking forward to someone correcting my mistakes here :).> > The package back from 10.3.3.4 is prohibited by 10.3.3.3, why? > > Obviously, the package that's been prohibited is the SYNC package that in > response of the SYNC sent out by 10.3.3.3, it should match the iptables > rule > -m state --state RELATED,ESTABLISHED -j ACCEPT, but still be prohibited by > the rule -j REJECT --reject-with icmp-host-prohibited. > > > > Also, the iptables -nvL output: > > > > # iptables -nvL > > Chain INPUT (policy ACCEPT 0 packets, 0 bytes) > > pkts bytes target prot opt in out source > destination > > 8258 615K ACCEPT all -- * * 0.0.0.0/0 > 0.0.0.0/0 state RELATED,ESTABLISHED > > 1 84 ACCEPT icmp -- * * 0.0.0.0/0 > 0.0.0.0/0 > > 0 0 ACCEPT all -- lo * 0.0.0.0/0 > 0.0.0.0/0 > > 0 0 ACCEPT tcp -- * * 172.22.0.0/16 > 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW > > 0 0 ACCEPT tcp -- * * 10.3.3.0/25 > 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW > > 0 0 ACCEPT tcp -- * * 172.22.0.0/16 > 0.0.0.0/0 tcp dpt:24007 ctstate NEW > > 918 55080 ACCEPT tcp -- * * 10.3.3.0/25 > 0.0.0.0/0 tcp dpt:24007 ctstate NEW > > 1 60 ACCEPT tcp -- * * 10.3.3.0/25 > 0.0.0.0/0 tcp dpt:22 ctstate NEW > > 0 0 ACCEPT tcp -- * * 10.3.7.0/25 > 0.0.0.0/0 tcp dpt:22 ctstate NEW > > 244 14640 REJECT all -- * * 0.0.0.0/0 > 0.0.0.0/0 reject-with icmp-host-prohibited > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) > > pkts bytes target prot opt in out source > destination > > 0 0 REJECT all -- * * 0.0.0.0/0 > 0.0.0.0/0 reject-with icmp-host-prohibited > > > > Chain OUTPUT (policy ACCEPT 7378 packets, 957K bytes) > > pkts bytes target prot opt in out source > destination > > > > I noticed that the 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited > one > increase fast. > > > > Is this a bug of iptables, or is there something that I have not noticed? > > Any hints will be appreciated. > > Likun > > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >-- Stephen J Smoogen.
Hello, Stephen, thank you for input. Yes, these servers have the same firewall rules, and both of them have the same problem from time to time, most of time they are good. Actually, these servers are newly installed to be used as the Glusterfs storage server, so not much data flowing at this time.>From the sysctl output, I suppose it can't be a conntrack table overflow :net.netfilter.nf_conntrack_count = 1116 net.netfilter.nf_conntrack_max = 262144 And another tcpdump ouput of a successful ssh connection between these two servers for reference: 21:41:53.225977 IP (tos 0x0, ttl 64, id 30083, offset 0, flags [DF], proto TCP (6), length 60) 10.3.3.3.49221 > 10.3.3.4.22: Flags [S], cksum 0x1ab0 (incorrect -> 0x62bc), seq 320483660, win 29200, options [mss 1460,sackOK,TS val 253838543 ecr 0,nop,wscale 7], length 0 21:41:53.226083 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.3.3.4.22 > 10.3.3.3.49221: Flags [S.], cksum 0x15da (correct), seq 3543971827, ack 320483661, win 28960, options [mss 1460,sackOK,TS val 94551278 ecr 253838543,nop,wscale 7], length 0 21:41:53.226116 IP (tos 0x0, ttl 64, id 30084, offset 0, flags [DF], proto TCP (6), length 52) 10.3.3.3.49221 > 10.3.3.4.22: Flags [.], cksum 0x1aa8 (incorrect -> 0xb4e1), seq 1, ack 1, win 229, options [nop,nop,TS val 253838543 ecr 94551278], length 0 Likun -----????----- ???: CentOS [mailto:centos-bounces at centos.org] ?? Stephen John Smoogen ????: 2019?4?24? 18:35 ???: CentOS mailing list ??: Re: [CentOS] Iptables blocks out going connetion some times On Wed, 24 Apr 2019 at 06:01, likun <kun.li at ucarinc.com> wrote:> Hi guys. > > There is a wierd problem with iptables recently, hopes somebody can > help me. > > I have installed Centos 7.2.1511 on a bare metal Dell server these > days, disabled firewalld and enabled iptables.services, and setup a > group of very simple rules, as the following: > >I believe you are saying both 10.3.3.3 and 10.3.3.4 have this same firewall but I am not sure.> # iptables-save > > # Generated by iptables-save v1.4.21 on Tue Apr 23 09:15:14 2019 > > *filter > > :INPUT ACCEPT [0:0] > > :FORWARD ACCEPT [0:0] > > :OUTPUT ACCEPT [2449555:327804572] > > -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT > > -A INPUT -p icmp -j ACCEPT > > -A INPUT -i lo -j ACCEPT > > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 49152:49664 -m > conntrack --ctstate NEW -j ACCEPT > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 49152:49664 -m conntrack > --ctstate NEW -j ACCEPT > > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 24007 -m conntrack > --ctstate NEW -j ACCEPT > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 24007 -m conntrack > --ctstate NEW -j ACCEPT > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 22 -m conntrack > --ctstate NEW -j ACCEPT > > -A INPUT -s 10.3.7.0/25 -p tcp -m tcp --dport 22 -m conntrack > --ctstate NEW -j ACCEPT > > -A INPUT -j REJECT --reject-with icmp-host-prohibited > > -A FORWARD -j REJECT --reject-with icmp-host-prohibited > > COMMIT > > # Completed on Tue Apr 23 09:15:14 2019 > > > > From time to time, when this server, say 10.3.3.3, trying to connect > to port > 24007 of another server 10.3.3.4, it will fail sometimes. from tcpdump > output, you can see these packages: > > 22:49:05.992737 IP 10.3.3.3.49149 > 10.3.3.4.24007: Flags [S], seq > 2454712274, win 29200, options [mss 1460,sackOK,TS val 24055648 ecr > 0,nop,wscale 7], length 0 > > 22:49:05.992847 IP 10.3.3.4.24007 > 10.3.3.3.49149: Flags [S.], seq > 3127562073, ack 2454712275, win 28960, options [mss 1460,sackOK,TS val > 17803660 ecr 24055648,nop,wscale 7], length 0 > > 22:49:05.992872 IP 10.3.3.3 > 10.3.3.4: ICMP host 10.3.3.3 unreachable > - admin prohibited, length 68 > > >So it looks like the packet getting rejected wasn't just a SYN packet but had something extra tagged NONE tagged in it [S.]. Do the packets which don't trigger an ICMP return contain a [SA] or a [S.] ? My guess is that something in that return packet is making the cstate not be NEW or ESTABLISHED and so it is failing down to the drop. If the packets which are accepted look the same then I would look to see if the conntrack table is overflowing.. if the memory is tight and the traffic is high, streams will get pushed out and you will get similar failures. That however usally puts something in dmesg when it happens. Sorry I don't have a better answer and I am looking forward to someone correcting my mistakes here :).> > The package back from 10.3.3.4 is prohibited by 10.3.3.3, why? > > Obviously, the package that's been prohibited is the SYNC package that > in response of the SYNC sent out by 10.3.3.3, it should match the > iptables rule -m state --state RELATED,ESTABLISHED -j ACCEPT, but > still be prohibited by the rule -j REJECT --reject-with > icmp-host-prohibited. > > > > Also, the iptables -nvL output: > > > > # iptables -nvL > > Chain INPUT (policy ACCEPT 0 packets, 0 bytes) > > pkts bytes target prot opt in out source > destination > > 8258 615K ACCEPT all -- * * 0.0.0.0/0 > 0.0.0.0/0 state RELATED,ESTABLISHED > > 1 84 ACCEPT icmp -- * * 0.0.0.0/0 > 0.0.0.0/0 > > 0 0 ACCEPT all -- lo * 0.0.0.0/0 > 0.0.0.0/0 > > 0 0 ACCEPT tcp -- * * 172.22.0.0/16 > 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW > > 0 0 ACCEPT tcp -- * * 10.3.3.0/25 > 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW > > 0 0 ACCEPT tcp -- * * 172.22.0.0/16 > 0.0.0.0/0 tcp dpt:24007 ctstate NEW > > 918 55080 ACCEPT tcp -- * * 10.3.3.0/25 > 0.0.0.0/0 tcp dpt:24007 ctstate NEW > > 1 60 ACCEPT tcp -- * * 10.3.3.0/25 > 0.0.0.0/0 tcp dpt:22 ctstate NEW > > 0 0 ACCEPT tcp -- * * 10.3.7.0/25 > 0.0.0.0/0 tcp dpt:22 ctstate NEW > > 244 14640 REJECT all -- * * 0.0.0.0/0 > 0.0.0.0/0 reject-with icmp-host-prohibited > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) > > pkts bytes target prot opt in out source > destination > > 0 0 REJECT all -- * * 0.0.0.0/0 > 0.0.0.0/0 reject-with icmp-host-prohibited > > > > Chain OUTPUT (policy ACCEPT 7378 packets, 957K bytes) > > pkts bytes target prot opt in out source > destination > > > > I noticed that the 0.0.0.0/0 0.0.0.0/0 reject-with > icmp-host-prohibited one increase fast. > > > > Is this a bug of iptables, or is there something that I have not noticed? > > Any hints will be appreciated. > > Likun > > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >-- Stephen J Smoogen. _______________________________________________ CentOS mailing list CentOS at centos.org https://lists.centos.org/mailman/listinfo/centos