Last week, I started seeing very strange behavior in one of the networks that I manage. The office LAN uses a Linux firewall (running shorewall) which masquerades their workstations over their DSL connection. There are probably ~75 workstations in the office LAN. Their mail server is in a collocated facility nearby. That server has an RFC1918 address; its router does SNAT to forward packets to the system. Both the office firewall and the mail server are currently running fully patched CentOS 5.2. Here''s the weird part: If a machine running Linux in the office lan attempts to connect to the mail server on any TCP port, there''s a small chance that the server will simply ignore the SYN packets. It doesn''t log any errors. If I''m running tcpdump, I see the incoming SYN packets, but no reply. If I use iptables to log the packets, information about the packet is saved in the messages file. If I capture the packets and use wireshark to analyze them, I don''t see anything odd: the checksums are good and I can''t see any difference between a SYN packet that gets a SYN+ACK and one that''s ignored (beyond the obvious: different timestamps and checksums). The problem doesn''t seem to affect Windows workstations in the office LAN. As far as I can tell, only SYN packets are dropped. I don''t see delays in established connections. I''m unable to reproduce the problem from any machine other than a Linux host which is inside the office network, using the shorewall firewall as its gateway. I''ve attached a file that contains, first, the output of tcpdump which shows packets to or from the office''s firewall address, as recorded by the destination server. The first four SYN packets are ignored, but the kernel proceeds with the TCP handshake after the fifth SYN packet. Second, the file contains the log messages which are recorded as a result of these iptables rules: iptables -A INPUT -p tcp -s officefw --dport 22 -j LOG iptables -A INPUT -p tcp -s officefw --dport 22 -j ACCEPT Those are the only iptables rules present on the server accepting the connections. Both of those appear to indicate that the server in the colo facility is receiving the SYN packets. What possible reasons are there that it would not reply with SYN+ACK? ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
Gordon Messmer wrote:> > Both of those appear to indicate that the server in the colo facility is > receiving the SYN packets. What possible reasons are there that it > would not reply with SYN+ACK? >This is all fascinating but I can''t, in my worse nightmares, think that this has anything to do with Shorewall. -tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
Tom Eastep wrote:> > This is all fascinating but I can''t, in my worse nightmares, think that > this has anything to do with Shorewall.I''m having a hard time with that, too, except that I can only reproduce the problem from inside that office LAN, which uses the Linux router running shorewall. I guess I was hoping that someone with enough knowledge of Linux''s IP stack would pipe in. I''ve never seen anything remotely like it. ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
Gordon Messmer wrote:> Tom Eastep wrote: >> This is all fascinating but I can''t, in my worse nightmares, think that >> this has anything to do with Shorewall. > > I''m having a hard time with that, too, except that I can only reproduce > the problem from inside that office LAN, which uses the Linux router > running shorewall.It doesn''t "run shorewall" -- Shorewall is used to configure Netfilter on the box; once ''shorewall start'' is finished, there is no Shorewall code running there at all. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
Tom Eastep wrote:> > It doesn''t "run shorewall" -- Shorewall is used to configure Netfilter > on the box; once ''shorewall start'' is finished, there is no Shorewall > code running there at all.I know, I understand how it works. I just didn''t phrase that very well, and I apologize. If you were looking for help with a problem of this nature, who would you ask? ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
Does this problem only occur with SSH or do you see it happening with other applications? When running tcpdump did you look at all traffic on the server or only the traffic you were interested in? -Tom Gordon Messmer wrote:> Tom Eastep wrote: >> It doesn''t "run shorewall" -- Shorewall is used to configure Netfilter >> on the box; once ''shorewall start'' is finished, there is no Shorewall >> code running there at all. > > I know, I understand how it works. I just didn''t phrase that very well, > and I apologize. > > If you were looking for help with a problem of this nature, who would > you ask? > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users-- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
i had a similar problem and i solved it by turning off the tcp_window_scaling(/proc/sys/net/ipv4/tcp_window_scaling). our linux-server was not able to communicate to some servers. i tried to turn off the tcp_window_scaling on our linux-server and then it works. try to turn off the window-scaling on your linux clients, maybe you are lucky.. ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/