I''m running shorewall-perl 4.0.6 on a Debian Etch box (2.6.18-5-k7 kernel, iptables 1.3.6). In general, Shorewall appears to run just fine. However, when I look at the logs, I see a lot of outgoing TCP traffic with the ACK flag set (and with the source port equal to a port where I''m running a service on the firewall). Some examples: Feb 24 12:06:44 whodunit kernel: fp=fw2int:3 a=ACCEPT IN= OUT=int0 SRC=172.29.0.53 DST=172.29.0.3 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=61586 DF PROTO=TCP SPT=3128 DPT=4959 WINDOW=6940 RES=0x00 ACK URGP=0 Feb 24 19:01:44 whodunit kernel: fp=fw2ext:3 a=ACCEPT IN= OUT=ext0 SRC=171.66.155.243 DST=71.106.254.194 LEN=94 TOS=0x00 PREC=0x00 TTL=64 ID=45863 DF PROTO=TCP SPT=25 DPT=2622 WINDOW=5840 RES=0x00 ACK PSH FIN URGP=0 (The first example above is presumably related to the Squid proxy, on port 3128, which I''m running for my local network. The other example presumably involves SMTP somehow.) What might be causing these? Should I be worried? What system settings or Shorewall options (if any) should I take a closer look at? -- Rich Wales === Palo Alto, CA, USA === richw@richw.org http://www.richw.org === http://en.wikipedia.org/wiki/User:Richwales ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
On Sun, Feb 24, 2008 at 08:52:45PM -0800, Rich Wales wrote:> I''m running shorewall-perl 4.0.6 on a Debian Etch box (2.6.18-5-k7 kernel, > iptables 1.3.6). >Incidentally, which packages are you using? Are you using the packages from my personal repository? I have just updated them to 4.0.9.> In general, Shorewall appears to run just fine. However, when I look at > the logs, I see a lot of outgoing TCP traffic with the ACK flag set (and > with the source port equal to a port where I''m running a service on the > firewall). > > Some examples: > > Feb 24 12:06:44 whodunit kernel: fp=fw2int:3 a=ACCEPT IN= OUT=int0 > SRC=172.29.0.53 DST=172.29.0.3 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 > ID=61586 DF PROTO=TCP SPT=3128 DPT=4959 WINDOW=6940 RES=0x00 ACK URGP=0 > > Feb 24 19:01:44 whodunit kernel: fp=fw2ext:3 a=ACCEPT IN= OUT=ext0 > SRC=171.66.155.243 DST=71.106.254.194 LEN=94 TOS=0x00 PREC=0x00 > TTL=64 ID=45863 DF PROTO=TCP SPT=25 DPT=2622 WINDOW=5840 RES=0x00 > ACK PSH FIN URGP=0 > > (The first example above is presumably related to the Squid proxy, on > port 3128, which I''m running for my local network. The other example > presumably involves SMTP somehow.) > > What might be causing these? Should I be worried? What system settings > or Shorewall options (if any) should I take a closer look at? >I am not sure without seeing a dump. Maybe Tom has seen this before and can make an educated guess, but I cannot. Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Roberto C. Sánchez wrote:> On Sun, Feb 24, 2008 at 08:52:45PM -0800, Rich Wales wrote:>> What might be causing these? Should I be worried? What system settings >> or Shorewall options (if any) should I take a closer look at? >> > I am not sure without seeing a dump. Maybe Tom has seen this before and > can make an educated guess, but I cannot.It''s impossible to interpret a few log messages completely out of context. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Tom Eastep wrote:> It''s impossible to interpret a few log messages completely out of context.Fair enough. See the attached status.txt.gz (output of "shorewall dump"). -- Rich Wales === Palo Alto, CA, USA === richw@richw.org http://www.richw.org === http://en.wikipedia.org/wiki/User:Richwales ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Rich Wales wrote:> Tom Eastep wrote: > >> It''s impossible to interpret a few log messages completely out of >> context. > > Fair enough. See the attached status.txt.gz (output of "shorewall dump"). >I''ll try to look at it over the next couple of days. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Rich Wales wrote:> I''m running shorewall-perl 4.0.6 on a Debian Etch box (2.6.18-5-k7 kernel, > iptables 1.3.6). > > In general, Shorewall appears to run just fine. However, when I look at > the logs, I see a lot of outgoing TCP traffic with the ACK flag set (and > with the source port equal to a port where I''m running a service on the > firewall). > > Some examples: > > Feb 24 12:06:44 whodunit kernel: fp=fw2int:3 a=ACCEPT IN= OUT=int0 > SRC=172.29.0.53 DST=172.29.0.3 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 > ID=61586 DF PROTO=TCP SPT=3128 DPT=4959 WINDOW=6940 RES=0x00 ACK URGP=0 > > Feb 24 19:01:44 whodunit kernel: fp=fw2ext:3 a=ACCEPT IN= OUT=ext0 > SRC=171.66.155.243 DST=71.106.254.194 LEN=94 TOS=0x00 PREC=0x00 > TTL=64 ID=45863 DF PROTO=TCP SPT=25 DPT=2622 WINDOW=5840 RES=0x00 > ACK PSH FIN URGP=0 > > (The first example above is presumably related to the Squid proxy, on > port 3128, which I''m running for my local network. The other example > presumably involves SMTP somehow.)I''ve seen this before on quite a few systems. In my experience, it''s seen on perfectly legitimate traffic that netfilter''s conntrack somehow loses track of. I''m sure Tom will be able to give you more information about why this happens when he looks at the dump, but my guesses are netfilter running out of memory (are those log messages really garbled like that?), slow links and/or busy servers, and (maybe?) non-compliant TCP stacks out there. Paul ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
On Mon, Feb 25, 2008 at 04:44:16PM +1000, Paul Gear wrote:> I''ve seen this before on quite a few systems. In my experience, it''s > seen on perfectly legitimate traffic that netfilter''s conntrack somehow > loses track of. > > I''m sure Tom will be able to give you more information about why this > happens when he looks at the dump, but my guesses are netfilter running > out of memory (are those log messages really garbled like that?), slow > links and/or busy servers, and (maybe?) non-compliant TCP stacks out there.Other notable sources of netfilter losing track of connections include: - broken routing, especially assymetric routing - out-of-order packet delivery at the end of a connection (relatively common on Internet links) - other NAT devices that are broken (pretty much every consumer DSL ''router'') - kernel bugs (usually in protocol-specific conntrack helpers) ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Andrew Suffield wrote:> On Mon, Feb 25, 2008 at 04:44:16PM +1000, Paul Gear wrote: > >> I''ve seen this before on quite a few systems. In my experience, it''s >> seen on perfectly legitimate traffic that netfilter''s conntrack somehow >> loses track of. >> >> I''m sure Tom will be able to give you more information about why this >> happens when he looks at the dump, but my guesses are netfilter running >> out of memory (are those log messages really garbled like that?), slow >> links and/or busy servers, and (maybe?) non-compliant TCP stacks out there. >> > > Other notable sources of netfilter losing track of connections include: > > - broken routing, especially assymetric routing > - out-of-order packet delivery at the end of a connection (relatively > common on Internet links) > - other NAT devices that are broken (pretty much every consumer DSL > ''router'') > - kernel bugs (usually in protocol-specific conntrack helpers) >If I''m not mistaken, there was a netfilter bug about reopened connections that were lost by conntrack, post 2.6.22.x I guess ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Rich Wales wrote:> I''m running shorewall-perl 4.0.6 on a Debian Etch box (2.6.18-5-k7 kernel, > iptables 1.3.6). > > In general, Shorewall appears to run just fine. However, when I look at > the logs, I see a lot of outgoing TCP traffic with the ACK flag set (and > with the source port equal to a port where I''m running a service on the > firewall). > > Some examples: > > Feb 24 12:06:44 whodunit kernel: fp=fw2int:3 a=ACCEPT IN= OUT=int0 > SRC=172.29.0.53 DST=172.29.0.3 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 > ID=61586 DF PROTO=TCP SPT=3128 DPT=4959 WINDOW=6940 RES=0x00 ACK URGP=0 > > Feb 24 19:01:44 whodunit kernel: fp=fw2ext:3 a=ACCEPT IN= OUT=ext0 > SRC=171.66.155.243 DST=71.106.254.194 LEN=94 TOS=0x00 PREC=0x00 > TTL=64 ID=45863 DF PROTO=TCP SPT=25 DPT=2622 WINDOW=5840 RES=0x00 > ACK PSH FIN URGP=0 >Shorewall sends packets in the INVALID state through the rules given in the NEW section of the rules file. This allows for ''connection pickup'' whereby the conntrack table can be cleared and existing connections can be re-entered in the table. My personal approach is to include this rule early in the NEW section: dropNotSyn net all tcp So incoming INVALID tcp packets get dropped. Outgoing packets aren''t similarly restricted in my setup. In your case, you are treating these INVALID packets the same as NEW state packets, including logging them. Note that the standard Drop and Reject actions don''t log packets in the INVALID state, regardless of protocol -- both of those actions invoke the dropInvalid action prior to logging to avoid log polution and to avoid 100''s of questions just like yours. As others have noted, out-of-order/invalid packets are not rare -- they happen frequently. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Thanks for your feedback, Andrew, Paul, and Tom. I seriously doubt there''s an out-of-memory issue here; my firewall box has 3 GB of RAM and isn''t even coming close to using it all. The stray packets with source port 25 (with ACK, PSH, and FIN flags) seem to correspond to scenarios where the SMTP service on my firewall rejected an incoming e-mail message after the RCPT -- e.g., because the destination address is unknown on my system, or because the source host is being blacklisted by Spamhaus -- and the client responded by abruptly disconnecting, without first issuing an SMTP QUIT command. So I assume I can probably safely drop these stray packets (per Tom''s suggestion) without breaking anything on my system. The clients on my home LAN generating the stray packets with source port 3128 (Squid) -- with the ACK flag, and sometimes PSH, and sometimes also FIN -- are running Windows XP Home and (in case it matters) are using the Firefox browser. Are there any known problems with the WinXP TCP stack that might account for these stray packets? If I follow Tom''s advice and drop these packets, am I likely to break anything on my family''s systems (e.g., will they start seeing browsing attempts time out or get rejected that are currently working OK)? Or is the mere fact that these packets aren''t being properly tracked as part of established connections going to mean that they are doomed already and that dropping them can''t do any further harm? -- Rich Wales === Palo Alto, CA, USA === richw@richw.org http://www.richw.org === http://en.wikipedia.org/wiki/User:Richwales ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
On Mon, Feb 25, 2008 at 09:22:22AM -0800, Rich Wales wrote:> I seriously doubt there''s an out-of-memory issue here; my firewall box > has 3 GB of RAM and isn''t even coming close to using it all.Netfilter has a memory limit set somewhere in /proc or /sys, it won''t use all available physical memory by default.> The stray packets with source port 25 (with ACK, PSH, and FIN flags) > seem to correspond to scenarios where the SMTP service on my firewall > rejected an incoming e-mail message after the RCPT -- e.g., because > the destination address is unknown on my system, or because the source > host is being blacklisted by Spamhaus -- and the client responded by > abruptly disconnecting, without first issuing an SMTP QUIT command.Then they are almost certainly just late packets from that session - the packet got held in a buffer on some upper-level internet router, the two peers treated the packet as dropped and resent it, and then the router finally delivered it later, after the connection had been torn down. It happens; internet routing is unreliable.> If I follow Tom''s > advice and drop these packets, am I likely to break anything on my > family''s systems (e.g., will they start seeing browsing attempts time > out or get rejected that are currently working OK)? Or is the mere fact > that these packets aren''t being properly tracked as part of established > connections going to mean that they are doomed already and that dropping > them can''t do any further harm?Yes, they''re already dead; they may possibly indicate the existence of some other problem, but you can''t make it worse by dropping them. ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/