I''ve been successfully using shorewall to support dnat and snat for my configuration ever since an ancient cisco pix died 2 years ago. This linux firewall (hostname kerner) runs ubuntu 12.04 LTS, and so has the 3.2.0-32.51-generic kernel, iptables 1.4.12 and shorewall 4.4.26.1. Until a few weeks ago my static internet subnet (mask 255.255.255.240) was hosted by my ADSL router, running routertech linux-based firmware. I had been hand-crafting iptables rules for the router to act as a coarse filter for incoming traffic to my internet-facing hosts, but it did not do any natting. I decided I needed to use blacklisting in the coarse filter and realised that iptables maintenance would have become too arduous. I decided to move my static subnet onto a second linux system, which runs shorewall and pppd, so now the ADSL router runs as a simple pppoe bridge to my single ISP. This coarse filter system (hostname chenin) has been running well for the last 3 weeks. It has ubuntu 12.10, with a custom built 3.5.0-28-generic kernel, iptables 1.4.12 and shorewall 4.5.5.3. I started to add blacklisting rules: forbidden ports to identify attackers, then forbidden hosts to DROP all traffic from known attackers. As the kernel iptables log files became less noisy, I noticed clusters of messages reporting the arrival of martians from my original natting firewall. The source addresses corresponded to only two hosts on my safe lan, both with 10.1.252.x addresses. I ran wireshark (with various capture filters because there is a LOT of legitimate traffic) on both firewalls. There was no evidence of the martians on the natting system because ALL traffic had source addresses on my safe lan - the nat mechanism appears to happen downstream of the wireshark probe point. The martians appeared on the inside (static internet subnet) interface of the coarse filter firewall, along with legitimate traffic from the same hosts (my capture filter here selected the destination host), i.e. the same source host communications to the same internet server included a mixture of natted and martian packets. I started reading the shorewall FAQ and troubleshooting guides, as well general googling. The best advice seemed to check the status of reverse path filtering - rp_filter is set to 1 on both interfaces of both firewalls when shorewall status is "running". I don''t think I have a problem with my shorewall rules on the natting system, but I wonder whether the kernel is actually using the rp_filter logic. I would have expected the natted outbound martian packets to be dropped at source, but perhaps rp_filter logic happens before nat? I don''t want to drag anyone into my problem at this stage (which is why I have not attached all the appropriate configurations). I feel I have to bring the kernel and shorewall (on the natting firewall) more up to date first, then see whether the problem goes away. I will schedule the upgrade to 12.10 for some time in the next week. My main reasons for making this initial post are to document the current symptoms and see whether this is a well-known problem that I''ve failed to discover with my searches. I don''t want to waste anyone''s time (yet!) So (I hear you asking)... what are the martian packets? Here are my preliminary observations: 1. The two client source hosts are running ubuntu desktop 12.04 and 12.10 (see kernel notes above). They each have single network interfaces and have totally-open iptables. 2. The traffic associated with martians appears to be http and https from firefox. With parallel connections and many tabs open, there can be a lot of traffic to analyse! 3. Clusters of martians /seem/ to be associated with these machines "coming back from idle". The user of one system (a notepad) tends to close the lid, which suspends the system - martians appear when the lid is opened and the system resumes. The other system is a desktop which is never suspended, but the martians appear when the user reauthenticates a timer-locked session. (I am unsure whether martians originate from these two systems at times other than reauthentication). 4. My installation has several other systems, but I have not yet noticed martians originating from any of them. 5. The small sample of useful traces I have analysed show all the martian packets are FIN/ACK''s associated with source ports different to those carrying properly natted data. The FIN/ACK''s are retried with exponentially increasing timeouts until the connection presumably gets thrown away. (Because the packets are martians, the coarse filter firewall is discarding them, so the remote server never sees them to have an opinion about replying). 6. The remote hosts are legitimate web servers, so there is no malicious traffic involved. I suppose these web servers (e.g. facebook) run javascript on the browsers to keep event notification sessions alive. The firewall conntrack entries probably expire when the client systems become idle. I realise I could just shrug and ignore the martians. They are being detected and discarded, so they are doing no harm. However, I want to understand what is going on first, just in case it turns out to be a manifestation of a more significant problem. Thanks for an excellent product... and for listening, Brian ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j
On Jun 4, 2013, at 3:08 AM, Brian Burch <brian@pingtoo.com> wrote:> > So (I hear you asking)... what are the martian packets? Here are my > preliminary observations: > > 1. The two client source hosts are running ubuntu desktop 12.04 and > 12.10 (see kernel notes above). They each have single network interfaces > and have totally-open iptables. > > 2. The traffic associated with martians appears to be http and https > from firefox. With parallel connections and many tabs open, there can be > a lot of traffic to analyse! > > 3. Clusters of martians /seem/ to be associated with these machines > "coming back from idle". The user of one system (a notepad) tends to > close the lid, which suspends the system - martians appear when the lid > is opened and the system resumes. The other system is a desktop which is > never suspended, but the martians appear when the user reauthenticates a > timer-locked session. (I am unsure whether martians originate from these > two systems at times other than reauthentication). > > 4. My installation has several other systems, but I have not yet noticed > martians originating from any of them. > > 5. The small sample of useful traces I have analysed show all the > martian packets are FIN/ACK''s associated with source ports different to > those carrying properly natted data. The FIN/ACK''s are retried with > exponentially increasing timeouts until the connection presumably gets > thrown away. (Because the packets are martians, the coarse filter > firewall is discarding them, so the remote server never sees them to > have an opinion about replying). > > 6. The remote hosts are legitimate web servers, so there is no malicious > traffic involved. I suppose these web servers (e.g. facebook) run > javascript on the browsers to keep event notification sessions alive. > The firewall conntrack entries probably expire when the client systems > become idle. > > I realise I could just shrug and ignore the martians. They are being > detected and discarded, so they are doing no harm. However, I want to > understand what is going on first, just in case it turns out to be a > manifestation of a more significant problem.Brian, It sounds like the inner firewall is receiving FIN/ACKs for source/dest pairs for which the corresponding conntrack entry has been discarded. Such packets are considered to be in the INVALID connection tracking state, and hence do not get NATted. A solution on the inner firewall is to add this rule early in your rules file: Invalid(DROP) all all Note that if you have this rule in place already Invalid(DROP) net all then simply replace it with the one above. Or you can simply disable reverse path filtering (routefilter) on the outer firewall''s local interface. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j
On 04/06/13 14:22, Tom Eastep wrote:> > On Jun 4, 2013, at 3:08 AM, Brian Burch <brian@pingtoo.com> wrote: > >> >> So (I hear you asking)... what are the martian packets? Here are my >> preliminary observations: >> >> 1. The two client source hosts are running ubuntu desktop 12.04 and >> 12.10 (see kernel notes above). They each have single network interfaces >> and have totally-open iptables. >> >> 2. The traffic associated with martians appears to be http and https >> from firefox. With parallel connections and many tabs open, there can be >> a lot of traffic to analyse! >> >> 3. Clusters of martians /seem/ to be associated with these machines >> "coming back from idle". The user of one system (a notepad) tends to >> close the lid, which suspends the system - martians appear when the lid >> is opened and the system resumes. The other system is a desktop which is >> never suspended, but the martians appear when the user reauthenticates a >> timer-locked session. (I am unsure whether martians originate from these >> two systems at times other than reauthentication). >> >> 4. My installation has several other systems, but I have not yet noticed >> martians originating from any of them. >> >> 5. The small sample of useful traces I have analysed show all the >> martian packets are FIN/ACK''s associated with source ports different to >> those carrying properly natted data. The FIN/ACK''s are retried with >> exponentially increasing timeouts until the connection presumably gets >> thrown away. (Because the packets are martians, the coarse filter >> firewall is discarding them, so the remote server never sees them to >> have an opinion about replying). >> >> 6. The remote hosts are legitimate web servers, so there is no malicious >> traffic involved. I suppose these web servers (e.g. facebook) run >> javascript on the browsers to keep event notification sessions alive. >> The firewall conntrack entries probably expire when the client systems >> become idle. >> >> I realise I could just shrug and ignore the martians. They are being >> detected and discarded, so they are doing no harm. However, I want to >> understand what is going on first, just in case it turns out to be a >> manifestation of a more significant problem. > > Brian, > > It sounds like the inner firewall is receiving FIN/ACKs for source/dest pairs for which the corresponding conntrack entry has been discarded. Such packets are considered to be in the INVALID connection tracking state, and hence do not get NATted.Thanks for your quick and helpful reply, Tom.> A solution on the inner firewall is to add this rule early in your rules file: > > Invalid(DROP) all all > > Note that if you have this rule in place already > > Invalid(DROP) net all > > then simply replace it with the one above. Or you can simply disable reverse path filtering (routefilter) on the outer firewall''s local interface.I did not have anything similar in the rules file, so I added the all/all variant as the first entry on my natting firewall. After this change, the martians are no longer appearing at my outside firewall, even when the client systems have resumed after a long idle time, so I conclude that your suggestion has resolved my problem. The inner firewall is running shorewall 4.4, so I couldn''t help wondering whether your rule would work... the docs imply that Invalid (which isn''t a macro) was introduced with version 4.5. However, it does seem to work, although I don''t have templates for variables such as INVALID_DISPOSITION and INVALID_LOG_LEVEL in the shorewall.conf file. Forgive me being a bit lazy, but does "invalid" mean simply "not in the conntrack cache", or does it have a wider definition that I ought to be cautious about? Once again, thanks very much for your help. Brian> -Tom > > Tom Eastep \ When I die, I want to go like my Grandfather who > Shoreline, \ died peacefully in his sleep. Not screaming like > Washington, USA \ all of the passengers in his car > http://shorewall.net \________________________________________________ > > > > > ------------------------------------------------------------------------------ > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer high-level views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenow-d2d-j > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users >------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j
On 06/04/2013 07:26 AM, Brian Burch wrote:> I did not have anything similar in the rules file, so I added the > all/all variant as the first entry on my natting firewall. > > After this change, the martians are no longer appearing at my outside > firewall, even when the client systems have resumed after a long idle > time, so I conclude that your suggestion has resolved my problem. > > The inner firewall is running shorewall 4.4, so I couldn''t help > wondering whether your rule would work... the docs imply that Invalid > (which isn''t a macro) was introduced with version 4.5. However, it does > seem to work, although I don''t have templates for variables such as > INVALID_DISPOSITION and INVALID_LOG_LEVEL in the shorewall.conf file.Those options were added when the INVALID section of the rules file was created.> > Forgive me being a bit lazy, but does "invalid" mean simply "not in the > conntrack cache", or does it have a wider definition that I ought to be > cautious about?Invalid means that there is not a matching conntrack entry and one should not be created from that particular packet. In your case, a FIN/ACK should not be the first packet seen as part of a connection; that should rather be a SYN packet (with no ACK). -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j
On 04/06/13 15:46, Tom Eastep wrote:> On 06/04/2013 07:26 AM, Brian Burch wrote: > >> I did not have anything similar in the rules file, so I added the >> all/all variant as the first entry on my natting firewall. >> >> After this change, the martians are no longer appearing at my outside >> firewall, even when the client systems have resumed after a long idle >> time, so I conclude that your suggestion has resolved my problem. >> >> The inner firewall is running shorewall 4.4, so I couldn''t help >> wondering whether your rule would work... the docs imply that Invalid >> (which isn''t a macro) was introduced with version 4.5. However, it does >> seem to work, although I don''t have templates for variables such as >> INVALID_DISPOSITION and INVALID_LOG_LEVEL in the shorewall.conf file. > > Those options were added when the INVALID section of the rules file was > created. > >> >> Forgive me being a bit lazy, but does "invalid" mean simply "not in the >> conntrack cache", or does it have a wider definition that I ought to be >> cautious about? > > Invalid means that there is not a matching conntrack entry and one > should not be created from that particular packet. In your case, a > FIN/ACK should not be the first packet seen as part of a connection; > that should rather be a SYN packet (with no ACK).Thanks for explaining. Now my wireshark traces make perfect sense. As the FIN/ACK retry intervals extend to several seconds for a particular client source port, I see new SYN 3-way connection initiation on a new client port to the same external web server - just the right trigger to create a new conntrack entry. Of course, there isn''t a linear progression because there are several asynchronous parallel connections. If I had a sufficiently comprehensive trace and made the effort to decrypt all the ssl streams, I expect I would be able to correlate each expiring FIN/ACK sequence with a new connection carrying the same class of payload. I am satisfied that I have killed my martians without causing any collateral damage! I hope this conversation will be helpful to others who notice similar log messages. Brian> > -Tom > > > > ------------------------------------------------------------------------------ > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer high-level views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenow-d2d-j > > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users >------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j