What experience have users had using ShoreWall as a bogon filter using the Team Cymru full bogon lists (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 full bogon list contains over 4,600 separate networks that need to be denied, and the IPv6 list over 68,300. Having not tried this myself, I would be concerned a priori about ShoreWall server meltdown. Jeffry A. Spain, Network Administrator Cincinnati Country Day School ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
The overhead associated with matching against the complete bogon list is too much in my humble opinion. Cory Oldford ----- Original Message ----- From: "Dr. Jeffry A. Spain" <spainj@countryday.net> To: "Shorewall Users" <shorewall-users@lists.sourceforge.net> Sent: Saturday, February 23, 2013 12:38:14 PM Subject: [Shorewall-users] Full Bogon Filtering What experience have users had using ShoreWall as a bogon filter using the Team Cymru full bogon lists (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 full bogon list contains over 4,600 separate networks that need to be denied, and the IPv6 list over 68,300. Having not tried this myself, I would be concerned a priori about ShoreWall server meltdown. Jeffry A. Spain, Network Administrator Cincinnati Country Day School ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
Using ipsets is the only way that I would try such a thing. -Tom On 2/23/13 10:57 AM, "Cory Oldford" <cory@peaceworks.ca> wrote:> The overhead associated with matching against the complete bogon list is too > much in my humble opinion. > > > Cory Oldford > > > From: "Dr. Jeffry A. Spain" <spainj@countryday.net> > To: "Shorewall Users" <shorewall-users@lists.sourceforge.net> > Sent: Saturday, February 23, 2013 12:38:14 PM > Subject: [Shorewall-users] Full Bogon Filtering > > What experience have users had using ShoreWall as a bogon filter using the > Team Cymru full bogon lists > (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 full bogon > list contains over 4,600 separate networks that need to be denied, and the > IPv6 list over 68,300. Having not tried this myself, I would be concerned a > priori about ShoreWall server meltdown. > > Jeffry A. Spain, Network Administrator > Cincinnati Country Day School > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. Make your web apps faster with > AppDynamics Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb____________________________________________ > ___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users-Tom You do not need a parachute to skydive. You only need a parachute to skydive twice. ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
Most likely your Internet provider already implements bogon filtering on their border routers. So filtering again on your firewall is a bit over the top. But that''s just my 2 cents Greets, Sander On 23 feb. 2013, at 19:38, "Spain, Dr. Jeffry A." <spainj@countryday.net> wrote:> What experience have users had using ShoreWall as a bogon filter using the Team Cymru full bogon lists (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 full bogon list contains over 4,600 separate networks that need to be denied, and the IPv6 list over 68,300. Having not tried this myself, I would be concerned a priori about ShoreWall server meltdown. > > Jeffry A. Spain, Network Administrator > Cincinnati Country Day School > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
> Most likely your Internet provider already implements bogon filtering on their border routers. So filtering again on your firewall is a bit over the top.I have a more skeptical view of our ISP, but perhaps I will ask them about it. Thanks to all for your insights. Jeff. ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
On 23/02/13 19:27, Tom Eastep wrote:> Using ipsets is the only way that I would try such a thing. > > -Tom > > On 2/23/13 10:57 AM, "Cory Oldford" <cory@peaceworks.ca > <mailto:cory@peaceworks.ca>> wrote: > > The overhead associated with matching against the complete bogon > list is too much in my humble opinion. > > > Cory Oldford > > ------------------------------------------------------------------------ > *From: *"Dr. Jeffry A. Spain" <spainj@countryday.net > <mailto:spainj@countryday.net>> > *To: *"Shorewall Users" <shorewall-users@lists.sourceforge.net > <mailto:shorewall-users@lists.sourceforge.net>> > *Sent: *Saturday, February 23, 2013 12:38:14 PM > *Subject: *[Shorewall-users] Full Bogon Filtering > > What experience have users had using ShoreWall as a bogon filter > using the Team Cymru full bogon lists > (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 > full bogon list contains over 4,600 separate networks that need to > be denied, and the IPv6 list over 68,300. Having not tried this > myself, I would be concerned a priori about ShoreWall server meltdown. > > Jeffry A. Spain, Network Administrator > Cincinnati Country Day School > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > <mailto:Shorewall-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/shorewall-users > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. Make your web apps faster > with AppDynamics Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb_______________________________________________ > Shorewall-users mailing list Shorewall-users@lists.sourceforge.net > <mailto:Shorewall-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/shorewall-users > > > > -Tom > You do not need a parachute to skydive. You only need a parachute to > skydive twice. > > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-usersI have tried this myself and I have to second what Tom said above, unfortunately I havn''t rewritten the shell scripts for it yet after losing them in a hard disk crash had other things needed resolving first however the actual update of the ipsets will cause a bit of a spike especially if your firewall system is a low end machine (An intel i7 hexacore was taking a minute or two to chew through it it) but I created a shell script to download the lists whenever they were modified, compare them and then update the ipsets with changes only. I have been intending to implement this again myself and it would probably take as long to write a detailed explanation as it would to just go ahead and create the scripts again with comments. I should have some free time this morning so I will try to get that done today and get back to you with the shell script and comments. ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
On 23.02.2013 22:40, Spain, Dr. Jeffry A. wrote:>> Most likely your Internet provider already implements bogon filtering >> on their border routers. So filtering again on your firewall is a bit >> over the top. > > I have a more skeptical view of our ISP, but perhaps I will ask them > about it. Thanks to all for your insights. Jeff.Even if your own provider isn''t doing any bogon filtering, other providers and most definitely all transit providers are doing bogon filtering. I''m not saying you shouldn''t implement it, but chances are very slim that it will ever help you. Moreover, the bogon list changes every now and the so you have to update them on your firewall. Greets, Sander ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
Matt Joyce
2013-Feb-26 22:22 UTC
ipset updating/filtering script (Was: Full Bogon Filtering)
On 24/02/13 08:09, Matt Joyce wrote:> On 23/02/13 19:27, Tom Eastep wrote: >> Using ipsets is the only way that I would try such a thing. >> >> -Tom >> >> On 2/23/13 10:57 AM, "Cory Oldford" <cory@peaceworks.ca >> <mailto:cory@peaceworks.ca>> wrote: >> >> The overhead associated with matching against the complete bogon >> list is too much in my humble opinion. >> >> >> Cory Oldford >> >> ------------------------------------------------------------------------ >> *From: *"Dr. Jeffry A. Spain" <spainj@countryday.net >> <mailto:spainj@countryday.net>> >> *To: *"Shorewall Users" <shorewall-users@lists.sourceforge.net >> <mailto:shorewall-users@lists.sourceforge.net>> >> *Sent: *Saturday, February 23, 2013 12:38:14 PM >> *Subject: *[Shorewall-users] Full Bogon Filtering >> >> What experience have users had using ShoreWall as a bogon filter >> using the Team Cymru full bogon lists >> (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 >> full bogon list contains over 4,600 separate networks that need >> to be denied, and the IPv6 list over 68,300. Having not tried >> this myself, I would be concerned a priori about ShoreWall server >> meltdown. >> >> Jeffry A. Spain, Network Administrator >> Cincinnati Country Day School >> >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. >> Make your web apps faster with AppDynamics >> Download AppDynamics Lite for free today: >> http://p.sf.net/sfu/appdyn_d2d_feb >> _______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> <mailto:Shorewall-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/shorewall-users >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. Make your web apps faster >> with AppDynamics Download AppDynamics Lite for free today: >> http://p.sf.net/sfu/appdyn_d2d_feb_______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> <mailto:Shorewall-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/shorewall-users >> >> >> >> -Tom >> You do not need a parachute to skydive. You only need a parachute to >> skydive twice. >> >> >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. >> Make your web apps faster with AppDynamics >> Download AppDynamics Lite for free today: >> http://p.sf.net/sfu/appdyn_d2d_feb >> >> >> _______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/shorewall-users > I have tried this myself and I have to second what Tom said above, > unfortunately I havn''t rewritten the shell scripts for it yet after > losing them in a hard disk crash had other things needed resolving > first however the actual update of the ipsets will cause a bit of a > spike especially if your firewall system is a low end machine (An > intel i7 hexacore was taking a minute or two to chew through it it) > but I created a shell script to download the lists whenever they were > modified, compare them and then update the ipsets with changes only. > I have been intending to implement this again myself and it would > probably take as long to write a detailed explanation as it would to > just go ahead and create the scripts again with comments. > > I should have some free time this morning so I will try to get that > done today and get back to you with the shell script and comments. > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-usersI have to apologize that things got a bit busy here and I forgot about this until this evening, I just worked on the script and tested it. It will provide a short summary after a run which I generally let cron mail to me so I know if the update stops working for any reason but you could just as easily redirect it to a file too. The script is attached with comments, note that this version works pretty quickly because it doesn''t go through and add them line by line every load, instead it compares the sorted new list against ipset list using comm to find added and deleted entries then only processes those, as a sanity check it will use md5sum if available to confirm the updated ipset matches checks the md5sum of the two lists match after updating provided the md5sum binary is installed. Few notes I thought I''d mention though the comments in the script discuss them too: Note that ipsets are lost on reboot but by default the script uses ipset save to save the ipset to it''s working directory, just need to use ipset restore before using it for filtering shorewall can handle this for you if you add something like the following to /etc/shorewall/start ipset -! restore -f /var/cache/bogon-lists/bogonsip4.ipset Yes you could just readd each time but ipset restore is a lot faster at it than a bash script. It''s your choice but I would suggest it may not hurt to consider my suggestion in there about using caps support if you have it available, that way you can avoid having a shell script running as root chewing on an externally sourced file which to me seemed a bit too much like asking for trouble. Do check the script before using it as there are a few other options you may wish to change. As a side note the script as is will work with any list of CIDR ranges that one might like to keep up to date and use for filtering in one per line format including files containing #comments whole or end of line, doesn''t care about whitespace either. It will need a small modification to work with ftp:// urls, currently it is written to expect a http_response code which is fine with these lists but might be an issue if anyone wanted to adapt it. Anyway, I''m happy to make freely available to anyone to use or modify as they see fit hope it''s useful to someone. Comments welcome if there is anything I missed somewhere. ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
John Brendler
2013-Feb-27 02:05 UTC
Re: ipset updating/filtering script (Was: Full Bogon Filtering)
On Tue, 26 Feb 2013 22:22:02 +0000 Matt Joyce <mjoyce@mttjocy.co.uk> wrote:> The script is attached with comments, note that this version works > pretty quickly because it doesn''t go through and add them line by line > every load, instead it compares the sorted new list against ipset list > using comm to find added and deleted entries then only processes > those, as a sanity check it will use md5sum if available to confirm > the updated ipset matches checks the md5sum of the two lists match > after updating provided the md5sum binary is installed.That''s a good idea about comparing the two as a means to save processing time. I have two questions: 1. Did you actually compare the time it takes to build an ipset from scratch (just by processing the text file line by line) versus the time it takes to make comparisons and modifications as you have it? What was the time differential? Did you also consider using ''diff'' and ''patch''? If you tried that, was there a difference? 2. In my script, I opted to build the new list as a "temporary ipset" and then simply swap it out with the one being used in the live firewall: ipset swap ${temp_ipset} ${firewall_ipset} ipset destroy ${temp_ipset} That''s effectively an instantaneous transaction. My theory was that this would minimize actual interference with the running firewall. I only looked at it briefly, but is your script actually carrying out atomic add and delete operations to the ipset in use by the running firewall? Won''t this cause some blocking (I assume ipset uses some kind of locking to prevent simultaneous modification and testing of an ipset) and thereby, over the course of so many transactions, delay the processing of packets? Why wouldn''t it be better to build a temporary ipset and then swap it out? 3. Not a question: as to shorewall integration, I avoided it. I just used my distro''s rc system to require ipsets to be up (as a service) before shorewall. Thanks for sharing your script. I got a couple of good ideas from it. I linked to mine the other day, but they''re at: http://forums.gentoo.org/viewtopic-t-863121.html ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
On 02/24/2013 04:38 AM, Spain, Dr. Jeffry A. wrote:> What experience have users had using ShoreWall as a bogon filter > using the Team Cymru full bogon lists > (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 full > bogon list contains over 4,600 separate networks that need to be > denied, and the IPv6 list over 68,300. Having not tried this myself, > I would be concerned a priori about ShoreWall server meltdown.With IPv4 moving towards 100% allocation, bogon filters are more likely to cause problems than block illegitimate traffic. See http://tools.ietf.org/html/rfc6441 the list of networks to filter http://tools.ietf.org/html/rfc5735 and the discussions at http://lists.ausnog.net/pipermail/ausnog/2012-February/012133.html and http://lists.ausnog.net/pipermail/ausnog/2011-October/011439.html Paul ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
Matt Joyce
2013-Feb-27 22:18 UTC
Re: ipset updating/filtering script (Was: Full Bogon Filtering)
After the discussion earlier I got a few ideas from the suggestions and decided to try some thoughts to improve the script, new script to share with anyone who might find it useful. Changes - Complete overhaul of list processing the new version manages to perform an atomic full rebuild of the entire list in ~0.03s for the IPv4 list and ~0.3s for the IPv6 list. - Uses the features of ipset restore building an ipset save file which can then be processed in a single transaction by ipset. - Building is now done using an intermediate temp list which is swapped in, there are no changes to your active filter list unless the entire rebuild process goes through correctly. Especially useful if updating using cron where a failure may not be immediately visible. I was really surprised just how well this method works that is until I thought it through some more and remembered something about what I was writing this for then it was a case of why on earth did I fail to think of this earlier, I''m pretty sure that shorewall uses a similar procedure for iptables because now I think about it doesn''t shorewall do the same thing with iptables? I am guessing probably for similar reasons, at least for atomicity but I suspect in this case again that perhaps the iptables C binary is probably also faster at doing the job than interpreted shell or perl. Being already pre-compiled into native machine code and able to interface directly with the kernel syscalls without having to deal with a middleman. Oh, and another thought came to me wanted to ask does shorewall have a similar hook for run_ipset as there is for such as run_tc and run_iptables just thinking it would be a better way to do the restore in shorewall-start is that the only way to make successful startup dependant on a start script succeeding, or would it be sufficient if my script exits nonzero on failure? New script is attached for anyone interested to look into the code or make use of it, md5sum bellow: 0a74aab398d14b157ec2371b3a16d1a0 bogon-ipsets-v1.1.tar.gz On 26/02/13 22:22, Matt Joyce wrote:> On 24/02/13 08:09, Matt Joyce wrote: >> On 23/02/13 19:27, Tom Eastep wrote: >>> Using ipsets is the only way that I would try such a thing. >>> >>> -Tom >>> >>> On 2/23/13 10:57 AM, "Cory Oldford" <cory@peaceworks.ca >>> <mailto:cory@peaceworks.ca>> wrote: >>> >>> The overhead associated with matching against the complete bogon >>> list is too much in my humble opinion. >>> >>> >>> Cory Oldford >>> >>> ------------------------------------------------------------------------ >>> *From: *"Dr. Jeffry A. Spain" <spainj@countryday.net >>> <mailto:spainj@countryday.net>> >>> *To: *"Shorewall Users" <shorewall-users@lists.sourceforge.net >>> <mailto:shorewall-users@lists.sourceforge.net>> >>> *Sent: *Saturday, February 23, 2013 12:38:14 PM >>> *Subject: *[Shorewall-users] Full Bogon Filtering >>> >>> What experience have users had using ShoreWall as a bogon filter >>> using the Team Cymru full bogon lists >>> (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 >>> full bogon list contains over 4,600 separate networks that need >>> to be denied, and the IPv6 list over 68,300. Having not tried >>> this myself, I would be concerned a priori about ShoreWall server >>> meltdown. >>> >>> Jeffry A. Spain, Network Administrator >>> Cincinnati Country Day School >>> >>> >>> ------------------------------------------------------------------------------ >>> Everyone hates slow websites. So do we. >>> Make your web apps faster with AppDynamics >>> Download AppDynamics Lite for free today: >>> http://p.sf.net/sfu/appdyn_d2d_feb >>> _______________________________________________ >>> Shorewall-users mailing list >>> Shorewall-users@lists.sourceforge.net >>> <mailto:Shorewall-users@lists.sourceforge.net> >>> https://lists.sourceforge.net/lists/listinfo/shorewall-users >>> >>> ------------------------------------------------------------------------------ >>> Everyone hates slow websites. So do we. Make your web apps faster >>> with AppDynamics Download AppDynamics Lite for free today: >>> http://p.sf.net/sfu/appdyn_d2d_feb_______________________________________________ >>> Shorewall-users mailing list >>> Shorewall-users@lists.sourceforge.net >>> <mailto:Shorewall-users@lists.sourceforge.net> >>> https://lists.sourceforge.net/lists/listinfo/shorewall-users >>> >>> >>> >>> -Tom >>> You do not need a parachute to skydive. You only need a parachute to >>> skydive twice. >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Everyone hates slow websites. So do we. >>> Make your web apps faster with AppDynamics >>> Download AppDynamics Lite for free today: >>> http://p.sf.net/sfu/appdyn_d2d_feb >>> >>> >>> _______________________________________________ >>> Shorewall-users mailing list >>> Shorewall-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/shorewall-users >> I have tried this myself and I have to second what Tom said above, >> unfortunately I havn''t rewritten the shell scripts for it yet after >> losing them in a hard disk crash had other things needed resolving >> first however the actual update of the ipsets will cause a bit of a >> spike especially if your firewall system is a low end machine (An >> intel i7 hexacore was taking a minute or two to chew through it it) >> but I created a shell script to download the lists whenever they were >> modified, compare them and then update the ipsets with changes only. >> I have been intending to implement this again myself and it would >> probably take as long to write a detailed explanation as it would to >> just go ahead and create the scripts again with comments. >> >> I should have some free time this morning so I will try to get that >> done today and get back to you with the shell script and comments. >> >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. >> Make your web apps faster with AppDynamics >> Download AppDynamics Lite for free today: >> http://p.sf.net/sfu/appdyn_d2d_feb >> >> >> _______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/shorewall-users > I have to apologize that things got a bit busy here and I forgot about > this until this evening, I just worked on the script and tested it. It > will provide a short summary after a run which I generally let cron mail > to me so I know if the update stops working for any reason but you could > just as easily redirect it to a file too. > > The script is attached with comments, note that this version works > pretty quickly because it doesn''t go through and add them line by line > every load, instead it compares the sorted new list against ipset list > using comm to find added and deleted entries then only processes those, > as a sanity check it will use md5sum if available to confirm the updated > ipset matches checks the md5sum of the two lists match after updating > provided the md5sum binary is installed. > > Few notes I thought I''d mention though the comments in the script > discuss them too: > Note that ipsets are lost on reboot but by default the script uses ipset > save to save the ipset to it''s working directory, just need to use ipset > restore before using it for filtering shorewall can handle this for you > if you add something like the following to /etc/shorewall/start > ipset -! restore -f /var/cache/bogon-lists/bogonsip4.ipset > Yes you could just readd each time but ipset restore is a lot faster at > it than a bash script. > > It''s your choice but I would suggest it may not hurt to consider my > suggestion in there about using caps support if you have it available, > that way you can avoid having a shell script running as root chewing on > an externally sourced file which to me seemed a bit too much like asking > for trouble. > > Do check the script before using it as there are a few other options you > may wish to change. > > As a side note the script as is will work with any list of CIDR ranges > that one might like to keep up to date and use for filtering in one per > line format including files containing #comments whole or end of line, > doesn''t care about whitespace either. It will need a small modification > to work with ftp:// urls, currently it is written to expect a > http_response code which is fine with these lists but might be an issue > if anyone wanted to adapt it. > > Anyway, I''m happy to make freely available to anyone to use or modify as > they see fit hope it''s useful to someone. Comments welcome if there is > anything I missed somewhere. > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
I can see there being a potential for issues, though there always was when it came to something like bogon filtering especially if using a list from an outdated source or failing to update it with any frequency. I personally think with this list at least one should be fairly safe I ran it for a good 6 months up until a gap starting like 2 weeks ago before and I know at least with every entry I saw in the shorewall log blocked by this filter I could temp disable it and attempt send a ping or any other icmp, tcp or udp packet towards the source address and get back an ICMP network unreachable. It does depend on having a good quality list with the updates though, these are produced every 30-120 minutes (Always by the 2 hour point but if changes go over about a dozen they seem to push it early), and the source is from snapshotting the BGP tables which when it comes down to it if there is no route advertised in BGP for a range unless that address is on the same AS as your default route then there is no way there will be a two way communication with that host filtering or otherwise. It will get as far as your default gateway, maybe even to your ISP''s edge routers but there you will get ICMP net unreach. Other thing that occurs to me is that the system administrators at most AS''es that have a clue know that it can sometimes take a day or two for a newly advertised route on BGP to actually reach all the tiny branches of the network so in much the same way as can be the case with DNS updates generally an AS will publish a route on BGP a good 24-48 hours before they actually try allocating the addresses to end users to do it any sooner is begging for unhappy customers, or plain incompetent capacity planning. Of course I''m not recommending anyone go ahead and set it up, for many it is likely unnecessary anyway, in my case I have it set up because it seems the VPN service I use has erm less than perfect filtering running, wouldn''t surprise me if the spoofed packets are from other users on their network actually, certainly see enough windows netbios packets from other hosts in the VPN''s virtual subnet. As for why I havn''t just gone and complained to them about it and pushed them to get their act in gear basically the main reason is that I don''t actually consider it my providers job to secure my hosts for me. That and if I set up filters or firewalls and screw it up then it''s on me and I go fix it, a third party starts doing it and it''s just a damned hastle. For most people I don''t suggest setting up bogon filtering however I will still publish the script to help those who may find it of value whether simply to study it, using it as is or using it for some other purpose, it was written t be as generic as I could make it without knowing beforehand the exact format of any other lists, thus setting it up to be able to cope with standard # comments including end of lines which the current list doesn''t use. Figure it might have some more mainstream use to others for example I''ve seen malware blacklists around in CIDR form that should work with it, advertising/tracking host lists also both of which also require a continuous regular update because malware gets fixed and new machine compromised all the time, ad hosts tend to move about or at least generally spread on a pretty regular basis. I also consider the IPv6 list in this case to be a different scenario, not only is there currently trillions of bogon IP''s in IPv6 but there also seems to me to be more potential for such suspect packets to reach end user networks through various tunnels, especially such a toredo or unmanaged 6to4, combine that with the many home and business ISP''s whos support for IPv6 ranges from little to none and I can see it being quite likely that there is a potential to end up with an end user who''s tunnel connects directly to a toredo/6to4 relay and has a sum total of 0 filtering of any kind beyond what is set up by the user themselves on their end. I''ve seen numerous suspect packets when I''ve had an unmanaged 6to4 tunnel running myself, don''t get any with the HE tunnel but HE is a large transit service provider which dual stacked their entire system feels like forever ago so they are a bit more on the ball than many. *Wishes his ISP were in the same boat there and decides to at least remain grateful that they are at least direct peers with HE and as a result the HE tunnel server is only 2 sub milisecond hops away from his ISP''s edge gateway, almost as close to a direct native routing as you can get with a tunnel I suspect.* On 27/02/13 21:57, Paul Gear wrote:> On 02/24/2013 04:38 AM, Spain, Dr. Jeffry A. wrote: >> What experience have users had using ShoreWall as a bogon filter >> using the Team Cymru full bogon lists >> (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 full >> bogon list contains over 4,600 separate networks that need to be >> denied, and the IPv6 list over 68,300. Having not tried this myself, >> I would be concerned a priori about ShoreWall server meltdown. > With IPv4 moving towards 100% allocation, bogon filters are more likely > to cause problems than block illegitimate traffic. See > http://tools.ietf.org/html/rfc6441 > the list of networks to filter > http://tools.ietf.org/html/rfc5735 > and the discussions at > http://lists.ausnog.net/pipermail/ausnog/2012-February/012133.html > and > http://lists.ausnog.net/pipermail/ausnog/2011-October/011439.html > > Paul > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
> From: Paul Gear <paul@gear.dyndns.org> > > With IPv4 moving towards 100% allocation, bogon filters are more > likely to cause problems than block illegitimate traffic.I''ve done it (at LAN gateway level), and it hasn''t caused problems, but it''s also probably not worth doing. I think the relevance here for shorewall users is not that they should be engaged in blacklisting full bogon lists, but that such scripts serve as useful examples of how to use ipsets. I wrote my earlier referenced scripts two years ago as a demonstration of how to use ipsets dynamically (for example, in block lists). I also created other examples using other periodically published lists. So while blacklisting full bogon lists may not be very useful, ipsets are. A couple of useful techniques have been shared through this cross-talk. By the way, dnsmasq is being modified to be able to populate ipsets based on name resolution. For example, you could allow or deny a set containing all addresses a given URL is actively resolved to. ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
Spain, Dr. Jeffry A. skrev den 2013-02-23 19:38:> I would be concerned a priori about ShoreWall server meltdown.are you talking of resolve bgp route in another level of managemant ? i use spamhaus drop here in a include / blacklist, but since it see few hits on it, i think my isp is doing it in bgp level :) i just like to be sure for your question aswell ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
John Brendler skrev den 2013-02-28 00:39:> By the way, dnsmasq is being modified to be able to populate ipsets > based on name resolution. For example, you could allow or deny a set > containing all addresses a given URL is actively resolved to.is the same as rpz policy zone in bind ? neat that bind is not the only one so :) ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
That was pretty much all I was hoping for personally, or even if it just gave people ideas kinda why I made the point a few times that it could easily be used for some other list. I should really get around to setting up a site with shell scripts on, I''ve been messing with them a lot for some time, I could be overestimating my own ability here but I feel like I''ve been getting pretty good with them and that some of them could be of use, I like my stats and graphs etc so I have a number of things for producing stats of which quite a lot are networking related as I run several services at various locations. Got another one to work on later actually idea came to me been curious for a while about the proportions of exit v relay traffic but for the longest time was at a loss trying to work out a way one could tell a relay on arbitrary ports and IP addresses from similarly arbitrary exit traffic with any reliability, that is until the obvious answer struck me earlier, the consensus has the IP and port of all currently active relay''s excluding bridges anyway hash:ip,port :) By the way a thought wondering if it''s possible or if anyone has actually tried using ipsets to help with traffic shaping of hard to shape traffic, I''m thinking using an ipset with timeout as a temporary storage to flag IP addresses of known/suspected p2p traffic my thought being this way would be a good option to enable combining of multiple detection strategies, and also get around the fact that connmarking is only so good as long as the connection remains so if the clients decide to recycle their connections or saturate themselves until they end up sendQ''ing themselves you would be able to keep track of them when they resurface again a few minutes later without having to wait for it to be found out again. On 27/02/13 23:39, John Brendler wrote:>> From: Paul Gear <paul@gear.dyndns.org> >> >> With IPv4 moving towards 100% allocation, bogon filters are more >> likely to cause problems than block illegitimate traffic. > I''ve done it (at LAN gateway level), and it hasn''t caused problems, but > it''s also probably not worth doing. > > I think the relevance here for shorewall users is not that they > should be engaged in blacklisting full bogon lists, but that such > scripts serve as useful examples of how to use ipsets. > > I wrote my earlier referenced scripts two years ago as a demonstration > of how to use ipsets dynamically (for example, in block lists). I also > created other examples using other periodically published lists. > > So while blacklisting full bogon lists may not be very useful, ipsets > are. A couple of useful techniques have been shared through this > cross-talk. > > By the way, dnsmasq is being modified to be able to populate ipsets > based on name resolution. For example, you could allow or deny a set > containing all addresses a given URL is actively resolved to. > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
Something like this will probably work well for most say small to medium sized sites, with the big sites there is no guarantee. Their DNS names will in the case of some large sites resolve to a selection of the actual server address and be rotated with each client query, a good clue a site is quite possibly doing something like that is when you query the name of a large well known site with a tool like dig and find you get a reply back with a TTL of something really short like 300 secs max, generally a setting that low is for one of two reasons 1) They are using rapid cycling Name->IP mapping or 2) It''s a dynamic DNS name. When it''s a major organization though is pretty clear which is likely. Some even will only give out a single IP despite having entire blocks of them, not so clever IMO there is a reason to provide several, fallback if a connection attempt fails. Google for example usually hands out a set of 5, Facebook will only give you 1, though will be gone from your cache in 60secs anyway in which case you may get a new one that way. Like the example bellow, .27 first try, minute later expires and it''s .21, I know if it was me and I was trying to block them would be more like just give up and drop 69.171.224.0/19. Anyway was just something I wanted to highlight for anyone intending on making use of such a feature it''s got a good potential to break on larger sites or sites which are actively evasive of censorship. My suggestion for anyone wanting to block DNS names would be to do so through DNS, run a local caching resolver either dnsmasq (Easier for the average user most likely especially with it''s easy configuration for overriding names etc), or bind is perfectly capable of being used in this way, just create a master zone in the bind config file for the domain you want to block either empty so it will return NXDOMAIN or redirect it to an IP of your choice with an actual zonefile, either way it will cause the local bind server to consider itself authoritative for the name and thus return the configured address instead of looking it up. Once you have that set up, make sure that your resolv.conf is set up to use the local DNS server, can use resolvconf to do this especially if you get your DNS servers from your ISP via DHCP, resolvconf can generate the configuration to automatically set those forwarders up in either bind or dnsmasq and set resolv.conf to point to the local DNS server instead. If you need the additional level of protection ie to defend against deliberate circumvention rather than mere accidental access where this would likely have you set already but you can set shorewall to drop all outgoing DNS queries *except* those originating from the daemon user your caching DNS server runs as. zeus bin # dig www.facebook.com IN A; sleep 60; dig www.facebook.com IN A ; <<>> DiG 9.9.2 <<>> www.facebook.com IN A ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50361 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 3 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;www.facebook.com. IN A ;; ANSWER SECTION: www.facebook.com. 1096 IN CNAME star.c10r.facebook.com. star.c10r.facebook.com. 60 IN A 69.171.242.27 ;; AUTHORITY SECTION: c10r.facebook.com. 37993 IN NS b.ns.c10r.facebook.com. c10r.facebook.com. 37993 IN NS a.ns.c10r.facebook.com. ;; ADDITIONAL SECTION: a.ns.c10r.facebook.com. 37993 IN A 69.171.239.11 b.ns.c10r.facebook.com. 37993 IN A 69.171.255.11 ;; Query time: 60 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Feb 28 02:14:03 2013 ;; MSG SIZE rcvd: 152 ; <<>> DiG 9.9.2 <<>> www.facebook.com IN A ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17328 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 3 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;www.facebook.com. IN A ;; ANSWER SECTION: www.facebook.com. 1031 IN CNAME star.c10r.facebook.com. star.c10r.facebook.com. 60 IN A 69.171.242.21 ;; AUTHORITY SECTION: c10r.facebook.com. 37929 IN NS b.ns.c10r.facebook.com. c10r.facebook.com. 37929 IN NS a.ns.c10r.facebook.com. ;; ADDITIONAL SECTION: a.ns.c10r.facebook.com. 37929 IN A 69.171.239.11 b.ns.c10r.facebook.com. 37929 IN A 69.171.255.11 ;; Query time: 59 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Feb 28 02:15:07 2013 ;; MSG SIZE rcvd: 152 On 28/02/13 00:49, Benny Pedersen wrote:> John Brendler skrev den 2013-02-28 00:39: > >> By the way, dnsmasq is being modified to be able to populate ipsets >> based on name resolution. For example, you could allow or deny a set >> containing all addresses a given URL is actively resolved to. > is the same as rpz policy zone in bind ? > > neat that bind is not the only one so :) > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb