Is there a way I could use the above two jump targets in my rules file? The reason is that for some hosts (well, ipset, actually) I would like the blacklist/blackout checks to be bypassed. Currently I have the following rather ugly hack in "start": run_iptables -R fw2net 1 -m set ! --match-set whitelist dst -j blackout run_iptables -R net2fw 1 -m set ! --match-set whitelist src -j blacklst If there is a way I could do this in the rules file without reverting to the above that would be perfect. The idea to my "whitelist" is pretty simple - creating pinholes in the blacklist/blackout chains for hosts/subnets I trust regardless of whether these are blacklisted or not. ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
On May 14, 2011, at 6:42 PM, Mr Dash Four wrote:> Is there a way I could use the above two jump targets in my rules file? > > The reason is that for some hosts (well, ipset, actually) I would like > the blacklist/blackout checks to be bypassed. Currently I have the > following rather ugly hack in "start": > > run_iptables -R fw2net 1 -m set ! --match-set whitelist dst -j blackout > run_iptables -R net2fw 1 -m set ! --match-set whitelist src -j blacklst > > If there is a way I could do this in the rules file without reverting to > the above that would be perfect. The idea to my "whitelist" is pretty > simple - creating pinholes in the blacklist/blackout chains for > hosts/subnets I trust regardless of whether these are blacklisted or not. >There is no other way to do that. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
> There is no other way to do that. >Maybe if I could do/specify something (not necessarily in the rules file, but in blacklist) - something similar to the test column in tcrules perhaps? ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
> Maybe if I could do/specify something (not necessarily in the rules > file, but in blacklist) - something similar to the test column in > tcrules perhaps?Maybe even a separate "whitelist" file doing exactly the opposite of what "blacklist" currently does - that''s another option. ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
On May 14, 2011, at 7:18 PM, Mr Dash Four wrote:> >> Maybe if I could do/specify something (not necessarily in the rules >> file, but in blacklist) - something similar to the test column in >> tcrules perhaps? > Maybe even a separate "whitelist" file doing exactly the opposite of > what "blacklist" currently does - that''s another option.Seems like that is what an exclusion list in the blacklist file does. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
On Sun, 15 May 2011 02:06:12 +0000 shorewall-users-request@lists.sourceforge.net wrote:> From: Tom Eastep <teastep@shorewall.net> > To: Shorewall Users <shorewall-users@lists.sourceforge.net> > Subject: Re: [Shorewall-users] creating pinholes in blacklist and > blackout Date: Sat, 14 May 2011 18:57:47 -0700 > Reply-To: Shorewall Users <shorewall-users@lists.sourceforge.net> > > > On May 14, 2011, at 6:42 PM, Mr Dash Four wrote: > > > Is there a way I could use the above two jump targets in my rules > > file? > > > > The reason is that for some hosts (well, ipset, actually) I would > > like the blacklist/blackout checks to be bypassed. Currently I have > > the following rather ugly hack in "start": > > > > run_iptables -R fw2net 1 -m set ! --match-set whitelist dst -j > > blackout run_iptables -R net2fw 1 -m set ! --match-set whitelist > > src -j blacklst > > > > If there is a way I could do this in the rules file without > > reverting to the above that would be perfect. The idea to my > > "whitelist" is pretty simple - creating pinholes in the > > blacklist/blackout chains for hosts/subnets I trust regardless of > > whether these are blacklisted or not. > > There is no other way to do that. > > -TomNot sure exactly what you mean by "pinholes" (forgive me if I''m off-base) but one thing you may want to consider is using the "iptreemap" type of ipset. The iptree and iptreemap ipset types are unique in that you can populate it and then selectively remove smaller pieces from it. In other words, you can add a network to it, and then delete a subnet or address of that network, and the ipset will dynamically (and instantaneously) break the previous entry into two parts around the subnet you have removed. In the other (non-tree) ipset types, you can only delete the exact entries you have created. If you put in a network, you can take out that network, but you can''t take out just part of it. So, creating a "pinhole" (unless I misunderstood what you mean by it) would require you to delete the network, calculate the subnets that should remain once your "pinhole" is removed, and the add back those subnets. The "tree" type of ipsets takes care of it for you. Example script: (scroll down to the one about iptreemap) http://forums.gentoo.org/viewtopic-t-863121.html ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
On Sat, 14 May 2011 22:46:11 -0400 John Brendler <brendlerjg@gmail.com> wrote:> The "tree" type of ipsets takes care of it for you.In case my explanation was confusing, it''s this simple: ~# ipset --create my_ipset iptreemap ~# ipset --add my_ipset 192.168.0.0/24 ~# ipset --list my_ipset Name: my_ipset Type: iptreemap References: 0 Header: gc: 300 Members: 192.168.0.0-192.168.0.255 ~# ipset --del my_ipset 192.168.0.10 ~# ipset --list my_ipset Name: my_ipset Type: iptreemap References: 0 Header: gc: 300 Members: 192.168.0.0-192.168.0.9 192.168.0.11-192.168.0.255 Of course, it can have as many adds and deletes as you like, subject to the typical constraints. Again, if you meant something else by "pinhole", please forgive the unnecessary traffic. ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
> Seems like that is what an exclusion list in the blacklist file does. >Is this currently implemented as I am not aware of such functionality in blacklist? ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
> Not sure exactly what you mean by "pinholes" (forgive me if I''m > off-base) but one thing you may want to consider is using the > "iptreemap" type of ipset. The iptree and iptreemap ipset types are > unique in that you can populate it and then selectively remove smaller > pieces from it. > > In other words, you can add a network to it, and then delete a subnet > or address of that network, and the ipset will dynamically (and > instantaneously) break the previous entry into two parts around the > subnet you have removed. >That is no longer supported in newer ipset versions (5 or 6) - the main reason I have implemented this in the way it is now. You are right in a way, with ipset 4.x that was indeed possible, though you have to know in what (blacklist) set exactly is the address/subnet you wish to delete, which is not always that easy.> So, creating a "pinhole" (unless I misunderstood what you mean by it) > would require you to delete the network, calculate the subnets that > should remain once your "pinhole" is removed, and the add back those > subnets. The "tree" type of ipsets takes care of it for you. >That is not a solution - I can''t just "delete" this network if I use any of the modern versions of ipset. ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
> ~# ipset --create my_ipset iptreemap >iptreemap has been discontinued in ipset versions 5 and 6 - it exists in the form you specify in ipset version 4 only. ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
On May 15, 2011, at 5:04 AM, Mr Dash Four wrote:> >> Seems like that is what an exclusion list in the blacklist file does. >> > Is this currently implemented as I am not aware of such functionality in > blacklist? >Yes -- it''s supported. I''ve updated the on-line manages to mention that capability. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
On Sun, 15 May 2011 12:51:51 +0000 shorewall-users-request@lists.sourceforge.net wrote:> That is no longer supported in newer ipset versions (5 or 6) - the > main reason I have implemented this in the way it is now. You are > right in a way, with ipset 4.x that was indeed possible, though you > have to know in what (blacklist) set exactly is the address/subnet > you wish to delete, which is not always that easy.Well, you have a choice. You can use ipset version 4 and do it this way (which results in having a single netfilter matching operation, against the ipset), or you can use an exclusion clause, which will result in an additional netfilter operation first performing a match operation against the excluded node. I am of the mindset that blacklist rules should be carefully optimized because they are expensive (universally applied). If you are constrained on which version of ipset you can use, then I suppose you ought to do the latter (as explained in SHOREWALL-BLACKLIST(5).> > So, creating a "pinhole" (unless I misunderstood what you mean by > > it) would require you to delete the network, calculate the subnets > > that should remain once your "pinhole" is removed, and the add back > > those subnets. The "tree" type of ipsets takes care of it for you. > > > That is not a solution - I can''t just "delete" this network if I use > any of the modern versions of ipset.I think I was unclear here. I am saying that if you do not use a tree type ipset, but you still want to create the "pinhole" within the ipset, then the equivalent would be to "manually" exclude it at the time you create the ipset by pre-processing the list and breaking in two any element that contains a trusted addresses. That would probably only be worth the effort of writing the code if you have a high-traffic situation where eliminating a single netfilter match would justify it. It would also increase the time of uploading a new blacklist (e.g. if you''re using published blocklists updated hourly) by a minute or so while this entry-by-entry processing occurs. From what it sounds like, you should probably just use exclusions in the blacklist file. Sorry if I wasted your time. ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
>>> Seems like that is what an exclusion list in the blacklist file does. >>> >>> >> Is this currently implemented as I am not aware of such functionality in >> blacklist? >> >> > > Yes -- it''s supported. I''ve updated the on-line manages to mention that capability. >That isn''t really what I am after. here is what happens: blacklist ~~~~~~~~~ !+whitelist shorewall show blackout ~~~~~~~~~~~~~~~~~~~~~~~ 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ! match-set whitelist dst 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 match-set ... dst That would DROP packets which do not belong to the whitelist set! In addition, the checks continue to propagate down the chain if there isn''t a match (i.e. the dest IP address matches the whitelist). Not what I am after at all. I need to bypass all checks (blacklst/blackout) if there is a whitelist match. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On Wed, 18 May 2011 00:35:16 +0000 shorewall-users-request@lists.sourceforge.net wrote:> That isn''t really what I am after. here is what happens: > > blacklist > ~~~~~~~~~ > !+whitelist > > shorewall show blackout > ~~~~~~~~~~~~~~~~~~~~~~~ > 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ! match-set whitelist dst > 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 match-set ... dst > > That would DROP packets which do not belong to the whitelist set! In > addition, the checks continue to propagate down the chain if there > isn''t a match (i.e. the dest IP address matches the whitelist). Not > what I am after at all. I need to bypass all checks > (blacklst/blackout) if there is a whitelist match.You''re doing it wrong. Think of it as two clauses: <what_to_block> !<what_to_exclude_from_that> If you''re using only ipsets, it would look like this: +blocklist!+whitelist If you have multiple ipsets in your blacklist file, and any of them could contain trusted addresses, then you''d probably want to do oneof the following: a) include the whitelist exclusion with each: +blocklist!+whitelist +redlist!+whitelist +shitlist!+whitelist or; b) combine the ipsets into a setlist, reducing your matching against the ipsets from six to to two netfilter operations: ipset create setlist blacklist size 3 ipset add blacklist blocklist ipset add blacklist redlist ipset add blacklist shitlist then you can use that setlist as a single entry in your blacklist file, with an exclusion clause: +blacklist!whitelist ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
> If you''re using only ipsets, it would look like this: > > +blocklist!+whitelist > > If you have multiple ipsets in your blacklist file, and any of them > could contain trusted addresses, then you''d probably want to do oneof > the following: > > a) include the whitelist exclusion with each: > > +blocklist!+whitelist > +redlist!+whitelist > +shitlist!+whitelist > > or; > b) combine the ipsets into a setlist, reducing your matching against > the ipsets from six to to two netfilter operations: > > ipset create setlist blacklist size 3 > ipset add blacklist blocklist > ipset add blacklist redlist > ipset add blacklist shitlist > > then you can use that setlist as a single entry in your blacklist file, > with an exclusion clause: > > +blacklist!whitelist >I have indeed multiple blacklist sets and they are split in separate groups for a reason - so that I could track/account how many packets/bytes have been dropped from each group. Example a) above is quite cumbersome and resource-consuming. Example b) won''t give me the information I have gathered previously (otherwise I would have lumped all blacklists together in one massive set and got on with it), so it looks as though there are two possible solutions: 1) new implementation of a whitelist in shorewall; or 2) I stick to my own solution and use the hack I listed in the first post of this thread. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 05/17/2011 08:03 PM, Mr Dash Four wrote:> so it looks as though there are two possible solutions: 1) new > implementation of a whitelist in shorewall; or 2) I stick to my own > solution and use the hack I listed in the first post of this thread.Please give the attached patch a try. It treats entries that begin with w: (or W:) as whitelist entries. If you use the ipset ''whitelist'' as your whitelist, then you can place the following in your blacklist file: W:+whitelist -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
> Please give the attached patch a try. It treats entries that begin with > w: (or W:) as whitelist entries. > > If you use the ipset ''whitelist'' as your whitelist, then you can place > the following in your blacklist file: > > W:+whitelist >Thanks, I''ll give it a proper go when I get back home later this evening. There is also something new I have been testing in the past few days, which I''ll post the results on then. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
Tom Eastep
2011-May-18 14:40 UTC
Re: creating pinholes in blacklist and blackout (better patch)
On 05/18/2011 07:32 AM, Mr Dash Four wrote:> >> Please give the attached patch a try. It treats entries that begin with >> w: (or W:) as whitelist entries. >> >> If you use the ipset ''whitelist'' as your whitelist, then you can place >> the following in your blacklist file: >> >> W:+whitelist >> > Thanks, I''ll give it a proper go when I get back home later this > evening. There is also something new I have been testing in the past few > days, which I''ll post the results on then.Thanks. Attached is a better patch - it implements a ''whitelist'' OPTION. Note that when ''whitelist'' is given, the BLACKLIST_LOGLEVEL setting is ignored and the packet/connection is silently passed on to the rules/policies. Are there any opinions, one way or the other, on that behavior? -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
Mr Dash Four
2011-May-18 15:14 UTC
Re: creating pinholes in blacklist and blackout (better patch)
> Note that when ''whitelist'' is given, the BLACKLIST_LOGLEVEL setting is > ignored and the packet/connection is silently passed on to the > rules/policies. Are there any opinions, one way or the other, on that > behavior? >I am uncertain on what that means? The behaviour I would expect from a ''whitelist'' entries (either via the "w:" or with the "whitelist" option specified) is that control is returned to the first iptables statement after the blacklst/blackout chains, provided there is a "whitelist" match (whether by the inclusion - or exclusion - of ipset, or by any other means). Your patch is actually better than my hack as it provides positioning - in other words, I could have certain (blacklisted entries) checks performed and then get a whitelist entry match: +whitelist1 +blacklist1 +blacklist2 +whitelist2 +blacklist3 +whitelist3 The above, if I understood your patch correctly, will return, unconditionally, if there is a whitelist1 match (this is what my hack currently does). It will then check for entries in blacklist1 and 2 and if there is still not a match it will check whitelist2 and return if there is a match (it won''t check blacklist3!). It will finally check blacklist3, whereas whitelist3 is completely meaningless as it is the last statement, but could be used for other purposes. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
Tom Eastep
2011-May-18 15:21 UTC
Re: creating pinholes in blacklist and blackout (better patch)
On 05/18/2011 08:14 AM, Mr Dash Four wrote:> >> Note that when ''whitelist'' is given, the BLACKLIST_LOGLEVEL setting is >> ignored and the packet/connection is silently passed on to the >> rules/policies. Are there any opinions, one way or the other, on that >> behavior? >> > I am uncertain on what that means? > > The behaviour I would expect from a ''whitelist'' entries (either via the > "w:" or with the "whitelist" option specified) is that control is > returned to the first iptables statement after the blacklst/blackout > chains, provided there is a "whitelist" match (whether by the inclusion > - or exclusion - of ipset, or by any other means).That''s what my patch does.> Your patch is actually better than my hack as it provides positioning > - in other words, I could have certain (blacklisted entries) checks > performed and then get a whitelist entry match: > > +whitelist1 > +blacklist1 > +blacklist2 > +whitelist2 > +blacklist3 > +whitelist3 > > The above, if I understood your patch correctly, will return, > unconditionally, if there is a whitelist1 match (this is what my hack > currently does). It will then check for entries in blacklist1 and 2 and > if there is still not a match it will check whitelist2 and return if > there is a match (it won''t check blacklist3!). It will finally check > blacklist3, whereas whitelist3 is completely meaningless as it is the > last statement, but could be used for other purposes.Yes. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
Mr Dash Four
2011-May-18 15:34 UTC
Re: creating pinholes in blacklist and blackout (better patch)
> Yes. >No objection from me then - the "whitelist" entries only provide means of directing the flow, so logging on the rest (blacklist entries) should only depend on that option! ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
Tom Eastep
2011-May-18 16:36 UTC
Re: creating pinholes in blacklist and blackout (better patch)
On 05/18/2011 08:34 AM, Mr Dash Four wrote:> >> Yes. >> > No objection from me then - the "whitelist" entries only provide means > of directing the flow, so logging on the rest (blacklist entries) should > only depend on that option!That''s also my feeling. Plus, if BLACKLISTNEWONLY=No, then every packet in a whitelisted connenection would get logged; clearly a foolish thing to do. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
Mr Dash Four
2011-May-18 21:54 UTC
Re: creating pinholes in blacklist and blackout (better patch)
> That''s what my patch does. >Yeah, it works well and does exactly what I wanted, though when I have "+whitelist - - whitelist" (i.e. without specifying "src" or "dst") this is not included anywhere, so I think shorewall should give a warning and not just ignore it. ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay