Beta 4 is now available for testing.
1) This release includes support for ''Condition Match'' which
is
included in xtables-addons. Condition match allows rules to be
predicated on the setting of a named switch in
/proc/net/nf_condition/.
See
http://www.shorewall.net/configuration_file_basics.htm#Switches
for details.
2) With the preceding change, the rules file now has 14 columns. That
makes it awkward to specify the last column as you have to insert
the correct number of ''-'' to get the right column.
To make that easier, it is now allowed to terminate the
column-oriented format with a semicolon (";"), and then specify
addition columns using a column-name=value format. See
http://www.shorewall.net/configuration_file_basics.htm#Pairs for
details.
Thank you for testing,
-Tom
--
Tom Eastep \ When I die, I want to go like my Grandfather who
Shoreline, \ died peacefully in his sleep. Not screaming like
Washington, USA \ all of the passengers in his car
http://shorewall.net \________________________________________________
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
On 01/10/2011 15:21, Tom Eastep wrote:> 2) With the preceding change, the rules file now has 14 columns. That > makes it awkward to specify the last column as you have to insert > the correct number of ''-'' to get the right column. > > To make that easier, it is now allowed to terminate the > column-oriented format with a semicolon (";"), and then specify > addition columns using a column-name=value format. See > http://www.shorewall.net/configuration_file_basics.htm#Pairs for > details. >Before this is released could I ask you to look over the JSON syntax? eg http://en.wikipedia.org/wiki/JSON If I squint a bit I can kind of see that you are doing something quite similar here and perhaps it''s possible to re-use existing formats before we re-invent a new format? I haven''t entirely thought this through, BUT what if the config file looked a bit like a hybrid of the existing column orientated format, but you can drop in a json snippet at the end of the line to set/override any params? This seems equivalent to what you are already doing, but with just a tiny change in syntax (semicolon becomes a { } pair and becomes : ) A potential side benefit would seem to be that you accidentally just added input support for JSON ... Note, based on my previous email I might come across as having a particular preference towards json - it''s not the case! This suggestion is purely based on the similarity with what you are doing and an existing config file format - reduction/re-use seems attractive! Note, I''m assuming that this change causes little development effort - I''m naively making the suggestion because at this stage it seems like just a change in escape characters? Seems like getting the extra format right at this stage prevents some pain later, hence hoping you will look at this from all angles! Cheers Ed W ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2
On Oct 1, 2011, at 9:18 AM, Ed W wrote:> On 01/10/2011 15:21, Tom Eastep wrote: >> 2) With the preceding change, the rules file now has 14 columns. That >> makes it awkward to specify the last column as you have to insert >> the correct number of ''-'' to get the right column. >> >> To make that easier, it is now allowed to terminate the >> column-oriented format with a semicolon (";"), and then specify >> addition columns using a column-name=value format. See >> http://www.shorewall.net/configuration_file_basics.htm#Pairs for >> details. >> > > Before this is released could I ask you to look over the JSON syntax? eg > http://en.wikipedia.org/wiki/JSON > >Ed, For the last time, I am NOT going to adopt a markup language for the Shorewall configuration. Get used to the idea. What I have done for RC 1 is eliminate the need for the columnar format. Here is an example of a blacklist file: ;proto=udp port=1024:1033,1434,5948,23773 ;proto=tcp port=57,1433,1434,2401,2745,3127,3306,3410,4899,5554,5948,6101,8081,9898,23773 ;networks=221.192.199.48 ;networks=61.158.162.9 ;networks=81.21.54.100 proto=tcp port=25 ;networks=84.108.168.139 ;networks=200.55.184.18 ;networks=1.2.3.4 options=dst -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2
Hi, On Sat, 01 Oct 2011 17:18:56 +0100 Ed W <lists@wildgooses.com> wrote:> Note, based on my previous email I might come across as having a > particular preference towards json - it''s not the case! This > suggestion is purely based on the similarity with what you are doing > and an existing config file format - reduction/re-use seems > attractive!This actually might be interesting, but let''s not put it on Tom''s plate. Maybe you can show some kind of comparison? In fact, you could just go ahead and fork shorewall... and when it works as you envision you can invite list members to review it. At least one other list member (Christ Schlacta - is that your real name?) seems to be interested. - Mark ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2
Why yes, Yes it is. But I am happy with the current Shorewall modifications. I would be willing to test it on one of my systems though :) On 10/1/2011 16:15, Mark van Dijk wrote:> Hi, > > On Sat, 01 Oct 2011 17:18:56 +0100 > Ed W<lists@wildgooses.com> wrote: > >> Note, based on my previous email I might come across as having a >> particular preference towards json - it''s not the case! This >> suggestion is purely based on the similarity with what you are doing >> and an existing config file format - reduction/re-use seems >> attractive! > This actually might be interesting, but let''s not put it on Tom''s > plate. Maybe you can show some kind of comparison? > > In fact, you could just go ahead and fork shorewall... and when it > works as you envision you can invite list members to review it. At > least one other list member (Christ Schlacta - is that your real name?) > seems to be interested. > > - Mark > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2dcopy2 > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2
On 01/10/2011 18:20, Tom Eastep wrote:> On Oct 1, 2011, at 9:18 AM, Ed W wrote: > >> On 01/10/2011 15:21, Tom Eastep wrote: >>> 2) With the preceding change, the rules file now has 14 columns. That >>> makes it awkward to specify the last column as you have to insert >>> the correct number of ''-'' to get the right column. >>> >>> To make that easier, it is now allowed to terminate the >>> column-oriented format with a semicolon (";"), and then specify >>> addition columns using a column-name=value format. See >>> http://www.shorewall.net/configuration_file_basics.htm#Pairs for >>> details. >>> >> Before this is released could I ask you to look over the JSON syntax? eg >> http://en.wikipedia.org/wiki/JSON >> >> > Ed, > > For the last time, I am NOT going to adopt a markup language for the Shorewall configuration. Get used to the idea.But you just *did* adopt a markup language - that''s my point. Every config file format is just an arbitrary markup format Look, I seem to have come across as being hostile - please read these suggestions as genuinely just trying to help get things nailed down in a future proof way? Personally I think I prefer the existing column orientated format, so please don''t misunderstand!> What I have done for RC 1 is eliminate the need for the columnar format. Here is an example of a blacklist file: > > ;proto=udp port=1024:1033,1434,5948,23773 > ;networks=221.192.199.48Sure - I''m just highlighting that the above is already an abitrary "markup" and you might want to consider if it''s optimal before committing to it... If it is, then no complaints here... Consider two other interesting alternatives (not claiming either is *better*, just alternatives) Perl style: proto=>udp, port=>1024:1033,1434,5948,23773 networks=>221.192.199.48 or "web" style {proto:"udp" port:"1024:1033,1434,5948,23773"}, {networks:"221.192.199.48"}, Both have pros and conns. Just highlighting some existing ideas really? Perl style feels natural to me With regards to your current key=value markup, a couple of things occur to me that might be nice to decide on while it''s new: 1) You are using whitespace as the break between value and the next key. Some people will assume that a comma is necessary (comma separated values being probably at least as common, possibly more common). Do we care? 2) I don''t know if there are currently any values which might contain spaces, however, it seems something that may happen in the future. I couldn''t quickly see whether the current config file allows something like key="value with spaces", but is that something you might want to allow? Look, don''t misundertand. All I''m saying is that personally I see little difference between ;key=value key2=value or ;key="value", key2=value or ;key=>value, key2=value or {key:value, key2:value} ...By all means pick your favourite. All I''m asking is if you looked at all the options? They all seem fairly similar to my eye... Note, I think the current column format is quite nice (I haven''t tried, but I bet it''s quite easy to edit using OpenOffice/Excel?). Is it possible to produce a VIM syntax that makes the config files easier to edit? (Not a vim expert, but it would seem that such a thing could largely eliminate editing issues?) All the best Ed W ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2
On Sun, 2011-10-02 at 13:03 +0100, Ed W wrote:> > What I have done for RC 1 is eliminate the need for the columnar format. Here is an example of a blacklist file: > > > > ;proto=udp port=1024:1033,1434,5948,23773 > > ;networks=221.192.199.48 > > Sure - I''m just highlighting that the above is already an abitrary > "markup" and you might want to consider if it''s optimal before > committing to it... If it is, then no complaints here...I think that ''optimal'' is likely to be hard to define but will rather be in the eye of the beholder.> > Consider two other interesting alternatives (not claiming either is > *better*, just alternatives) > > Perl style: > proto=>udp, port=>1024:1033,1434,5948,23773 > networks=>221.192.199.48It''s trivial to support that notion in addition to what I have currently implemented. I notice that you used a comma after the first pair and below you mention the possibility of adding a comma separator. That''s okay so long as we require that the comma be followed by whitespace. Otherwise, the syntax is ambiguous in as much as comma is used frequently as a separator in column values.> > or > > "web" style > {proto:"udp" port:"1024:1033,1434,5948,23773"}, > {networks:"221.192.199.48"}, >I assume that the curly braces denote column/value pairs and that the semicolon is unnecessary in this syntax.> Both have pros and conns. Just highlighting some existing ideas really? > Perl style feels natural to me> > With regards to your current key=value markup, a couple of things occur > to me that might be nice to decide on while it''s new: > > 1) You are using whitespace as the break between value and the next key. > Some people will assume that a comma is necessary (comma separated > values being probably at least as common, possibly more common). Do we care?Noted above.> > 2) I don''t know if there are currently any values which might contain > spaces, however, it seems something that may happen in the future. I > couldn''t quickly see whether the current config file allows something > like key="value with spaces", but is that something you might want to allow?There are no instances of that and never will be. There really isn''t a lexical analyzer in the compiler; it rather simply uses "split('' '', $line)" to isolate the individual columns. That precludes embedded whitespace in column values.> > > Look, don''t misundertand. All I''m saying is that personally I see > little difference between > ;key=value key2=value > or > ;key="value", key2=value > or > ;key=>value, key2=value > or > {key:value, key2:value} > > ...By all means pick your favourite. All I''m asking is if you looked at > all the options? They all seem fairly similar to my eye... >They are. And now, all are supported and in combination. The following is equivalent to the file that I posted earlier. { proto:udp, port=1024:1033,1434,5948,23773 } { proto=tcp port=>"57,1433,1434,2401,2745,3127,3306,3410,4899,5554,5948,6101,8081,9898,23773" } ;networks=>221.192.199.48 ; networks=61.158.162.9 ; networks=81.21.54.100\ proto=tcp\ port=25 ; networks=84.108.168.139 ; networks=200.55.184.18 ; networks=1.2.3.4, options:dst> > Note, I think the current column format is quite nice (I haven''t tried, > but I bet it''s quite easy to edit using OpenOffice/Excel?).Yes -- I''ve tried that. Excel can export a space-separated format but OpenOffice cannot (unless I''m missing something).> Is it possible to produce a VIM syntax that makes the config files easier to > edit? (Not a vim expert, but it would seem that such a thing could > largely eliminate editing issues?)Don''t know -- I prefer emacs. I hope that this topic can now be put to bed once and for all. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2
Hi>> Consider two other interesting alternatives (not claiming either is >> *better*, just alternatives) >> >> Perl style: >> proto=>udp, port=>1024:1033,1434,5948,23773 >> networks=>221.192.199.48 > It''s trivial to support that notion in addition to what I have currently > implemented.Actually, can I suggest that you *don''t* support too many formats here? I only intended to show that there are various options, but I really think after that you should limit shorewall to "fewer" formats? (reduces scope for bugs and misunderstandings)> I notice that you used a comma after the first pair and > below you mention the possibility of adding a comma separator. That''s > okay so long as we require that the comma be followed by whitespace. > Otherwise, the syntax is ambiguous in as much as comma is used > frequently as a separator in column values.Agreed and understood. I can see a disadvantage in the comma from the point of view of this syntax, but equally I can see people putting it in accidentally when learning... Personally don''t have an opinion either way?>> "web" style >> {proto:"udp" port:"1024:1033,1434,5948,23773"}, >> {networks:"221.192.199.48"}, >> > I assume that the curly braces denote column/value pairs and that the > semicolon is unnecessary in this syntax.Aha, see now you come to a format which I think is quite interesting in that give or take some syntax tweaking it''s JSON. Hence my original email. Roughly the syntax maps to a perl hash or array. An element is surrounded by brackets, either { } for a perl hash, or [ ] for an array. The semicolon could easily be done away with here because it''s quite clear when we drop into "web" format. If you don''t see a problem with this style then I would recommend going all the way and eliminate any corner cases so that you can accept literal JSON format for the entire file? (The main thing I can think of is that the line needs to end with a comma, ie "{ ... },") ie it a file can be round tripped using: use JSON; #slurp local $/; open( my $fh, ''<'', ''/etc/shorewall/rules'' ); $rules_json = <$fh>; # decode $rules = decode_json($rules_json); # encode again (just for kicks...) $rules_json = encode_json($rules); Obviously shorewall can''t use the json parser this way because it has to support columnar format also, but I think the advantages for guis/editors is obvious if the file format can be completely round tripped this way? It''s also cool in that at least some tools can then use off the shelf parsing tools, without needing to build a new config file reader? If I risked sticking my neck out again, personally for automated third party shorewall utilities (ie my hypothetical web based editor) I think the easiest formats to parse are the pure column format (ie existing) and the pure other config files above (key-value, perl, web, etc). Hybrids are the easiest to get wrong or make mistakes parsing and so less desired. As such whilst I don''t have strong feelings over which format you finally stabilise on, I think desirable properties would be: - entire file can be column or.. - entire file can be alternative style (as with your key-value) - ideally whole file can be parsed by off the shelf parsers from cpan or trivial code (true of column formats, key-value formats and many others) Note, I''m definitely not claiming your key value format doesn''t fit those criteria, it does! Just highlighting that it''s so very close to a couple of other formats it might be worth considering if one of those is a good/better alternative?>> 2) I don''t know if there are currently any values which might contain >> spaces, however, it seems something that may happen in the future. I >> couldn''t quickly see whether the current config file allows something >> like key="value with spaces", but is that something you might want to allow? > There are no instances of that and never will be. There really isn''t a > lexical analyzer in the compiler; it rather simply uses "split('' '', > $line)" to isolate the individual columns. That precludes embedded > whitespace in column values.Oh, one last thing: This really sounds like the kind of feature you might require at some future point? OK, I''m struggling to figure out the what right now, but as you say it''s because it wasn''t possible before. Just some random ideas, but - what about allowing the logging text to be specified as as one of the key-value pairs? - Could some of the new "conditions" matches stuff need spaces? - I guess it will be any feature of iptables/tc which is expressed in ".." ? Are there many others right now? Just trying to think ahead! Cheers Ed W ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1
Ed, On Mon, 2011-10-03 at 14:04 +0100, Ed W wrote:> > Actually, can I suggest that you *don''t* support too many formats > here? I only intended to show that there are various options, but I > really think after that you should limit shorewall to "fewer" formats? > (reduces scope for bugs and misunderstandings)The code that I''ve implemented should be pretty foolproof.> > > I notice that you used a comma after the first pair and > > below you mention the possibility of adding a comma separator. That''s > > okay so long as we require that the comma be followed by whitespace. > > Otherwise, the syntax is ambiguous in as much as comma is used > > frequently as a separator in column values.> > Aha, see now you come to a format which I think is quite interesting > in that give or take some syntax tweaking it''s JSON. Hence my > original email. > > Roughly the syntax maps to a perl hash or array. An element is > surrounded by brackets, either { } for a perl hash, or [ ] for an > array. The semicolon could easily be done away with here because it''s > quite clear when we drop into "web" format. > > If you don''t see a problem with this style then I would recommend > going all the way and eliminate any corner cases so that you can > accept literal JSON format for the entire file? (The main thing I can > think of is that the line needs to end with a comma, ie "{ ... },") ie > it a file can be round tripped using: > > use JSON; > #slurp > local $/; > open( my $fh, ''<'', ''/etc/shorewall/rules'' ); > $rules_json = <$fh>; > > # decode > $rules = decode_json($rules_json); > > # encode again (just for kicks...) > $rules_json = encode_json($rules); > > > Obviously shorewall can''t use the json parser this way because it has > to support columnar format also, but I think the advantages for > guis/editors is obvious if the file format can be completely round > tripped this way? > > It''s also cool in that at least some tools can then use off the shelf > parsing tools, without needing to build a new config file reader?If you want JSON so badly, then I suggest that *you* implement and document a construct such as this: BEGIN JSON <JSON goes here> END JSON You already have examples in BEGIN PERL...END PERL and BEGIN SHELL...END SHELL. That way, your configuration files can be pure JSON and noone else will have to deal with JSON unless they want to.> If I risked sticking my neck out again, personally for automated third > party shorewall utilities (ie my hypothetical web based editor) I > think the easiest formats to parse are the pure column format (ie > existing) and the pure other config files above (key-value, perl, web, > etc). Hybrids are the easiest to get wrong or make mistakes parsing > and so less desired. > > As such whilst I don''t have strong feelings over which format you > finally stabilise on, I think desirable properties would be: > - entire file can be column or.. > - entire file can be alternative style (as with your key-value) > - ideally whole file can be parsed by off the shelf parsers from cpan > or trivial code (true of column formats, key-value formats and many > others) > > Note, I''m definitely not claiming your key value format doesn''t fit > those criteria, it does! Just highlighting that it''s so very close to > a couple of other formats it might be worth considering if one of > those is a good/better alternative?The formats that I have defined all fit on one line so there is no requirement to match ''{'' and ''}'' over multiple lines. Any JSON produced by a standard tool is unlikely to meet that requirement.> > > > 2) I don''t know if there are currently any values which might contain > > > spaces, however, it seems something that may happen in the future. I > > > couldn''t quickly see whether the current config file allows something > > > like key="value with spaces", but is that something you might want to allow? > > There are no instances of that and never will be. There really isn''t a > > lexical analyzer in the compiler; it rather simply uses "split('' '', > > $line)" to isolate the individual columns. That precludes embedded > > whitespace in column values. > > Oh, one last thing: This really sounds like the kind of feature you > might require at some future point? OK, I''m struggling to figure out > the what right now, but as you say it''s because it wasn''t possible > before. Just some random ideas, but > - what about allowing the logging text to be specified as as one of > the key-value pairs? > - Could some of the new "conditions" matches stuff need spaces? > - I guess it will be any feature of iptables/tc which is expressed in > ".." ? Are there many others right now? > > Just trying to think ahead!I''m not loosing any sleep over that one; we''ve escaped for 10 years without having to deal with that problem. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1