Hello,
While working on the packaging of shorewall 4.0.4 for Fedora[1], a few
issues have arisen related to the shorewall-perl compiler, and the
file Ports.pm, the file which is essentially a hash of
/etc/{services,protocols}.
Firstly, since this is an application file representing local machine
state, I think it should be kept under /var/lib/shorewall-perl rather
than /usr/share/shorewall-perl/Shorewall. This is particularly
pertinent for machines which have /usr mounted read only.
Secondly, the generation of Ports.pm presents a few challenges from a
packaging perspective. The simple approach is to generate it at
package build time against the default /etc/{services,protocols}.
However this is only useful if the user has made no local
modifications to/etc/{services,protocols}. The user who modifies these
files may not know to run buildports again. Also, another package may
(silently) modify /etc/{services,protocols} on installation, again
causing a problem.
One half-way house to fixing this is to generate Ports.pm at install
time. But this still breaks for the user under the above conditions.
So this led me to think it would be better if the compiler employed
some sort of logic like this:
Check if mtime of /etc/{services,protocols} is greater than mtime of
Ports.pm. If it is, regenerate Ports.pm before continuing. Otherwise
use the current Ports.pm.
(It might also be nice to have a shorewall.conf file flag to disable
this behaviour and always use Ports.pm)
What do you think? I think this would be a helpful change in favour of
usability, and would greatly simplify packaging for distributions.
If this would be an acceptable change, I could look at working up a
patch (which will push me to learn some Perl), unless someone beats me
to it :)
Best wishes, and thanks for a great tool,
Jonathan.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=321731
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
On Tue, Oct 09, 2007 at 12:16:21AM +0100, Jonathan Underwood wrote:> While working on the packaging of shorewall 4.0.4 for Fedora[1], a few > issues have arisen related to the shorewall-perl compiler, and the > file Ports.pm, the file which is essentially a hash of > /etc/{services,protocols}. > > Firstly, since this is an application file representing local machine > state, I think it should be kept under /var/lib/shorewall-perl rather > than /usr/share/shorewall-perl/Shorewall. This is particularly > pertinent for machines which have /usr mounted read only. > > Secondly, the generation of Ports.pm presents a few challenges from a > packaging perspective. The simple approach is to generate it at > package build time against the default /etc/{services,protocols}. > However this is only useful if the user has made no local > modifications to/etc/{services,protocols}. The user who modifies these > files may not know to run buildports again. Also, another package may > (silently) modify /etc/{services,protocols} on installation, again > causing a problem.I find myself wondering why buildports.pl exists at all. Tom, did you have anything in particular in mind? I observe the following on my (admittedly pretty fast) desktop: asuffield@cyclone:~/src/shorewall-perl-4.0.3$ time perl buildports.pl >/dev/null real 0m0.084s Given that the relevant files can be loaded and parsed in under .1 seconds (and I would not expect them to be difficult to parse, since libc does it all the time anyway, in each new process spawned), there does not appear to be a performance issue here that would prevent parsing services and protocols every time the shorewall compiler is invoked. Is there some other reason for doing it this way? ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On 09/10/2007, Andrew Suffield <asuffield@suffields.me.uk> wrote:> I find myself wondering why buildports.pl exists at all. Tom, did you > have anything in particular in mind? I observe the following on my > (admittedly pretty fast) desktop: > > asuffield@cyclone:~/src/shorewall-perl-4.0.3$ time perl buildports.pl >/dev/null > real 0m0.084s > > Given that the relevant files can be loaded and parsed in under .1 > seconds (and I would not expect them to be difficult to parse, since > libc does it all the time anyway, in each new process spawned), there > does not appear to be a performance issue here that would prevent > parsing services and protocols every time the shorewall compiler is > invoked. Is there some other reason for doing it this way? >A previous announcement from Tom contained this: "3) Previously, Shorewall-perl read /etc/protocols and /etc/services during compiler startup to build internal protocol and service tables. This had a fixed cost of up to one half second or more, depending on the speed of the system and the distribution (The /etc/services released with OpenSuSE 10.2 is over 14,000 lines!!) These tables are now initialized by the Perl compiler which speeds up compilation considerably." So, I am guessing that you (Andrew) are using a distribution with a sane number of lines in /etc/services, rather than OpenSuSe :). Jonathan. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Andrew Suffield wrote:> > I find myself wondering why buildports.pl exists at all. Tom, did you > have anything in particular in mind? I observe the following on my > (admittedly pretty fast) desktop: > > asuffield@cyclone:~/src/shorewall-perl-4.0.3$ time perl buildports.pl >/dev/null > real 0m0.084s > > Given that the relevant files can be loaded and parsed in under .1 > seconds (and I would not expect them to be difficult to parse, since > libc does it all the time anyway, in each new process spawned), there > does not appear to be a performance issue here that would prevent > parsing services and protocols every time the shorewall compiler is > invoked. Is there some other reason for doing it this way?Not all distributions include a tiny /etc/services (like Debian does). gateway:~ # wc -l /etc/services 14705 /etc/services gateway:~ # time shorewall restart Compiling... Shorewall configuration compiled to /root/shorewall/.restart Restarting Shorewall.... done. real 0m1.131s user 0m0.404s sys 0m0.304s gateway:~ # # time /usr/share/shorewall-perl/buildports.pl > /dev/null real 0m0.412s user 0m0.368s sys 0m0.040s gateway:~ # -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On Mon, Oct 08, 2007 at 04:35:22PM -0700, Tom Eastep wrote:> Andrew Suffield wrote: > > I find myself wondering why buildports.pl exists at all. Tom, did you > > have anything in particular in mind? I observe the following on my > > (admittedly pretty fast) desktop: > > ... > > Not all distributions include a tiny /etc/services (like Debian does). > > gateway:~ # wc -l /etc/services > 14705 /etc/services > gateway:~ # time shorewall restart > Compiling... > Shorewall configuration compiled to /root/shorewall/.restart > Restarting Shorewall.... > done. > > real 0m1.131s > user 0m0.404s > sys 0m0.304s > gateway:~ # # time /usr/share/shorewall-perl/buildports.pl > /dev/null > > real 0m0.412s > user 0m0.368s > sys 0m0.040s > gateway:~ #Well, given that glibc is parsing both those files during the startup of most significant network applications and I doubt SuSE would accept that big a penalty, my working theory is that your perl parser is just slow compared to the libc one (likely, since C code is usually much faster than perl). There''s no good reason for reimplementing the libc logic anyway, and you should be letting libc handle it just to get nsswitch.conf support. Try the attached patch which switches it over to using perl''s builtin interface to NSS, see if that helps. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On 09/10/2007, Andrew Suffield <asuffield@suffields.me.uk> wrote:> Well, given that glibc is parsing both those files during the startup > of most significant network applications and I doubt SuSE would accept > that big a penalty, my working theory is that your perl parser is just > slow compared to the libc one (likely, since C code is usually much > faster than perl). There''s no good reason for reimplementing the libc > logic anyway, and you should be letting libc handle it just to get > nsswitch.conf support. Try the attached patch which switches it over > to using perl''s builtin interface to NSS, see if that helps. >On a Fedora 7 machine: $ time ./buildports-orig.pl >/dev/null real 0m0.261s user 0m0.249s sys 0m0.012s $ time ./buildports-patched.pl >/dev/null real 0m0.130s user 0m0.115s sys 0m0.015s J. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Andrew Suffield wrote:> Try the attached patch which switches it over > to using perl''s builtin interface to NSS, see if that helps.Still not cheap: gateway:~ # time /usr/share/shorewall-perl/buildports.pl > /dev/null real 0m0.207s user 0m0.200s sys 0m0.000s gateway:~ # -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On Mon, Oct 08, 2007 at 05:02:23PM -0700, Tom Eastep wrote:> Andrew Suffield wrote: > > > Try the attached patch which switches it over > > to using perl''s builtin interface to NSS, see if that helps. > > Still not cheap: > > gateway:~ # time /usr/share/shorewall-perl/buildports.pl > /dev/null > > real 0m0.207s > user 0m0.200s > sys 0m0.000s > gateway:~ #How much of that is the overhead of compiling Shorewall::Config and generating the output? On my system, around 90% of it turns out to be overhead. Realistic test script follows, this should be the real penalty for running on startup: our %protocols; our %services; while ( my ($proto1, $aliases, $number) = getprotoent() ) { my @aliases = split('' '', $aliases); foreach my $name ($proto1, @aliases) { $protocols{$name} = $number; } } while ( my ($name1, $aliases, $number, $proto) = getservent() ) { my @names = split('' '', $aliases); next unless $proto && ($proto eq ''tcp'' || $proto eq ''udp''); foreach my $name ($name1, @aliases) { $services{$name} = $number; } } Note also that if you only need lookup functions, and not a full listing of all the ports, then using the builtins getservbyname() and getprotobyname() directly will be even faster, since it won''t have to drag the entire thing into perl''s working space. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On Tue, Oct 09, 2007 at 12:57:23AM +0100, Jonathan Underwood wrote:> > $ time ./buildports-orig.pl >/dev/null >Just as a note, you are almost always better off running /usr/bin/time explicitly. This is because time is a shell builtin function (at least in Bash, not sure about other shells), which is inferior to the /usr/bin/time program. Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On Tue, Oct 09, 2007 at 01:12:31AM +0100, Andrew Suffield wrote:> Note also that if you only need lookup functions,A brief grep indicates that this is indeed the case, so I strongly suggest scrapping Shorewall::Ports entirely, and modifying Shorewall::Chains: my $value = $protocols{$proto}; to my $value = getprotobyname($proto); Etcetera. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Roberto C. Sánchez wrote:> On Tue, Oct 09, 2007 at 12:57:23AM +0100, Jonathan Underwood wrote: >> $ time ./buildports-orig.pl >/dev/null >> > Just as a note, you are almost always better off running /usr/bin/time > explicitly. This is because time is a shell builtin function (at least > in Bash, not sure about other shells), which is inferior to the > /usr/bin/time program.I hate /usr/bin/time -- the output looks like someone''s first program; it often takes me longer to parse the output than it does to run the program that I''m trying to time. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On Mon, Oct 08, 2007 at 06:37:00PM -0700, Tom Eastep wrote:> Roberto C. Sánchez wrote: > > On Tue, Oct 09, 2007 at 12:57:23AM +0100, Jonathan Underwood wrote: > >> $ time ./buildports-orig.pl >/dev/null > >> > > Just as a note, you are almost always better off running /usr/bin/time > > explicitly. This is because time is a shell builtin function (at least > > in Bash, not sure about other shells), which is inferior to the > > /usr/bin/time program. > > I hate /usr/bin/time -- the output looks like someone''s first program; > it often takes me longer to parse the output than it does to run the > program that I''m trying to time. >That can easily be fixed by sending the output through a sed, awk or perl script. :-) Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Roberto C. Sánchez wrote:>> > That can easily be fixed by sending the output through a sed, awk or > perl script. :-) >No comment. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Andrew Suffield wrote:> On Tue, Oct 09, 2007 at 01:12:31AM +0100, Andrew Suffield wrote: >> Note also that if you only need lookup functions, > > A brief grep indicates that this is indeed the case, so I strongly > suggest scrapping Shorewall::Ports entirely, and modifying > Shorewall::Chains: > > my $value = $protocols{$proto}; > > to > > my $value = getprotobyname($proto); >Or something vaguely similar... buildports.pl and Shorewall::Ports.pm will be gone in 4.0.5. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
> Hello, > > While working on the packaging of shorewall 4.0.4 for Fedora[1], a few > issues have arisen related to the shorewall-perl compiler, and the > file Ports.pm, the file which is essentially a hash of > /etc/{services,protocols}.Hi, I''m maintaining Shorewall packages for RedHat/Fedora for a long time now and my packages deal with the issue in a smart way. Ports.pm is located in /var/lib/shorewall/ and also a cached copy of /etc/{services,protocols}. Ports.pm is rebuilt as part of the startup process whenever a change has been made to /etc/{services,protocols}. Why reinvent the wheel :) The packages can be found via the Shorewall download site: http://www.shorewall.net/download.htm#Sites Regards, Simon ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On 09/10/2007, Simon Matter <simon.matter@invoca.ch> wrote:> > I''m maintaining Shorewall packages for RedHat/Fedora for a long time now > and my packages deal with the issue in a smart way. Ports.pm is located in > /var/lib/shorewall/ and also a cached copy of /etc/{services,protocols}. > Ports.pm is rebuilt as part of the startup process whenever a change has > been made to /etc/{services,protocols}. Why reinvent the wheel :) > > The packages can be found via the Shorewall download site: > http://www.shorewall.net/download.htm#SitesThanks for the pointer Simon, I''ll take a look, but it sounds like the issue will go away with the release 4.0.5 anyway. Have you considered becoming a Fedora contributor to help maintain the shorewall packages in the Fedora and EPEL repositories? J. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/