Shorewall 4.4.22.3 is now available for download. Problems Corrected: 1) On older distributions where ''shorewall show capabilities'' indicates ''Connection Tracking Match: Not Available'', harmless Perl diagnostics like the following could be issued: Use of uninitialized value $list in pattern match (m//) at /usr/share/shorewall/Shorewall/Config.pm line 1273, <$currentfile> line 14. Use of uninitialized value $list in split at /usr/share/shorewall/Shorewall/Config.pm line 1275, <$currentfile> line 14. 2) On older distributions where ''shorewall show capabilities'' indicates ''Mangle FORWARD Chain: Not Available'', entries in the ecn file generated the following Perl Diagnostic: Use of uninitialized value in hash element at /usr/share/shorewall/Shorewall/Chains.pm line 1119. 3) Previously, if a provider interface was derived from an optional wildcard entry in /etc/shorewall/providers, then the interface was never considered to be usable. Example: /etc/shorewall/interfaces: #ZONE INTERFACE BROADCAST OPTIONS net ppp+ - optionsl /etc/shorewall/providers: #PROVIDER NUMBER MARK INTERFACE ... ISP1 1 1 ppp0 ... Also: 1) When ''shorewall update'' or ''shorewall6 update'' results in no change to the .conf file, a message is issued, the .bak file is removed and the command terminates without error. Thank you for using Shorewall, -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On 08/20/2011 06:29 PM, Tom Eastep wrote:> Shorewall 4.4.22.3 is now available for download.Where ? as the tar packages are not in any of the dowmload sites. or am I missing the obvious Togan ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On Aug 20, 2011, at 11:12 AM, Togan Muftuoglu wrote:> On 08/20/2011 06:29 PM, Tom Eastep wrote: >> Shorewall 4.4.22.3 is now available for download. > > Where ? as the tar packages are not in any of the dowmload sites. or am > I missing the obvious >I''ve checked four sites (other than my own), and it is in all of them. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On Aug 20, 2011, at 11:29 AM, Tom Eastep wrote:> > On Aug 20, 2011, at 11:12 AM, Togan Muftuoglu wrote: > >> On 08/20/2011 06:29 PM, Tom Eastep wrote: >>> Shorewall 4.4.22.3 is now available for download. >> >> Where ? as the tar packages are not in any of the dowmload sites. or am >> I missing the obvious >> > > I''ve checked four sites (other than my own), and it is in all of them.For example, http://slovakia.shorewall.net/pub/shorewall/4.4/shorewall-4.4.22/ -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On 20/08/2011 17:29, Tom Eastep wrote:> 3) Previously, if a provider interface was derived from an optional > wildcard entry in /etc/shorewall/providers, then the interface was > never considered to be usable.Thanks for fixing this. I notice a new gremlin in 4.4.22.3: shorewall restart seems to choke? Possibly a locking issue, struggling a little to debug. sh -x shorewall restart gives: ... + /bin/sh /var/lib/shorewall/firewall version + sed s/-.*// + temp=4.4.22.3 + [ 0 -ne 0 ] + ifs + IFS=. + echo 4 4 22 3 + temp=4 4 22 3 + IFS + digits=0 + printf %02d 4 + version=04 + digits=1 + [ 1 -eq 3 ] + printf %02d 4 + version=0404 + digits=2 + [ 2 -eq 3 ] + printf %02d 22 + version=040422 + digits=3 + [ 3 -eq 3 ] + break + echo 040422 + VERBOSITY=1 + version=040422 + [ 040422 -lt 040408 ] + [ xrestart = xtrace -o xrestart = xdebug ] + options=- + [ -n ] + [ -n ] + [ -n ] + [ -n ] + options=-V 1 + [ -n restore ] + options=-V 1 -R restore + /bin/sh /var/lib/shorewall/firewall -V 1 -R restore restart Restarting Shorewall.... Initializing... Processing /etc/shorewall/init ... Command: restart then a long pause here and then every 45 (ish) seconds, I get: ...pause... ipset v6.8: Set cannot be created: set with the same name already exists ...pause... ipset v6.8: Set cannot be created: set with the same name already exists ..pause... Which corresponds with my init file of: if [ "$COMMAND" = start -o "$COMMAND" = restart ]; then ipset create cp1 bitmap:ip,mac range $LOC_IP ipset create cp2 bitmap:ip,mac range $LOC_IP ipset create cp3 bitmap:ip,mac range $LOC_IP ... fi I don''t have 4.4.22.1 around to double check, but I don''t think I was hitting this before A follow-on minor gremlin is that if I control-C to kill this, then there is a stale lock file left - this causes some follow on slowness trying to do anything subsequently (a stale lock test might be nice?) Thanks for any help troubleshooting? Ed W ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On Sun, 2011-08-21 at 15:43 +0100, Ed W wrote:> On 20/08/2011 17:29, Tom Eastep wrote: > > 3) Previously, if a provider interface was derived from an optional > > wildcard entry in /etc/shorewall/providers, then the interface was > > never considered to be usable. > > Thanks for fixing this. > > I notice a new gremlin in 4.4.22.3:That gremlin has been around for a long time and as you point out below, it can happen if the generated script is killed while it is running.> > shorewall restart seems to choke? Possibly a locking issue, struggling > a little to debug. > > sh -x shorewall restart gives: > > ... > + /bin/sh /var/lib/shorewall/firewall version > + sed s/-.*// > + temp=4.4.22.3 > + [ 0 -ne 0 ] > + ifs> > + IFS=. > + echo 4 4 22 3 > + temp=4 4 22 3 > + IFS> > + digits=0 > + printf %02d 4 > + version=04 > + digits=1 > + [ 1 -eq 3 ] > + printf %02d 4 > + version=0404 > + digits=2 > + [ 2 -eq 3 ] > + printf %02d 22 > + version=040422 > + digits=3 > + [ 3 -eq 3 ] > + break > + echo 040422 > + VERBOSITY=1 > + version=040422 > + [ 040422 -lt 040408 ] > + [ xrestart = xtrace -o xrestart = xdebug ] > + options=- > + [ -n ] > + [ -n ] > + [ -n ] > + [ -n ] > + options=-V 1 > + [ -n restore ] > + options=-V 1 -R restore > + /bin/sh /var/lib/shorewall/firewall -V 1 -R restore restart > Restarting Shorewall.... > Initializing... > Processing /etc/shorewall/init ... > Command: restart > > then a long pause here and then every 45 (ish) seconds, I get: > > ...pause... > ipset v6.8: Set cannot be created: set with the same name already exists > ...pause... > ipset v6.8: Set cannot be created: set with the same name already exists > ..pause... > > > Which corresponds with my init file of: > > if [ "$COMMAND" = start -o "$COMMAND" = restart ]; thenThe above is wrong. Doing a blanket create of ipsets during a restart will always result in failures if the ipsets are used in the current configuration.> ipset create cp1 bitmap:ip,mac range $LOC_IP > ipset create cp2 bitmap:ip,mac range $LOC_IP > ipset create cp3 bitmap:ip,mac range $LOC_IP > ... > fi > > > I don''t have 4.4.22.1 around to double check, but I don''t think I was > hitting this beforeThe patches from 4.4.22.1 -> 4.4.22.2 -> 4.4.22.3 are available from the download sites. And nothing in this area has changed for quite a while.> > > A follow-on minor gremlin is that if I control-C to kill this, then > there is a stale lock file left - this causes some follow on slowness > trying to do anything subsequently (a stale lock test might be nice?)As a workaround, simply remove the lockfile or set MUTEX_TIMEOUT to a few seconds. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On Sun, 2011-08-21 at 08:11 -0700, Tom Eastep wrote:> On Sun, 2011-08-21 at 15:43 +0100, Ed W wrote:> > > > A follow-on minor gremlin is that if I control-C to kill this, then > > there is a stale lock file left - this causes some follow on slowness > > trying to do anything subsequently (a stale lock test might be nice?) > > As a workaround, simply remove the lockfile or set MUTEX_TIMEOUT to a > few seconds.Or set it to zero and avoid locking altogether. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
Hi> That gremlin has been around for a long time and as you point out below, > it can happen if the generated script is killed while it is running.Hmm, just to be clear - are you saying that "shorewall restart" is *expected* to cause problems in the init script with locking? ("expected" as in, with the current code, not saying it''s a deliberate choice) I had a tiny glance through the "sh -x" trace because I wasn''t quite clear when the lock file was created - it''s not there before either start or restart, yet one of them creates a lock file.>> Which corresponds with my init file of: >> >> if [ "$COMMAND" = start -o "$COMMAND" = restart ]; then > > The above is wrong. Doing a blanket create of ipsets during a restart > will always result in failures if the ipsets are used in the current > configuration.Well, originally I deleted and recreated the ipsets each time, after some time I decided I just wanted to create any missing entries. It''s just a harmless warning to try and create an already existing ipset On reflection I still don''t understand what is happening here? I thought "init" was a single script which would be run as a single unit under a lock? What seems to be happening is that each individual line is getting individually subjected to some timeout - how is that happening? Is "ipset" somehow getting intercepted as a function instead of calling the ipset binary?>> A follow-on minor gremlin is that if I control-C to kill this, then >> there is a stale lock file left - this causes some follow on slowness >> trying to do anything subsequently (a stale lock test might be nice?) > > As a workaround, simply remove the lockfile or set MUTEX_TIMEOUT to a > few seconds.Thought occurs we could wrap access to the lockfile in a function and in there check there is a valid process with that pid? If you felt this was excessive stat-ing and a performance issue then we could solve the most obvious occurrence by checking of the pid is stale (once) near the start? What do you think? Ed W ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On Aug 21, 2011, at 8:54 AM, Ed W wrote:> > >>> Which corresponds with my init file of: >>> >>> if [ "$COMMAND" = start -o "$COMMAND" = restart ]; then >> >> The above is wrong. Doing a blanket create of ipsets during a restart >> will always result in failures if the ipsets are used in the current >> configuration. > > Well, originally I deleted and recreated the ipsets each time, after > some time I decided I just wanted to create any missing entries. It''s > just a harmless warning to try and create an already existing ipset > > On reflection I still don''t understand what is happening here?I couldn''t understand from you trace either.> I thought "init" was a single script which would be run as a single unit > under a lock?All of the shell code being executed can be found at /var/lib/shorewall/.start or .restart depending on the command. The use of a lock file depends on whether the command being executed changes the configuration or not. Those commands that change the configuration create a lock file, do their thing, and then remove the lock file.> What seems to be happening is that each individual line is > getting individually subjected to some timeout - how is that happening? > Is "ipset" somehow getting intercepted as a function instead of calling > the ipset binary? >I don''t know -- it is happening on your system, not mine. But again, you have access to the code. FWIW, your init file is wrapped in a function named run_init_exit().> > >>> A follow-on minor gremlin is that if I control-C to kill this, then >>> there is a stale lock file left - this causes some follow on slowness >>> trying to do anything subsequently (a stale lock test might be nice?) >> >> As a workaround, simply remove the lockfile or set MUTEX_TIMEOUT to a >> few seconds. > > Thought occurs we could wrap access to the lockfile in a function and in > there check there is a valid process with that pid? If you felt this > was excessive stat-ing and a performance issue then we could solve the > most obvious occurrence by checking of the pid is stale (once) near the > start? > > What do you think?/usr/bin/lockfile does not store the locker''s PID in the file; as far as I can tell, the file always contains ''0'' (with no newline). Using that program is the preferred way to create a lock file, since doing it in the shell is race-prone. But I''ll see what I can do for 4.4.23. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On Aug 21, 2011, at 9:46 AM, Tom Eastep wrote:> > /usr/bin/lockfile does not store the locker''s PID in the file; as far as I can tell, the file always contains ''0'' (with no newline). Using that program is the preferred way to create a lock file, since doing it in the shell is race-prone. But I''ll see what I can do for 4.4.23.Ed, I''ve added stale lock file detection/removal in the 4.4.23 branch. Regards, -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On Aug 21, 2011, at 10:31 AM, Tom Eastep wrote:> > On Aug 21, 2011, at 9:46 AM, Tom Eastep wrote: >> >> /usr/bin/lockfile does not store the locker''s PID in the file; as far as I can tell, the file always contains ''0'' (with no newline). Using that program is the preferred way to create a lock file, since doing it in the shell is race-prone. But I''ll see what I can do for 4.4.23. > > > Ed, > > I''ve added stale lock file detection/removal in the 4.4.23 branch. >I''ve also discovered that the way that I''m invoking lockfile is rather braindead. I''ve also corrected that in the 4.4.23 branch and, if I create another 4.4.22 point release, I''ll include it there as well. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
On 21/08/2011 20:00, Tom Eastep wrote:> > On Aug 21, 2011, at 10:31 AM, Tom Eastep wrote: > >> >> On Aug 21, 2011, at 9:46 AM, Tom Eastep wrote: >>> >>> /usr/bin/lockfile does not store the locker''s PID in the file; as far as I can tell, the file always contains ''0'' (with no newline). Using that program is the preferred way to create a lock file, since doing it in the shell is race-prone. But I''ll see what I can do for 4.4.23. >> >> >> Ed, >> >> I''ve added stale lock file detection/removal in the 4.4.23 branch. >> > > I''ve also discovered that the way that I''m invoking lockfile is rather braindead. I''ve also corrected that in the 4.4.23 branch and, if I create another 4.4.22 point release, I''ll include it there as well.Many thanks - I will look into the changes ASAP Ed W ------------------------------------------------------------------------------ Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2