Hi, I am using shorewall on a small "embedded" board and I don''t have/want "make" installed. At present there are two calls to make in /sbin/shorewall and neither do anything interesting other than check if "some file newer than some other file" (something we can easily do without make) Additionally, I tried copying gnu make across but the performance is pretty poor: $ time make make: Nothing to be done for `all''. real 0m 0.48s user 0m 0.35s sys 0m 0.09s So the attached patch uses "find" to instead check if any shorewall files have been updated (could also have been done in pure shell code easily enough if it''s a worry that find is installed?) Notes: - There is a redundant function "automake_restore_file", which emulates the complete make function - I left this in the patch purely in case there was some desire to simplify the make file (probably not, so likely it''s dead code?) - The new function to check if anything changed is "automake_restore_needed" - this should probably be moved to one of the /usr/share/shorewall/lib.xxx files - I left it in the main file for the patch since surely I will pick the wrong place to move it.... Grateful if you would consider accepting this functionality into the next shorewall release (for me a large performance increase, and eliminated of a dependency) Thanks Ed W ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 05/20/2011 05:20 AM, Ed W wrote:> Grateful if you would consider accepting this functionality into the > next shorewall release (for me a large performance increase, and > eliminated of a dependency)Hi Ed, This patch *should* be unnecessary. If you don''t want Shorewall to use ''make'' then: a) Set AUTO_MAKE=No in shorewall.conf b) Configure the init script (using /etc/default/shorewall or /etc/sysconfig/shorewall) to not supply the -f option. If you find a case where both of those are being done but ''make'' is still being run, then I would gladly accept a patch that corrects that behavior. Thanks, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
Hi>> Grateful if you would consider accepting this functionality into the >> next shorewall release (for me a large performance increase, and >> eliminated of a dependency) > > Hi Ed, > > This patch *should* be unnecessary. If you don''t want Shorewall to use > ''make'' then:Let me rephrase my problem description: I *want* the AUTO_MAKE=Yes type functionality, ie that the rules are only compiled if they have changed, but I don''t want to have to install "make" on the target box to achieve this functionality Does that make sense? The only place I can see "make" being used in the code is two small places which only call "make -q" (-q = run no commands), ie shorewall does not appear to *call* make to do any work, it only uses it as a (very expensive) test to see if any /etc/shorewall/* files are updated. However, as you can easily see, this can easily be done much more cheaply without calling the make binary itself. So if you apply my patch and test it with AUTO_MAKE=Yes, then you should see that shorewall still functions as before, however, performance is substantially improved during restart and there is no dependency on the make executable (as my stats show, it saves around 0.5sec or 1/3 of the time needed to restart) Note, nothing in this patch stops a user installing "make" and running it in /etc/shorewall - this still works fine. However, that was what my mutterings about unused code meant - you *could* simplify the Makefile and move the functionality into the /sbin/shorewall file, however, I suspect you will actually prefer to leave as is, hence cull that additional function. However, I left it in so you can see the idea? I hope this clarifies the situation and you will look again at the patch? Thanks Ed W ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On May 22, 2011, at 3:44 AM, Ed W wrote:> Hi > >>> Grateful if you would consider accepting this functionality into the >>> next shorewall release (for me a large performance increase, and >>> eliminated of a dependency) >> >> Hi Ed, >> >> This patch *should* be unnecessary. If you don''t want Shorewall to use >> ''make'' then: > > Let me rephrase my problem description: > > I *want* the AUTO_MAKE=Yes type functionality, ie that the rules are > only compiled if they have changed, but I don''t want to have to install > "make" on the target box to achieve this functionality > > Does that make sense?Yes -- attached is a similar patch that avoids all of the manipulation of RESTOREFILE. -Tom Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 5/22/11 9:48 AM, Tom Eastep wrote:> > On May 22, 2011, at 3:44 AM, Ed W wrote:>> Does that make sense? > > Yes -- attached is a similar patch that avoids all of the manipulation of RESTOREFILE. >After applying the patch, the body of the added function needs to be modified as follows: # # Determine if there are config files newer than the passed object but # uptodate() { [ -f $1 ] && [ -z "$(find ${CONFDIR} -newer $1)" ] } -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 5/22/11 1:31 PM, Tom Eastep wrote:> On 5/22/11 9:48 AM, Tom Eastep wrote: >> >> On May 22, 2011, at 3:44 AM, Ed W wrote: > >>> Does that make sense? >> >> Yes -- attached is a similar patch that avoids all of the manipulation of RESTOREFILE. >> > > After applying the patch, the body of the added function needs to be > modified as follows: > > # > # Determine if there are config files newer than the passed object but > # > uptodate() { > [ -f $1 ] && [ -z "$(find ${CONFDIR} -newer $1)" ] > }Also notice that I corrected the name of the function. I only tested the Shorewall6 patch before posting it; obviously I should have tested Shorewall too. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 22/05/2011 21:31, Tom Eastep wrote:> On 5/22/11 9:48 AM, Tom Eastep wrote: >> >> On May 22, 2011, at 3:44 AM, Ed W wrote: > >>> Does that make sense? >> >> Yes -- attached is a similar patch that avoids all of the manipulation of RESTOREFILE. >> > > After applying the patch, the body of the added function needs to be > modified as follows: > > # > # Determine if there are config files newer than the passed object but # > uptodate() { > [ -f $1 ] && [ -z "$(find ${CONFDIR} -newer $1)" ] > }Good catch - I will test it with "shorewall" shortly and report back. Thanks for taking the idea and completing it! Can I get confirmation you intend to commit to core shorewall for the next release? Many thanks! Ed W ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 5/23/11 7:45 AM, Ed W wrote:> > Can I get confirmation you intend to commit to core shorewall for the > next release?Confirmed - 4.4.20. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 22/05/2011 21:50, Tom Eastep wrote:> Also notice that I corrected the name of the function. I only tested the > Shorewall6 patch before posting it; obviously I should have tested > Shorewall too.Just a heads up, but "diff -u /sbin/shorewall /sbin/shorewall6" is quite instructive? There are large chunks of differences such as * code re-ordered @@ -643,14 +553,14 @@ g_profile=Yes option=${option#p} ;; - d*) - g_debug=Yes; - option=${option#d} - ;; r*) g_preview=Yes; option=${option#r} ;; + d*) + g_debug=Yes; + option=${option#d} + ;; *) * some changed whitespace * slightly different calling conventions - if compiler $g_debugging $nolock compile ${VARDIR}/.start; then + if compiler run $g_debugging $nolock compile ${VARDIR}/.start; then * Possible bug? - [ -z "$LOGFORMAT" ] && LOGFORMAT=''Shorewall:%s.%s'' + [ -n "$LOGFORMAT" ] || LOGFORMAT=''Shorewall6:%s.%s'' It seems like we can largely merge both files though - the differences seem mostly down to support for ipset/tc/ipcalc/iprange? I''m hesitant to send a patch for the simple bits because it will be fragile if you are editing - do you see a future in trying to factor out the common code here? Thanks Ed W ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 5/23/11 8:02 AM, Ed W wrote:> On 22/05/2011 21:50, Tom Eastep wrote: >> Also notice that I corrected the name of the function. I only tested the >> Shorewall6 patch before posting it; obviously I should have tested >> Shorewall too. > > Just a heads up, but "diff -u /sbin/shorewall /sbin/shorewall6" is quite > instructive? > > There are large chunks of differences such as...> > > * Possible bug? > > - [ -z "$LOGFORMAT" ] && LOGFORMAT=''Shorewall:%s.%s'' > + [ -n "$LOGFORMAT" ] || LOGFORMAT=''Shorewall6:%s.%s''While the two are different, they are equivalent.> > > > It seems like we can largely merge both files though - the differences > seem mostly down to support for ipset/tc/ipcalc/iprange?We actually need ipset support to be added to Shorewall6 now that ipset supports IPv6. The thought of trying to add ipcalc and iprange for IPv6 has always made my head hurt.> > I''m hesitant to send a patch for the simple bits because it will be > fragile if you are editing - do you see a future in trying to factor out > the common code here?I think it would be a big benefit to factor out the common code in both shorewall/shorewall6 and shorewall-lite/shorewall6-lite. If you want to try to take that on, then I recommend that you clone the git tree so that you can be working with my latest changes. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 23/05/2011 16:34, Tom Eastep wrote:> I think it would be a big benefit to factor out the common code in both > shorewall/shorewall6 and shorewall-lite/shorewall6-lite. If you want to > try to take that on, then I recommend that you clone the git tree so > that you can be working with my latest changes.Sure - happy to give it a go I think we need an initial pass to clean up the easy bits, eg line endings, re-orderings and as you correctly say things that are equivalent but done differently. I develop on OSX and the big missing feature is some decent merge tool like Meld... Anyone got a visual merge suggestion for osx? Cheers Ed W ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 5/23/11 9:13 AM, Ed W wrote:> On 23/05/2011 16:34, Tom Eastep wrote: >> I think it would be a big benefit to factor out the common code in both >> shorewall/shorewall6 and shorewall-lite/shorewall6-lite. If you want to >> try to take that on, then I recommend that you clone the git tree so >> that you can be working with my latest changes. > > Sure - happy to give it a go > > I think we need an initial pass to clean up the easy bits, eg line > endings, re-orderings and as you correctly say things that are > equivalent but done differently. > > I develop on OSX and the big missing feature is some decent merge tool > like Meld... Anyone got a visual merge suggestion for osx?Wish I did. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
> I develop on OSX and the big missing feature is some decent merge tool > like Meld... Anyone got a visual merge suggestion for osx? >Try SourceGear''s DiffMerge (http://sourcegear.com/diffmerge) - I use it a lot as it is a cross-platform tool and is well-suited for more complicated (diff/merge) tasks like the one you are about to embark - this tool would be of help I am sure. Failing that, Perforce''s P4Merge or even Kaleidoscope (I am not sure the latter is free though!). ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 5/23/11 9:39 AM, Mr Dash Four wrote:> >> I develop on OSX and the big missing feature is some decent merge tool >> like Meld... Anyone got a visual merge suggestion for osx? >> > Try SourceGear''s DiffMerge (http://sourcegear.com/diffmerge) - I use it > a lot as it is a cross-platform tool and is well-suited for more > complicated (diff/merge) tasks like the one you are about to embark - > this tool would be of help I am sure. Failing that, Perforce''s P4Merge > or even Kaleidoscope (I am not sure the latter is free though!).Thanks for the tip -- I just installed DiffMerge. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 23/05/2011 18:07, Tom Eastep wrote:> On 5/23/11 9:39 AM, Mr Dash Four wrote: >> >>> I develop on OSX and the big missing feature is some decent merge tool >>> like Meld... Anyone got a visual merge suggestion for osx? >>> >> Try SourceGear''s DiffMerge (http://sourcegear.com/diffmerge) - I use it >> a lot as it is a cross-platform tool and is well-suited for more >> complicated (diff/merge) tasks like the one you are about to embark - >> this tool would be of help I am sure. Failing that, Perforce''s P4Merge >> or even Kaleidoscope (I am not sure the latter is free though!). > > Thanks for the tip -- I just installed DiffMerge.I finally tried vimdiff. OK, I have a love/hate affair with vim, but since I mostly develop in a console it''s handy Seems you get a good left/right diff, and dp/do push/pull the diffs each way. Ed W ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 23/05/2011 17:13, Ed W wrote:> I think we need an initial pass to clean up the easy bits, eg line > endings, re-orderings and as you correctly say things that are > equivalent but done differently.Can you please examine and commit the attached patch. Please check carefully the change to g_perllib ... I sense it''s previously wrong in one of the versions? Could you please look through the remaining diff: there is a bunch of calls with "runit" added in shorewall6 - I haven''t looked through the chain of code, but perhaps these are easy for you to reconcile? The end result is near identical left/right bar the change of names shorewall -> shorewall6 and iptables -> iptables6 How do you feel about referring to just "$IPTABLES" in both scripts? Any issue with abstracting something like $SHOREWALL=shorewall6 to unify all commandline calls? Cheers Ed W ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 5/23/11 2:06 PM, Ed W wrote:> Please check carefully the change to g_perllib ... I sense it''s > previously wrong in one of the versions?The installer rewrites the incorrect line so it really doesn''t make any difference. But you are correct that the shorewall6 value was wrong.> > Could you please look through the remaining diff: there is a bunch of > calls with "runit" added in shorewall6 - I haven''t looked through the > chain of code, but perhaps these are easy for you to reconcile? > > The end result is near identical left/right bar the change of names > shorewall -> shorewall6 and iptables -> iptables6 > > How do you feel about referring to just "$IPTABLES" in both scripts? Any > issue with abstracting something like $SHOREWALL=shorewall6 to unify all > commandline calls?IPTABLES and IP6TABLES come from /etc/shorewall/shorewall.conf and /etc/shorewall6/shorewall6.conf respectively. So neither can be renamed. What could be done is to define g_tool and set it appropriately. I''ve applied your patch to my tree. Thanks! -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay
On 23/05/2011 22:39, Tom Eastep wrote:>> >> Could you please look through the remaining diff: there is a bunch of >> calls with "runit" added in shorewall6 - I haven''t looked through the >> chain of code, but perhaps these are easy for you to reconcile?Hi Tom, Could you look over the usage of "run", "runit" and I think there is also a couple of "$cmd" that show if you do a simple diff shorewall{,6} I''m sure I can figure it out, but presumably you have the answer at your fingertips as to which way to merge the changes? Modify to look like shorewall6? (or is the call change more subtle?) Thanks Ed W ------------------------------------------------------------------------------ vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1
On 5/24/11 2:40 AM, Ed W wrote:> On 23/05/2011 22:39, Tom Eastep wrote:> Hi Tom, Could you look over the usage of "run", "runit" and I think > there is also a couple of "$cmd" that show if you do a simple diff > shorewall{,6} > > I''m sure I can figure it out, but presumably you have the answer at your > fingertips as to which way to merge the changes? Modify to look like > shorewall6? (or is the call change more subtle?)Hi Ed, I''ve done quite a bit of merging but haven''t tested it well enough yet to commit my work. I''ll try to do that in the next hour or so. Thanks, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1
On 5/24/11 6:10 AM, Tom Eastep wrote:> On 5/24/11 2:40 AM, Ed W wrote: >> On 23/05/2011 22:39, Tom Eastep wrote: > >> Hi Tom, Could you look over the usage of "run", "runit" and I think >> there is also a couple of "$cmd" that show if you do a simple diff >> shorewall{,6} >> >> I''m sure I can figure it out, but presumably you have the answer at your >> fingertips as to which way to merge the changes? Modify to look like >> shorewall6? (or is the call change more subtle?) > > Hi Ed, > > I''ve done quite a bit of merging but haven''t tested it well enough yet > to commit my work. I''ll try to do that in the next hour or so.If you do a ''pull origin master'', you should see that the two files are now very similar. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1