Shorewall users - May 2006 - Three questions: Error message during startup, load balancing, masq file

Hi list,

I have three questions.  The first is related to an error message that I 
receive when Shorewall is starting.  The second is related to load 
balancing through 1 gateway, and the last question relates to the masq 
file when using multiple providers.

All three questions are related to the same Shorewall configuration 
running on only one server.  I am running:
Debian 3.1 (using the unstable branch)
Shorewall 3.0.7
iptables 1.3.3-2
Kernel 2.6.16-12

Question #1:
I just noticed an error message in my Shorewall startup logs.  This 
seems to have appeared since I upgraded to Shorewall 3.0.7 (from 3.0.5). 
While starting, Shorewall displays the following error message:
<begin copied text>
Processing /etc/shorewall/providers...
/usr/share/shorewall/firewall: line 1393: 20000 + (1 - 1) * 256 + 
$rulenum : syntax error: operand expected (error token is "$rulenum ")
   Provider CPE1 1 1 main eth1 detect track,balance br0,eth4 Added
/usr/share/shorewall/firewall: line 1393: 20000 + (2 - 1) * 256 + 
$rulenum : syntax error: operand expected (error token is "$rulenum ")
   Provider CPE2 2 2 main eth2 detect track,balance br0,eth4 Added
/usr/share/shorewall/firewall: line 1393: 20000 + (3 - 1) * 256 + 
$rulenum : syntax error: operand expected (error token is "$rulenum ")
   Provider CPE3 3 3 main eth3 detect track,balance br0,eth4 Added
   Default route  nexthop via 67.60.64.1 dev eth1 weight 1 nexthop via 
67.60.64.1 dev eth2 weight 1 nexthop via 67.60.64.1 dev eth3 weight 1 
Added.
<end copied text>

Shorewall starts successfully, and everything seems to work as expected, 
but I don''t know what this error message indicates.  I''ve
tried
searching the mailing list archives as well as Google, but I was not 
successful in finding a solution. My configuration files are attached.

Question #2:
This question is mostly out of curiosity.  The problem described here 
doesn''t affect me in any material way, but I''m just interested
in what
might be going on.

My ISP provides me with three routable IP addresses, which are all 
assigned to my router/firewall Shorewall box (it has 5 NICs in it, 3 are 
for connecting to my ISP).  I use the tcrules file to direct certain 
connections out of certain interfaces.  That all works beautifully.

Recently, all three interfaces leased IP addresses in the 67.60.64.0/21 
subnet with the same gateway IP address (67.60.64.1).  Prior to this 
happening, I had leases from three different IP subnets with three 
different gateways.  When I had the different gateways, all of my 
outbound connections were load balanced.  Now with the single gateway 
for all three interfaces, my outbound connections are all made through 
eth3.  This doesn''t affect me in any significant way since I can still 
use the tcrules file to direct traffic out of a desired interface, but 
my question is this:
Is it possible to load balance when all three interfaces share the same 
gateway?  I suspect that it is not possible, but I''m just curious.

Question #3:
In the Shorewall documentation on multiple Internet connections through 
a single firewall, it reads:

<begin copied text>
Regardless of whether you have masqueraded hosts or not, YOU MUST ADD 
THESE TWO ENTRIES TO /etc/shorewall/masq:

#INTERFACE       SUBNET            ADDRESS
eth0             130.252.99.27     206.124.146.176
eth1             206.124.146.176   130.252.99.27
<end copied text>

I don''t have any entries similar to the ones above in my masq file and 
it still seems to work properly.  I have tried adding the following 
entries into my masq file (which is my best guess about how to extend 
the above example for a three-interface configuration such as mine):
<begin copied text>
eth1                   $ETH2_IP        $ETH1_IP
eth1                   $ETH3_IP        $ETH1_IP
eth2                   $ETH1_IP        $ETH2_IP
eth2                   $ETH3_IP        $ETH2_IP
eth3                   $ETH1_IP        $ETH3_IP
eth3                   $ETH2_IP        $ETH3_IP
<end copied text>

With the above entries in my configuration file, everything still seems 
to work, but it takes about 10 seconds to establish a new connection. 
So I guess that my question #3 is really a three-in-one:
1. Have I correctly expanded the two entries in the Shorewall 
documentation to match a three-connection configuration?
2. What purpose do those two entries actually serve?
3. Is there a source of documentation that someone can recommend that 
might help me more fully understand the purpose of those entries?


I appreciate all your efforts, and thanks for the help.

Thanks and best regards
-Russel Riley

Note:
The attached shorewall.tar.gz has all my config files with the comments 
stripped away.  For the full comments, try 
http://myweb.cableone.net/rusabus/shorewallwithcomments.tar.gz

Russel wrote:
> /usr/share/shorewall/firewall: line 1393: 20000 + (1 - 1) * 256 +
> $rulenum : syntax error: operand expected (error token is "$rulenum
")
>
That''s a bug -- there is an updated ''firewall'' script
at
http://www.shorewall.net/pub/shorewall/3.0/shorewall-3.0.7/errata/
> Question #2:
> This question is mostly out of curiosity.  The problem described here
> doesn''t affect me in any material way, but I''m just
interested in what
> might be going on.
> 
> My ISP provides me with three routable IP addresses, which are all
> assigned to my router/firewall Shorewall box (it has 5 NICs in it, 3 are
> for connecting to my ISP).  I use the tcrules file to direct certain
> connections out of certain interfaces.  That all works beautifully.

And is completely silly unless you have a *very* fast internet
connection (in which case, you had better have a really fast PC for a
firewall if it requires 3 fast ethernet NICs to drive the internet
connection). Otherwise, it provides no performance benefit over using
one NIC with 3 IP addresses with different entries in
/etc/shorewall/masq to divide
> 
> Recently, all three interfaces leased IP addresses in the 67.60.64.0/21
> subnet with the same gateway IP address (67.60.64.1).  Prior to this
> happening, I had leases from three different IP subnets with three
> different gateways.  When I had the different gateways, all of my
> outbound connections were load balanced.  Now with the single gateway
> for all three interfaces, my outbound connections are all made through
> eth3.  This doesn''t affect me in any significant way since I can
still
> use the tcrules file to direct traffic out of a desired interface, but
> my question is this:
> Is it possible to load balance when all three interfaces share the same
> gateway?  I suspect that it is not possible, but I''m just curious.
Again, I think that this is a really silly idea unless there are factors
you aren''t telling us about -- it provides absolutely no benefit and
getting it to work when the addresses are in the same IP network is
difficult (as you are finding out). This has the same root cause that is
at the heart of the warnings in the documentation to not connect
multiple firewall interfaces to the same hub/switch. And you are
probably also seeing Martian warnings in your log if you are logging
martians on your external interfaces.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

Tom Eastep wrote:
> Russel wrote:
> 
> Again, I think that this is a really silly idea unless there are factors
> you aren''t telling us about 
I''m guessing that you came up with this idea because your IP addresses
are dynamic? If so, I would configure the firewall as a combination
bridge/router so that two hosts in a DMZ can use two of the three IP
addresses.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

Tom Eastep wrote:
>> Is it possible to load balance when all three interfaces share the same
>> gateway?  I suspect that it is not possible, but I''m just
curious.
> 
> Again, I think that this is a really silly idea unless there are factors
> you aren''t telling us about -- it provides absolutely no benefit
and
> getting it to work when the addresses are in the same IP network is
> difficult (as you are finding out). This has the same root cause that is
> at the heart of the warnings in the documentation to not connect
> multiple firewall interfaces to the same hub/switch. And you are
> probably also seeing Martian warnings in your log if you are logging
> martians on your external interfaces.
If you want to continue to try to run with your current configuration,
you can probably restore your "load-balancing" by setting arp_ignore=1
on each of the three interfaces in /etc/shorewall/interfaces.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

Russel wrote:
>
> With the above entries in my configuration file, everything still seems
> to work, but it takes about 10 seconds to establish a new connection. So
> I guess that my question #3 is really a three-in-one:
> 1. Have I correctly expanded the two entries in the Shorewall
> documentation to match a three-connection configuration?
Yes.
> 2. What purpose do those two entries actually serve?
> 3. Is there a source of documentation that someone can recommend that
> might help me more fully understand the purpose of those entries?
> 
I''ve updated the Multi-ISP document to explain why those entries are
there.

	http://www1.shorewall.net/MultiISP.html

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

> Russel wrote:
>
> > /usr/share/shorewall/firewall: line 1393: 20000 + (1 - 1) * 256 +
> > $rulenum : syntax error: operand expected (error token is
"$rulenum
> > ")
> >
>
> That''s a bug -- there is an updated ''firewall''
script at
> http://www.shorewall.net/pub/shorewall/3.0/shorewall-3.0.7/errata/
>
The updated script eliminated the error messages.
Thanks!
> > Question #2:
> > This question is mostly out of curiosity.  The problem described 
> > here
> > doesn''t affect me in any material way, but I''m just
interested in
> > what
> > might be going on.
> >
> > My ISP provides me with three routable IP addresses, which are all
> > assigned to my router/firewall Shorewall box (it has 5 NICs in it, 3 
> > are
> > for connecting to my ISP).  I use the tcrules file to direct certain
> > connections out of certain interfaces.  That all works beautifully.
>
>
> And is completely silly unless you have a *very* fast internet
> connection (in which case, you had better have a really fast PC for a
> firewall if it requires 3 fast ethernet NICs to drive the internet
> connection). Otherwise, it provides no performance benefit over using
> one NIC with 3 IP addresses with different entries in
> /etc/shorewall/masq to divide
I agree that it is pretty silly and that it dosen''t provide me with any
performance advantage.  My internet connection is not nearly fast enough 
to saturate a single 100 mbit (or even 10 mbit) NIC.

I would actually like to get rid of two of the nics that are connected 
to my cable modem.  The purpose for having them is to allow three 
simultaneous connections to the same Starcraft game on Battle.net.  I 
would like to do the same thing through only one interface, but I don''t
know how to make dhclient lease three IP addresses for only one 
interface.
>
> >
> > Recently, all three interfaces leased IP addresses in the 
> > 67.60.64.0/21
> > subnet with the same gateway IP address (67.60.64.1).  Prior to this
> > happening, I had leases from three different IP subnets with three
> > different gateways.  When I had the different gateways, all of my
> > outbound connections were load balanced.  Now with the single 
> > gateway
> > for all three interfaces, my outbound connections are all made 
> > through
> > eth3.  This doesn''t affect me in any significant way since I
can
> > still
> > use the tcrules file to direct traffic out of a desired interface, 
> > but
> > my question is this:
> > Is it possible to load balance when all three interfaces share the 
> > same
> > gateway?  I suspect that it is not possible, but I''m just
curious.
>
> Again, I think that this is a really silly idea unless there are 
> factors
> you aren''t telling us about -- it provides absolutely no benefit
and
> getting it to work when the addresses are in the same IP network is
> difficult (as you are finding out). This has the same root cause that 
> is
> at the heart of the warnings in the documentation to not connect
> multiple firewall interfaces to the same hub/switch. And you are
> probably also seeing Martian warnings in your log if you are logging
> martians on your external interfaces.
>
> -Tom
> -- 
> Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
> Shoreline,     \ http://shorewall.net
> Washington USA  \ teastep@shorewall.net
> PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
>

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

> Russel wrote:
>
>> /usr/share/shorewall/firewall: line 1393: 20000 + (1 - 1) * 256 +
>> $rulenum : syntax error: operand expected (error token is
"$rulenum ")
>>
>
> That''s a bug -- there is an updated ''firewall''
script at
> http://www.shorewall.net/pub/shorewall/3.0/shorewall-3.0.7/errata/
With the new firewall script, I get en error like this:
Validating /etc/shorewall/tcdevices...
Validating /etc/shorewall/tcclasses...
Processing /etc/shorewall/tcdevices...
/usr/share/shorewall/firewall: line 3375: get_device_mtu: command not found
Command line is not complete. Try option "help"
Processing /etc/shorewall/stop ...
   WARNING: DISABLE_IPV6=Yes in shorewall.conf but this system does not
appear to have ip6tables
IP Forwarding Enabled

The function get_device_mtu doesn''t exist, is there another file which
should be updated for the new firewall script to work?

Simon
>
>> Question #2:
>> This question is mostly out of curiosity.  The problem described here
>> doesn''t affect me in any material way, but I''m just
interested in what
>> might be going on.
>>
>> My ISP provides me with three routable IP addresses, which are all
>> assigned to my router/firewall Shorewall box (it has 5 NICs in it, 3
are
>> for connecting to my ISP).  I use the tcrules file to direct certain
>> connections out of certain interfaces.  That all works beautifully.
>
>
> And is completely silly unless you have a *very* fast internet
> connection (in which case, you had better have a really fast PC for a
> firewall if it requires 3 fast ethernet NICs to drive the internet
> connection). Otherwise, it provides no performance benefit over using
> one NIC with 3 IP addresses with different entries in
> /etc/shorewall/masq to divide
>
>>
>> Recently, all three interfaces leased IP addresses in the 67.60.64.0/21
>> subnet with the same gateway IP address (67.60.64.1).  Prior to this
>> happening, I had leases from three different IP subnets with three
>> different gateways.  When I had the different gateways, all of my
>> outbound connections were load balanced.  Now with the single gateway
>> for all three interfaces, my outbound connections are all made through
>> eth3.  This doesn''t affect me in any significant way since I
can still
>> use the tcrules file to direct traffic out of a desired interface, but
>> my question is this:
>> Is it possible to load balance when all three interfaces share the same
>> gateway?  I suspect that it is not possible, but I''m just
curious.
>
> Again, I think that this is a really silly idea unless there are factors
> you aren''t telling us about -- it provides absolutely no benefit
and
> getting it to work when the addresses are in the same IP network is
> difficult (as you are finding out). This has the same root cause that is
> at the heart of the warnings in the documentation to not connect
> multiple firewall interfaces to the same hub/switch. And you are
> probably also seeing Martian warnings in your log if you are logging
> martians on your external interfaces.
>
> -Tom
> --
> Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
> Shoreline,     \ http://shorewall.net
> Washington USA  \ teastep@shorewall.net
> PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
>
>

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Simon Matter wrote:
>> Russel wrote:
>>
>>> /usr/share/shorewall/firewall: line 1393: 20000 + (1 - 1) * 256 +
>>> $rulenum : syntax error: operand expected (error token is
"$rulenum ")
>>>
>> That''s a bug -- there is an updated
''firewall'' script at
>> http://www.shorewall.net/pub/shorewall/3.0/shorewall-3.0.7/errata/
> 
> With the new firewall script, I get en error like this:
> Validating /etc/shorewall/tcdevices...
> Validating /etc/shorewall/tcclasses...
> Processing /etc/shorewall/tcdevices...
> /usr/share/shorewall/firewall: line 3375: get_device_mtu: command not found
> Command line is not complete. Try option "help"
>
I''ve corrected the problem -- the firewall script in the errata no
longer requires that new function.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key