Hello all, I''m having a problem using multiple providers and shorewall on an ubuntu linux box. (Debian clone) My interfaces file does not have any routes assigned in it I read the shorewall and routing docs and set up my providers file as: #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS cogent 2 2 main eth1 66.28.28.79 track,balance=1 birns 3 3 main eth3 63.211.45.129 track,balance=2 MASQ file is set up: #INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC eth3 eth5 63.211.45.130 eth1 eth5 66.28.28.78 I don''t think rules are important here, and I haven''t set up anything else fancy except openvpn Tunnels file: # TYPE ZONE GATEWAY GATEWAY # ZONE openvpn net 4.79.128.50 Shorewall check completes successfully. A shorewall restart, however exits unsuccessfully Processing /etc/shorewall/providers... RTNETLINK answers: File exists Shorewall debug restart shows a lot of stuff, but I believe this is pertinent: + run_ip route show table main + ip route show table main + read net route + case $net in + ensure_and_save_command ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 + eval ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 ++ ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 + echo ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 + read net route + case $net in + ensure_and_save_command ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 + eval ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 ++ ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 + echo ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 + read net route + case $net in + ensure_and_save_command ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 + eval ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 ++ ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 + echo ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 + read net route + case $net in + ensure_and_save_command ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope link src 63.211.45.130 + eval ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope link src 63.211.45.130 ++ ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope link src 63.211.45.130 RTNETLINK answers: File exists Any ideas on what''s going on? Thanks in advance for any help. -Derek ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
I forgot one file, I am running some NAT: #EXTERNAL INTERFACE INTERNAL ALL INTERFACES LOCAL 63.211.45.142 eth3 192.168.1.245 No No 63.211.45.133 eth3 192.168.1.246 No No -----Original Message----- From: Derek Murawsky Sent: Tuesday, July 26, 2005 7:13 PM To: ''Mailing List for Shorewall Users'' Subject: Problems with providers Hello all, I''m having a problem using multiple providers and shorewall on an ubuntu linux box. (Debian clone) My interfaces file does not have any routes assigned in it I read the shorewall and routing docs and set up my providers file as: #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS cogent 2 2 main eth1 66.28.28.79 track,balance=1 birns 3 3 main eth3 63.211.45.129 track,balance=2 MASQ file is set up: #INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC eth3 eth5 63.211.45.130 eth1 eth5 66.28.28.78 I don''t think rules are important here, and I haven''t set up anything else fancy except openvpn Tunnels file: # TYPE ZONE GATEWAY GATEWAY # ZONE openvpn net 4.79.128.50 Shorewall check completes successfully. A shorewall restart, however exits unsuccessfully Processing /etc/shorewall/providers... RTNETLINK answers: File exists Shorewall debug restart shows a lot of stuff, but I believe this is pertinent: + run_ip route show table main + ip route show table main + read net route + case $net in + ensure_and_save_command ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 + eval ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 ++ ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 + echo ip route add table 2 10.1.0.1 dev tun0 proto kernel scope link src 10.1.0.2 + read net route + case $net in + ensure_and_save_command ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 + eval ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 ++ ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 + echo ip route add table 2 66.28.28.76/30 dev eth1 proto kernel scope link src 66.28.28.78 + read net route + case $net in + ensure_and_save_command ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 + eval ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 ++ ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 + echo ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 + read net route + case $net in + ensure_and_save_command ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope link src 63.211.45.130 + eval ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope link src 63.211.45.130 ++ ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope link src 63.211.45.130 RTNETLINK answers: File exists Any ideas on what''s going on? Thanks in advance for any help. -Derek ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
I forgot one file, I am running some NAT: #EXTERNAL INTERFACE INTERNAL ALL INTERFACES LOCAL 63.211.45.142 eth3 192.168.1.245 No No 63.211.45.133 eth3 192.168.1.246 No No -----Original Message----- From: Derek Murawsky Sent: Tuesday, July 26, 2005 7:13 PM To: ''Mailing List for Shorewall Users'' Subject: Problems with providers Hello all, I''m having a problem using multiple providers and shorewall on an ubuntu linux box. (Debian clone) My interfaces file does not have any routes assigned in it I read the shorewall and routing docs and set up my providers file as: #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS cogent 2 2 main eth1 66.28.28.79 track,balance=1 birns 3 3 main eth3 63.211.45.129 track,balance=2 MASQ file is set up: #INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC eth3 eth5 63.211.45.130 eth1 eth5 66.28.28.78 I don''t think rules are important here, and I haven''t set up anything else fancy except openvpn Tunnels file: # TYPE ZONE GATEWAY GATEWAY # ZONE openvpn net 4.79.128.50 <snip> Any ideas on what''s going on? Thanks in advance for any help. -Derek I know shorewall is not started, but can I get a shorewall status, just to see the routing and interface info. A discription of what each nic is plugged into can be helpful also. Jerry ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
After going through the debug output I had a brainstorm (more like a trickle) and got the dual providers to work. It had something to do with how I had assigned the extra external IP using ip addr in the interfaces (Debian) file. However after getting that working, the DNATs that I had set up no longer functioned. Well, I ended up toasting something obscure and had to reinstall the box. I had two DNAT rules set up that were virtually identical, one of which worked (port 25 to an exchange box), the other didn''t (1494 to citrix server). Despite my best efforts at debugging, I couldn''t figure out what happened and even copying in the old known-good configs didn''t repair it. Any idea what could have caused this? I had been messing around with "ip addr" to add extra IPs to external interfaces for the forwarding. Anyway, I''m going to rework this from the ground up. -Derek -----Original Message----- From: shorewall-users-admin@lists.sourceforge.net [mailto:shorewall-users-admin@lists.sourceforge.net] On Behalf Of Jerry Vonau Sent: Tuesday, July 26, 2005 7:33 PM To: shorewall-users@lists.sourceforge.net Subject: Re: [Shorewall-users] RE: Problems with providers I forgot one file, I am running some NAT: #EXTERNAL INTERFACE INTERNAL ALL INTERFACES LOCAL 63.211.45.142 eth3 192.168.1.245 No No 63.211.45.133 eth3 192.168.1.246 No No -----Original Message----- From: Derek Murawsky Sent: Tuesday, July 26, 2005 7:13 PM To: ''Mailing List for Shorewall Users'' Subject: Problems with providers Hello all, I''m having a problem using multiple providers and shorewall on an ubuntu linux box. (Debian clone) My interfaces file does not have any routes assigned in it I read the shorewall and routing docs and set up my providers file as: #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS cogent 2 2 main eth1 66.28.28.79 track,balance=1 birns 3 3 main eth3 63.211.45.129 track,balance=2 MASQ file is set up: #INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC eth3 eth5 63.211.45.130 eth1 eth5 66.28.28.78 I don''t think rules are important here, and I haven''t set up anything else fancy except openvpn Tunnels file: # TYPE ZONE GATEWAY GATEWAY # ZONE openvpn net 4.79.128.50 <snip> Any ideas on what''s going on? Thanks in advance for any help. -Derek I know shorewall is not started, but can I get a shorewall status, just to see the routing and interface info. A discription of what each nic is plugged into can be helpful also. Jerry ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
Derek:>After going through the debug output I had a brainstorm (more >like a trickle) and got the dual providers to work. It had something to >do with how I had assigned the extra external IP using ip addr in the >interfaces (Debian) file.Can you post the commands you were using?>However after getting that working, the DNATs >that I had set up no longer functioned. Well, I ended up toasting >something obscure and had to reinstall the box. >I had two DNAT rules set up that were virtually identical, one >of which worked (port 25 to an exchange box), the other didn''t (1494 to >citrix server). Despite my best efforts at debugging, I couldn''t figure >out what happened and even copying in the old known-good configs didn''t >repair it. Any idea what could have caused this? I had been messing >around with "ip addr" to add extra IPs to external interfaces for the >forwarding.From your earlier trace, looks like you''re adding network routes for the aliases also, there should be only one network route and it should be with the primary ip address of that interface.>Anyway, I''m going to rework this from the ground up. >-DerekWhy not just let shorewall create the aliases for you? Jerry ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
inline> -----Original Message----- > From: shorewall-users-admin@lists.sourceforge.net[mailto:shorewall-users-> admin@lists.sourceforge.net] On Behalf Of Jerry Vonau > Sent: Wednesday, July 27, 2005 1:07 PM > To: shorewall-users@lists.sourceforge.net > Subject: Re: [Shorewall-users] RE: Problems with providers > > > Derek: > > >After going through the debug output I had a brainstorm (more > >like a trickle) and got the dual providers to work. It had somethingto> >do with how I had assigned the extra external IP using ip addr in the > >interfaces (Debian) file. > > Can you post the commands you were using?Unfortunately no, I was in a bit of a caffeine induced frenzy trying to get the system up again. All I recall was that I had a stanza in /etc/network/interfaces to set up an interface with 63.211.45.132 and in the up section I had "ip addr 63.211.45.134/28 brd [broadcast] dev eth2 in the Debian interfaces file. That worked fine initially, however, I had to remove that to get dual providers to work. After trying it and noticing that it failed, I restored the old configs and it stopped working. I did flush the routing table and restarted a few times as well.> > >However after getting that working, the DNATs > >that I had set up no longer functioned. Well, I ended up toasting > >something obscure and had to reinstall the box. > >I had two DNAT rules set up that were virtually identical, one > >of which worked (port 25 to an exchange box), the other didn''t (1494to> >citrix server). Despite my best efforts at debugging, I couldn''tfigure> >out what happened and even copying in the old known-good configsdidn''t> >repair it. Any idea what could have caused this? I had been messing > >around with "ip addr" to add extra IPs to external interfaces for the > >forwarding. > > From your earlier trace, looks like you''re adding network routes forthe> aliases > also, there should be only one network route and it should be with the > primary ip > address of that interface.I didn''t think I had added any extra routes, just an ip addr.> >Anyway, I''m going to rework this from the ground up. > >-Derek > > Why not just let shorewall create the aliases for you?I was under the impression from http://www.shorewall.net/Shorewall_and_Aliased_Interfaces.html#id2452731 That I needed to do this through either shorewall''s start script or through my own startup scripts. How do you recommend I proceed?> Jerry > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-usersThanks again, -Derek ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
>> Derek:<snip>>> Can you post the commands you were using? >Unfortunately no, I was in a bit of a caffeine induced frenzy trying to >get the system up again. All I recall was that I had a stanza in >/etc/network/interfaces to set up an interface with 63.211.45.132 and in >the up section I had "ip addr 63.211.45.134/28 brd [broadcast] dev eth2 >in the Debian interfaces file. That worked fine initially, however, I >had to remove that to get dual providers to work. After trying it and >noticing that it failed, I restored the old configs and it stopped >working. I did flush the routing table and restarted a few times as >well.I believe that should be just 63.211.45.134 dev eth2 <snip>>> From your earlier trace, looks like you''re adding network routes for >the >> aliases >> also, there should be only one network route and it should be with the >> primary ip >> address of that interface. >I didn''t think I had added any extra routes, just an ip addr.You don''t, but when you did the "ip addr 63.211.45.134/28 brd [broadcast] dev eth2", the /28 added/changed the route, and it most likely changed the src in the routing table to .134 and not .132 which is the primary ip address. Need to see the routing tables to tell.>> >Anyway, I''m going to rework this from the ground up. >> >-Derek >> >> Why not just let shorewall create the aliases for you? >I was under the impression from >http://www.shorewall.net/Shorewall_and_Aliased_Interfaces.html#id2452731 >That I needed to do this through either shorewall''s start script or >through my own startup scripts. How do you recommend I proceed?Either way works fine, what alaises do you want/need and what are the used in shorewall? I need to see the routing table, just to check, that is why I asked for the shorewall status before. Jerry ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> >> Derek: > <snip> > >> Can you post the commands you were using? > >Unfortunately no, I was in a bit of a caffeine induced frenzy trying to > >get the system up again. All I recall was that I had a stanza in > >/etc/network/interfaces to set up an interface with 63.211.45.132 and in > >the up section I had "ip addr 63.211.45.134/28 brd [broadcast] dev eth2 > >in the Debian interfaces file. That worked fine initially, however, I > >had to remove that to get dual providers to work. After trying it and > >noticing that it failed, I restored the old configs and it stopped > >working. I did flush the routing table and restarted a few times as > >well. > > I believe that should be just 63.211.45.134 dev eth2my error, it should work as you had it...> <snip> > >> From your earlier trace, looks like you''re adding network routes for > >the > >> aliases > >> also, there should be only one network route and it should be with the > >> primary ip > >> address of that interface. > >I didn''t think I had added any extra routes, just an ip addr. > > You don''t, but when you did the "ip addr 63.211.45.134/28 > brd [broadcast] dev eth2", the /28 added/changed the route, > and it most likely changed the src in the routing table to .134 > and not .132 which is the primary ip address. Need to see the > routing tables to tell. >Ok it''s getting clearer, eth1 and eth3 are your providers, eth5 is the local zone. what are eth2 and eth4, dmz interfaces? Your error is: ++ ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope link src 63.211.45.130 ++ ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope link src 63.211.45.132 Your adding 2 nework routes to the same provider''s table, one from eth2(provider) and one from the eth3(dmz?). The way to do this is not have a network route on the providers interface at all, to begin with, use a /32 instead of /28 on the providers interface only, and add a host route on the provider''s interface to the gateway. Where does eth4 fit into all this? Jerry ------------------------------------------------------- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> -----Original Message----- > From: shorewall-users-admin@lists.sourceforge.net[mailto:shorewall-users-> admin@lists.sourceforge.net] On Behalf Of Jerry Vonau > Sent: Wednesday, July 27, 2005 2:52 PM > To: Jerry Vonau; shorewall-users@lists.sourceforge.net > Subject: Re: [Shorewall-users] RE: Problems with providers > > > > >> Derek: > > <snip> > > >> Can you post the commands you were using? > > >Unfortunately no, I was in a bit of a caffeine induced frenzytrying to> > >get the system up again. All I recall was that I had a stanza in > > >/etc/network/interfaces to set up an interface with 63.211.45.132and> in > > >the up section I had "ip addr 63.211.45.134/28 brd [broadcast] deveth2> > >in the Debian interfaces file. That worked fine initially,however, I> > >had to remove that to get dual providers to work. After trying itand> > >noticing that it failed, I restored the old configs and it stopped > > >working. I did flush the routing table and restarted a few timesas> > >well. > > > > I believe that should be just 63.211.45.134 dev eth2 > > my error, it should work as you had it... > > > <snip> > > >> From your earlier trace, looks like you''re adding network routesfor> > >the > > >> aliases > > >> also, there should be only one network route and it should bewith> the > > >> primary ip > > >> address of that interface. > > >I didn''t think I had added any extra routes, just an ip addr. > > > > You don''t, but when you did the "ip addr 63.211.45.134/28 > > brd [broadcast] dev eth2", the /28 added/changed the route, > > and it most likely changed the src in the routing table to .134 > > and not .132 which is the primary ip address. Need to see the > > routing tables to tell. > > > Ok it''s getting clearer, eth1 and eth3 are your providers, eth5 is the > local zone. what are eth2 and eth4, dmz interfaces?Eth2 is just another interface that has some additional IPs from one of our providers. This is where the DNAT entries forward from. I suppose the logic behind it was not to bog down one interface with too much physical traffic. I take it that these should be on the regular provider interface to avoid problems down the line?> > Your error is: > ++ ip route add table 2 63.211.45.128/28 dev eth3 proto kernel scope > link src 63.211.45.130 > > ++ ip route add table 2 63.211.45.128/28 dev eth2 proto kernel scope > link src 63.211.45.132 > > Your adding 2 nework routes to the same provider''s table, one from > eth2(provider) and one from the eth3(dmz?). > > The way to do this is not have a network route on the providersinterface> at all, > to begin with, use a /32 instead of /28 on the providers interfaceonly,> and add > a host route on the provider''s interface to the gateway. >Ok, I''ll check this out.> Where does eth4 fit into all this? >Eth4 is a second interface that is the DMZ interface for a class C that we have through one of our providers. We''re in the process of getting an ASN number to route between the providers, but that comes later.> Jerry > > > ------------------------------------------------------- > SF.Net email is Sponsored by the Better Software Conference & EXPO > September > 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices > Agile & Plan-Driven Development * Managing Projects & Teams * Testing& QA> Security * Process Improvement & Measurement *http://www.sqe.com/bsce5sf> _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-usersAgain, thanks -Derek ------------------------------------------------------- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
Derek: Ok, I''m tried of trying to extract the info required to debug this, from you. <snip>>> Ok it''s getting clearer, eth1 and eth3 are your providers, eth5 is the >> local zone. what are eth2 and eth4, dmz interfaces? >Eth2 is just another interface that has some additional IPs from one of >our providers.This is a useless statement, which ips from what provider?>This is where the DNAT entries forward from. I suppose >the logic behind it was not to bog down one interface with too much >physical traffic. I take it that these should be on the regular >provider interface to avoid problems down the line?Not really, but you need to have the routes setup correctly. <snip>>> and add >> a host route on the provider''s interface to the gateway. >> >Ok, I''ll check this out.>> Where does eth4 fit into all this? >> >Eth4 is a second interface that is the DMZ interface for a class C that >we have through one of our providers. We''re in the process of getting >an ASN number to route between the providers, but that comes later.Again, useless info, which provider? What address range? Unless I see a shorewall status, I''m done with this. No, an "ip route" is not enough info, you have to capture the providers tables also, which the status does. Jerry ------------------------------------------------------- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf