Lito Kusnadi
2012-May-01 04:58 UTC
packet fragmentation between LAN and DMZ for multi ISP firewall
I managed to get multi ISPs firewall running with 4 zones: - net (internet) - loc - dmz - road (openvpn rwarrior) Browsing from loc to internet, vpn to dmz, vpn to loc are working fine with reasonable response. However, loc to dmz traffic is having issue. I found when browsing from a workstation in the LAN (loc) zone to a web server hosted in DMZ zone takes a very long time. I did packet analyzer using wireshark. I found that the data traffic from the web server is being fragmented and eventually experience out-of-order fragmented packets. This results in slow speed and the data from the web server eventually not delivered completely to the browser (e.g. only partial data). wireshark reports "TCP segment of a reassembled PDU" for many times until the "out-of-order segment" trace. Google around about fragmentation, the first thing to look at is MTU. However, when I move a LAN workstation to dmz zone, by just changing the IP address to suit the dmz subnet (not playing with the MTU setup), I have no problem browsing the web server. So that eliminates MTU as the cause. The firewall, workstation, and web server runs on VMWare. So no physical switch involved (yet). I doubt it is the virtual switch causing the problem because when workstation and web server are connected to the same virtual switch, it''s working fine. When I do "shorewall dump", I can only tell that the traffic between loc and dmz, in fact, was established. tcp 6 431994 ESTABLISHED src=10.35.249.53 dst=192.168.168.11 sport=1295 dport=80 packets=3 bytes=132 src=192.168.168.11 dst =10.35.249.53 sport=80 dport=1295 packets=1 bytes=52 [ASSURED] mark=0 secmark=0 use=1 where 10.35.249.53 = LAN workstation, 192.168.168.11 = web server dmz. I did however suspect the marking in tcrules might play along with this problem (however, not being an expert in traffic shapping, I can comment further). As displayed from the dump (above) the mark is "0" for this packet which shall be default and processed normally in shorewall without packet mark redirection. When i comment out all the tcrules, this problem still persists, so is it because of the providers entries? Anyone who has seen this issue or have some hints would be appreciated. Thank you. /etc/shorewall/interfaces: #ZONE INTERFACE OPTIONS net eth0 detect net eth1 detect dmz eth2 detect loc eth3 detect road tun+ /etc/shorewall/zones: fw firewall net ipv4 loc ipv4 dmz ipv4 road ipv4 /etc/shorewall/policy: net net DROP info # traffic to internet loc net ACCEPT dmz net REJECT info # traffic to dmz loc dmz ACCEPT # traffic from roadwarrior road loc ACCEPT road dmz ACCEPT # traffic to firewall loc $FW REJECT info # traffic from firewall $FW all ACCEPT # traffic from internet net all DROP info # catch all all all DROP info /etc/shorewall/providers: ISPIIN 1 1 main eth1 218.206.228.101 track,balance=100 eth2,eth3,tun0 ISPAC3 2 2 main eth0 218.202.139.222 track,balance=50 eth2,eth3 /etc/shorewall/tcrules: # redirect service from LAN to (1) iinet or (2) ac3 1:P 10.35.249.0/24 0.0.0.0/0 tcp 80 2:P 10.35.249.0/24 0.0.0.0/0 tcp 3389 # redirect service from DMZ to (1) iinet or (2) ac3 1:P 192.168.168.0/24 0.0.0.0/0 tcp 80 1:P 192.168.168.0/24 0.0.0.0/0 tcp 443 1:P 192.168.168.0/24 0.0.0.0/0 tcp 53 1:P 192.168.168.0/24 0.0.0.0/0 udp 53 1:P 192.168.168.0/24 0.0.0.0/0 udp 123 1:P 192.168.168.0/24 0.0.0.0/0 icmp 8 1:P 192.168.168.0/24 0.0.0.0/0 tcp 21 #2:P 192.168.168.0/24 0.0.0.0/0 icmp 8 /etc/shorewall/rtrules: #here not using USE_DEFAULT_RT - 172.16.1.0/24 main 1000 /etc/shorewall/tunnels: openvpnserver:tcp:443 net 0.0.0.0/0 # Internet to ALL -- drop NewNotSyn packets dropNotSyn net fw tcp dropNotSyn net loc tcp dropNotSyn net dmz tcp #---FROM LAN--- SSH(ACCEPT) loc $FW - - - - s:1/min:3 #---FROM DMZ--- HTTP(ACCEPT) dmz net HTTPS(ACCEPT) dmz net DNS(ACCEPT) dmz net NTP(ACCEPT) dmz net Ping(ACCEPT) dmz net FTP(ACCEPT) dmz net #---DNAT Rules--- DNAT net:eth1 dmz:192.168.168.10:80 tcp 8080 - 218.206.228.102 DNAT net:eth1 loc:10.35.249.53:80 tcp 8081 - 218.206.228.102 #---Services within Firewall--- # openvpn - tcp 443 ACCEPT net $FW tcp 443 - 218.202.228.102 #---Temporary Rules--- ACCEPT net:124.149.32.217 $FW tcp 22 - 218.206.228.102 #Block stealth Auth port 113 (must be at the end) Auth(DROP) net $FW /etc/shorewall/shorewall.conf (diff from the original setting): < STARTUP_ENABLED=Yes < LOGLIMIT=s:5/sec < STARTUP_LOG=/var/log/firewall/shorewall-init.log < MARK_IN_FORWARD_CHAIN=Yes /etc/shorewall/masq: eth0 192.168.168.15 218.202.139.210 tcp 80 eth1 192.168.168.10 218.206.228.102 tcp 8080 eth1 10.35.249.53 218.206.228.102 tcp 8081 eth0 0.0.0.0/0 218.202.139.210 eth1 0.0.0.0/0 218.206.228.102 ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Tom Eastep
2012-May-01 13:45 UTC
Re: packet fragmentation between LAN and DMZ for multi ISP firewall
On 04/30/2012 09:58 PM, Lito Kusnadi wrote:> I managed to get multi ISPs firewall running with 4 zones: - net > (internet) - loc - dmz - road (openvpn rwarrior) > > Browsing from loc to internet, vpn to dmz, vpn to loc are working > fine with reasonable response. However, loc to dmz traffic is having > issue. > > I found when browsing from a workstation in the LAN (loc) zone to a > web server hosted in DMZ zone takes a very long time. > > I did packet analyzer using wireshark. I found that the data traffic > from the web server is being fragmented and eventually experience > out-of-order fragmented packets. This results in slow speed and the > data from the web server eventually not delivered completely to the > browser (e.g. only partial data). > > wireshark reports "TCP segment of a reassembled PDU" for many times > until the "out-of-order segment" trace. > > Google around about fragmentation, the first thing to look at is MTU. > However, when I move a LAN workstation to dmz zone, by just changing > the IP address to suit the dmz subnet (not playing with the MTU > setup), I have no problem browsing the web server. So that eliminates > MTU as the cause.Well, there is certainly nothing in a Shorewall configuration that can *cause* fragmentation. What are the MTU settings on the web server and on the LAN workstation? If either is > 1500, you can set CLAMPMSS=Yes in shorewall.conf and see if that helps. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Lito Kusnadi
2012-May-04 01:46 UTC
Re: packet fragmentation between LAN and DMZ for multi ISP firewall
Hi Tom, Thank you. It turns out to be centos 5.8 build that causes the segment to be out of order. I switched to centos 6.2 and the problem is gone. One question about /etc/shorewall/providers: If I want to design in a way that: a. THE DEFAULT traffic will go out using provider1 (no rule need to be applied to tcrule or rtrule b. Use provider2 ONLY when i define rules in tcrule c. I want to direct the traffic in (b) based on outgoing ports d. Using the minimal amount of rules :) Is there a way i can achieve that? I can only find such note in the doc: "... If you are using /etc/shorewall/providers because you have multiple Internet connections, we recommend that you specify balance even if you don''t need it. You can still use entries in /etc/shorewall/tcrules and /etc/shorewall/rtrules to force all traffic to one provider or another... " Without specifying specific rule, the default behavior is load balance hence 100:50 proportion for both providers. I guess I can make the proportion to be 100:1, but it still bleeds 1% of the traffic to the other provider. Many thanks. --- On Tue, 1/5/12, Tom Eastep <teastep@shorewall.net> wrote:> From: Tom Eastep <teastep@shorewall.net> > Subject: Re: [Shorewall-users] packet fragmentation between LAN and DMZ for multi ISP firewall > To: shorewall-users@lists.sourceforge.net > Received: Tuesday, 1 May, 2012, 2:45 PM > On 04/30/2012 09:58 PM, Lito Kusnadi > wrote: > > I managed to get multi ISPs firewall running with 4 > zones: - net > > (internet) - loc - dmz - road (openvpn rwarrior) > > > > Browsing from loc to internet, vpn to dmz, vpn to loc > are working > > fine with reasonable response. However, loc to dmz > traffic is having > > issue. > > > > I found when browsing from a workstation in the LAN > (loc) zone to a > > web server hosted in DMZ zone takes a very long time. > > > > I did packet analyzer using wireshark. I found that the > data traffic > > from the web server is being fragmented and eventually > experience > > out-of-order fragmented packets. This results in slow > speed and the > > data from the web server eventually not delivered > completely to the > > browser (e.g. only partial data). > > > > wireshark reports "TCP segment of a reassembled PDU" > for many times > > until the "out-of-order segment" trace. > > > > Google around about fragmentation, the first thing to > look at is MTU. > > However, when I move a LAN workstation to dmz zone, by > just changing > > the IP address to suit the dmz subnet (not playing with > the MTU > > setup), I have no problem browsing the web server. So > that eliminates > > MTU as the cause. > > Well, there is certainly nothing in a Shorewall > configuration that can > *cause* fragmentation. > > What are the MTU settings on the web server and on the LAN > workstation? > If either is > 1500, you can set CLAMPMSS=Yes in > shorewall.conf and see > if that helps. > > -Tom > -- > Tom Eastep \ When I die, I want > to go like my Grandfather who > Shoreline, \ died > peacefully in his sleep. Not screaming like > Washington, USA \ all of the > passengers in his car > http://shorewall.net > \________________________________________________ > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today''s > security and > threat landscape has changed and how IT managers can > respond. Discussions > will include endpoint security, mobile security and the > latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users >------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Tom Eastep
2012-May-04 13:36 UTC
Re: packet fragmentation between LAN and DMZ for multi ISP firewall
On 05/03/2012 06:46 PM, Lito Kusnadi wrote:> Hi Tom, > > Thank you. It turns out to be centos 5.8 build that causes the > segment to be out of order. I switched to centos 6.2 and the problem > is gone. > > One question about /etc/shorewall/providers: If I want to design in a > way that: a. THE DEFAULT traffic will go out using provider1 (no rule > need to be applied to tcrule or rtrule b. Use provider2 ONLY when i > define rules in tcrule c. I want to direct the traffic in (b) based > on outgoing ports d. Using the minimal amount of rules :) > > Is there a way i can achieve that? I can only find such note in the > doc: "... If you are using /etc/shorewall/providers because you have > multiple Internet connections, we recommend that you specify balance > even if you don''t need it. You can still use entries in > /etc/shorewall/tcrules and /etc/shorewall/rtrules to force all > traffic to one provider or another..." > > Without specifying specific rule, the default behavior is load > balance hence 100:50 proportion for both providers. > > I guess I can make the proportion to be 100:1, but it still bleeds 1% > of the traffic to the other provider. >You can specify ''balance'' for one and ''fallback'' for the other, which is what the "Complete Working Example" in the Multi-ISP doc does. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/