Asim Ahmed Khan
2009-Dec-15 21:26 UTC
Shorewall 4.4.4.-1 with squid 3.0Stable20-1 as non-transparnt proxy
hi, First i tried to run squid as transparent (interception) proxy that didn''t work. Browsing and other internet usage became too inconsisten. too many break ups were occuring and all of a sudden browsing stop and restart after some time ranging from a 30 seconds to a few minutes. hitting F5 keys numerous times opens up the page. I used this rule from http://www.shorewall.net/Shorewall_Squid_Usage.html#Firewall to redirect traffic to squid on port 3128 #ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE ORIGINAL # PORT(S) DEST ACCEPT $FW net tcp www REDIRECT loc 3128 tcp www - - Now I am running as non-transparent mode. Browsing is working fine but there are a few major problems i m facing: 1. All users have to enter proxy settings in default browsers. Now some applications don''t have proxy setting and some don''t work with proxy servers. These applications are having great difficulty with this new proxy setting hence users getting frustrated. 2. Ideally squid should only interfere with port 80 traffic and rest of the traffic should be handled by shorewall as before but it seems like this is not happening. I am using these rules as mentioned in following link http://www.shorewall.net/Shorewall_Squid_Usage.html#Firewall with non-transparent proxy in my rules file: Squid as a Manual Proxy /etc/shorewall/rules: #ACTION SOURCE DEST PROTO DEST PORT(S) ACCEPT loc $FW tcp 3128 ACCEPT $FW net tcp 80 Now I have two questions, if any one can answer, it might help me: Q-1 -> Does placement of both rules above (transparent / non-transparent) in rules file is significant? I am placing these rules on first line in rules file rite now in both cases. Q-2 -> Do i need to modify any other shorewall file if I install squid on same machine (firewall) as the shorewall? Q-3 -> What do I need to do to let https traffic go through proxy as well? If I modify rule in 2nd line as 80,443 and chck squid access.log, TCP_DENIED shows up although SSL_Ports & Safe_Ports are both allowed access explicitly in squid. Q-4: If I have a link to access as (applogy for being so kinky, but i m exhausted by config fixes b/w shorewall & squid) as https://64.50.169.94:20098 Where should this traffic go, to shorewall or squid (incase 2nd line reads as 80,443) http://w.x.y.z:8080 where shud this traffic go provided that squid is listening for port 80 traffic (http). Does port 8080 in URL change its traffic type from http(port 80)? Q-5 -> Do i need to setup some thing in squid to let people use a code repository running on a remote server of URL like http://w.x.y.z:8080/ requiring users to authenticate to access code? I see requests going through but returned with TCP_MISS/401 (Unauthorized) and user get an error message on application interface as "you are not authorized to access this server" users give correct username/pwd on the box that appears for authentication. -- Regards, Asim Ahmed Khan ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon''s best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev
Tom Eastep
2009-Dec-15 22:02 UTC
Re: Shorewall 4.4.4.-1 with squid 3.0Stable20-1 as non-transparnt proxy
Asim Ahmed Khan wrote:> hi, > > First i tried to run squid as transparent (interception) proxy that > didn''t work. Browsing and other internet usage became too > inconsisten. too many break ups were occuring and all of a sudden > browsing stop and restart after some time ranging from a 30 seconds > to a few minutes. hitting F5 keys numerous times opens up the page. I > used this rule from > > http://www.shorewall.net/Shorewall_Squid_Usage.html#Firewall to > redirect traffic to squid on port 3128 > > #ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE > ORIGINAL # PORT(S) DEST ACCEPT $FW net tcp www > REDIRECT loc 3128 tcp www - - > > Now I am running as non-transparent mode. Browsing is working fine > but there are a few major problems i m facing: > > 1. All users have to enter proxy settings in default browsers. Now > some applications don''t have proxy setting and some don''t work with > proxy servers. These applications are having great difficulty with > this new proxy setting hence users getting frustrated. > > 2. Ideally squid should only interfere with port 80 traffic and rest > of the traffic should be handled by shorewall as before but it seems > like this is not happening.Nonsense. But there *are* sites that simply don''t work with transparent proxying; the Sun VirtualBox registration site is one that I''ve run into.> > I am using these rules as mentioned in following link > http://www.shorewall.net/Shorewall_Squid_Usage.html#Firewall with > non-transparent proxy in my rules file: > > Squid as a Manual Proxy > > > |/etc/shorewall/rules:| > > #ACTION SOURCE DEST PROTO DEST PORT(S) > ACCEPT loc $FW tcp 3128 > ACCEPT $FW net tcp 80 > > Now I have two questions, if any one can answer, it might help me:I count five questions... :-)> > Q-1 -> Does placement of both rules above (transparent / > non-transparent) in rules file is significant? I am placing these > rules on first line in rules file rite now in both cases.Entries in the rules file are based on first-match. So the first terminating rule (and both ACCEPT and REDIRECT are terminating) determines the disposition of the connection.> Q-2 -> Do i need to modify any other shorewall file if I install > squid on same machine (firewall) as the shorewall?This is covered in the Shorewall Squid documentation; if there were more files to modify, we would mention them in the documentation.> Q-3 -> What do I need to do to let https traffic go through proxy as > well? If I modify rule in 2nd line as 80,443 and chck squid > access.log, TCP_DENIED shows up although SSL_Ports & Safe_Ports are > both allowed access explicitly in squid.As detailed in the Shorewall Squid documentation (and many other places on the web), you *cannot* transparently proxy HTTPS.> Q-4: If I have a link to access as (applogy for being so kinky, but i > m exhausted by config fixes b/w shorewall & squid) as > > https://64.50.169.94:20098 Where should this traffic go, to shorewall > or squid (incase 2nd line reads as 80,443)All traffic goes through the Shorewall-configured firewall rules. It depends on whether you have configured an HTTPS manual proxy whether squid will handle the request or if it is simply routed to 64.50.169.94.> http://w.x.y.z:8080 where should this traffic go > provided that squid is listening for port 80 traffic (http). Does > port 8080 in URL change its traffic type from http(port 80)?No -- it is still HTTP. But it changes the port that is opened. So your REDIRECT rule for port 80 will not redirect traffic to 8080.> > > Q-5 -> Do i need to setup some thing in squid to let people use a > code repository running on a remote server of URL like > http://w.x.y.z:8080/ requiring users to authenticate to access code?I have no idea how to allow access to port 8080 through Squid. You will have to ask the Squid folks about that.> I see requests going through but returned with TCP_MISS/401 > (Unauthorized) and user get an error message on application interface > as "you are not authorized to access this server" users give correct > username/pwd on the box that appears for authentication.One more word of advise -- when you are testing, be sure to check the configuration of your browser; double check it! It is easy to forget that and you will end up drawing completely wrong conclusions about your tests if you think that you don''t have a manual proxy configured but you do and vice versa. Also note that you can set up both squid and Shorewall to act on traffic from a single test computer. So you can do your testing without annoying your users any more than you already have. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon''s best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev
Asim Ahmed Khan
2009-Dec-16 14:07 UTC
Re: Shorewall 4.4.4.-1 with squid 3.0Stable20-1 as non-transparnt proxy
thanks tom for your help. But i would like to mention the fact that i tried these rules on a single test computer first. There they worked fine or you can say i couldn''t test as much as 100 users with all sorts of traffic needs can test! All problems started except a few after opening it for general users. In transparent proxy i had too many issues of net access braeking too often. But on non-transparent atleast for general users internet is working fine. I''ll try to setup a test computer again and see if i can diagnose problem with transparent mode. thanks, -Asim Ahmed On Wed, Dec 16, 2009 at 3:02 AM, Tom Eastep <teastep@shorewall.net> wrote:> Asim Ahmed Khan wrote: > > hi, > > > > First i tried to run squid as transparent (interception) proxy that > > didn''t work. Browsing and other internet usage became too > > inconsisten. too many break ups were occuring and all of a sudden > > browsing stop and restart after some time ranging from a 30 seconds > > to a few minutes. hitting F5 keys numerous times opens up the page. I > > used this rule from > > > > http://www.shorewall.net/Shorewall_Squid_Usage.html#Firewall to > > redirect traffic to squid on port 3128 > > > > #ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE > > ORIGINAL # PORT(S) DEST ACCEPT $FW net tcp www > > REDIRECT loc 3128 tcp www - - > > > > Now I am running as non-transparent mode. Browsing is working fine > > but there are a few major problems i m facing: > > > > 1. All users have to enter proxy settings in default browsers. Now > > some applications don''t have proxy setting and some don''t work with > > proxy servers. These applications are having great difficulty with > > this new proxy setting hence users getting frustrated. > > > > 2. Ideally squid should only interfere with port 80 traffic and rest > > of the traffic should be handled by shorewall as before but it seems > > like this is not happening. > > Nonsense. But there *are* sites that simply don''t work with transparent > proxying; the Sun VirtualBox registration site is one that I''ve run into. > > > > > I am using these rules as mentioned in following link > > http://www.shorewall.net/Shorewall_Squid_Usage.html#Firewall with > > non-transparent proxy in my rules file: > > > > Squid as a Manual Proxy > > > > > > |/etc/shorewall/rules:| > > > > #ACTION SOURCE DEST PROTO DEST PORT(S) > > ACCEPT loc $FW tcp 3128 > > ACCEPT $FW net tcp 80 > > > > Now I have two questions, if any one can answer, it might help me: > > I count five questions... :-) > > > > > Q-1 -> Does placement of both rules above (transparent / > > non-transparent) in rules file is significant? I am placing these > > rules on first line in rules file rite now in both cases. > > Entries in the rules file are based on first-match. So the first > terminating rule (and both ACCEPT and REDIRECT are terminating) > determines the disposition of the connection. > > > Q-2 -> Do i need to modify any other shorewall file if I install > > squid on same machine (firewall) as the shorewall? > > This is covered in the Shorewall Squid documentation; if there > were more files to modify, we would mention them in the documentation. > > > Q-3 -> What do I need to do to let https traffic go through proxy as > > well? If I modify rule in 2nd line as 80,443 and chck squid > > access.log, TCP_DENIED shows up although SSL_Ports & Safe_Ports are > > both allowed access explicitly in squid. > > As detailed in the Shorewall Squid documentation (and many other places > on the web), you *cannot* transparently proxy HTTPS. > > > Q-4: If I have a link to access as (applogy for being so kinky, but i > > m exhausted by config fixes b/w shorewall & squid) as > > > > https://64.50.169.94:20098 Where should this traffic go, to shorewall > > or squid (incase 2nd line reads as 80,443) > > All traffic goes through the Shorewall-configured firewall rules. It > depends on whether you have configured an HTTPS manual proxy whether > squid will handle the request or if it is simply routed to 64.50.169.94. > > > http://w.x.y.z:8080 where should this traffic go > > provided that squid is listening for port 80 traffic (http). Does > > port 8080 in URL change its traffic type from http(port 80)? > > No -- it is still HTTP. But it changes the port that is opened. So your > REDIRECT rule for port 80 will not redirect traffic to 8080. > > > > > > Q-5 -> Do i need to setup some thing in squid to let people use a > > code repository running on a remote server of URL like > > http://w.x.y.z:8080/ requiring users to authenticate to access code? > > I have no idea how to allow access to port 8080 through Squid. You will > have to ask the Squid folks about that. > > > I see requests going through but returned with TCP_MISS/401 > > (Unauthorized) and user get an error message on application interface > > as "you are not authorized to access this server" users give correct > > username/pwd on the box that appears for authentication. > > One more word of advise -- when you are testing, be sure to check the > configuration of your browser; double check it! It is easy to forget > that and you will end up drawing completely wrong conclusions about your > tests if you think that you don''t have a manual proxy configured but you > do and vice versa. > > Also note that you can set up both squid and Shorewall to act on traffic > from a single test computer. So you can do your testing without annoying > your users any more than you already have. > > -Tom > -- > Tom Eastep \ When I die, I want to go like my Grandfather who > Shoreline, \ died peacefully in his sleep. Not screaming like > Washington, USA \ all of the passengers in his car > http://shorewall.net \________________________________________________ > > > > ------------------------------------------------------------------------------ > This SF.Net email is sponsored by the Verizon Developer Community > Take advantage of Verizon''s best-in-class app development support > A streamlined, 14 day to market process makes app distribution fast and > easy > Join now and get one step closer to millions of Verizon customers > http://p.sf.net/sfu/verizon-dev2dev > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > >-- Regards, Asim Ahmed Khan ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon''s best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev
Tom Eastep
2009-Dec-16 16:10 UTC
Re: Shorewall 4.4.4.-1 with squid 3.0Stable20-1 as non-transparnt proxy
Asim Ahmed Khan wrote:> thanks tom for your help. But i would like to mention the fact that i > tried these rules on a single test computer first. There they worked > fine or you can say i couldn''t test as much as 100 users with all sorts > of traffic needs can test! All problems started except a few after > opening it for general users. In transparent proxy i had too many issues > of net access braeking too often. But on non-transparent atleast for > general users internet is working fine.That''s interesting. From the point of view of system resources, transparent and non-transparent are the same. Each connection which fetches a non-cached page requires a second connection from the proxy (squid) to the net. So if you were running out of conntrack entries (for example) with transparent proxying, you should also run out with manual proxying. In the absence of any limiting rules or traffic shaping (as in your case), the Shorewall-configured firewall does exactly the same thing for each connection of a given type. So issues that arise when volume is increased are extremely unlikely to be associated with the firewall configuration. I can''t speak to any possible volume-related issues with squid because the volume on my own site is so light.> > I''ll try to setup a test computer again and see if i can diagnose > problem with transparent mode.-Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon''s best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev
Asim Ahmed Khan
2009-Dec-17 10:58 UTC
Re: Shorewall 4.4.4.-1 with squid 3.0Stable20-1 as non-transparnt proxy
yes i believe that it is true what you are saying. I really suspect the glitch is somewhere around the rule that redirects traffic to squid in transparent mode. But where, this is beyond my understanding at this point. I have not issues (either professional or personal) :)with non-transparent proxy mode but it is making life more difficult for just a few users who access some resources that require authentication at their end like some code repositories. Besides, running it transparently liberates you from hastle of configuring all different types of devices. We use quite a lot of smart phones / hand helds / pocket pcs at our office so it becomes a pain some times to set proxy on all of them. Introducing proxy has proved to be so beneficial in terms of speed & bandwidth savings that i don''t want to give it up just for the sake of few problems. I''ll keep working until i resolve them. appologies for repeating but just confirm once more that my settings for redirecting traffic were correct. This is the first rule in rules file. After that i have rules for handling other types of traffic. ########################################### # REDIRECTING PORT 80 TRAFFIC TO SQUID ########################################### ACCEPT $FW net tcp 80 REDIRECT loc 4044 tcp 80 ########################################### This is my policy file #SOURCE DEST POLICY LOG LIMIT:BURST # #all all ACCEPT net fw ACCEPT fw net ACCEPT fw loc ACCEPT loc fw ACCEPT loc net REJECT net loc REJECT all all REJECT #LAST LINE -- DO NOT REMOVE Let me know if posting any thing file content will help anybody find any mistake i m making. Regards, -Asim. On Wed, Dec 16, 2009 at 9:10 PM, Tom Eastep <teastep@shorewall.net> wrote:> Asim Ahmed Khan wrote: > > thanks tom for your help. But i would like to mention the fact that i > > tried these rules on a single test computer first. There they worked > > fine or you can say i couldn''t test as much as 100 users with all sorts > > of traffic needs can test! All problems started except a few after > > opening it for general users. In transparent proxy i had too many issues > > of net access braeking too often. But on non-transparent atleast for > > general users internet is working fine. > > That''s interesting. From the point of view of system resources, > transparent and non-transparent are the same. Each connection which > fetches a non-cached page requires a second connection from the proxy > (squid) to the net. So if you were running out of conntrack entries (for > example) with transparent proxying, you should also run out with manual > proxying. In the absence of any limiting rules or traffic shaping (as in > your case), the Shorewall-configured firewall does exactly the same > thing for each connection of a given type. So issues that arise when > volume is increased are extremely unlikely to be associated with the > firewall configuration. > > I can''t speak to any possible volume-related issues with squid because > the volume on my own site is so light. > > > > > I''ll try to setup a test computer again and see if i can diagnose > > problem with transparent mode. > > -Tom > -- > Tom Eastep \ When I die, I want to go like my Grandfather who > Shoreline, \ died peacefully in his sleep. Not screaming like > Washington, USA \ all of the passengers in his car > http://shorewall.net \________________________________________________ > > > > ------------------------------------------------------------------------------ > This SF.Net email is sponsored by the Verizon Developer Community > Take advantage of Verizon''s best-in-class app development support > A streamlined, 14 day to market process makes app distribution fast and > easy > Join now and get one step closer to millions of Verizon customers > http://p.sf.net/sfu/verizon-dev2dev > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > >-- Regards, Asim Ahmed Khan ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon''s best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev
Tom Eastep
2009-Dec-17 14:50 UTC
Re: Shorewall 4.4.4.-1 with squid 3.0Stable20-1 as non-transparnt proxy
Asim Ahmed Khan wrote:> yes i believe that it is true what you are saying. I really suspect the > glitch is somewhere around the rule that redirects traffic to squid in > transparent mode.Then there is nothing more that I can do to help you. Because your configuration is correct. I''ll repeat one more time -- if the rule redirects one request from the loc zone to tcp port 80, then it will redirect all requests. There is nothing that can cause the rule to work on some requests and to then fail for 30 seconds to several minutes.> > appologies for repeating but just confirm once more that my settings for > redirecting traffic were correct. This is the first rule in rules file. > After that i have rules for handling other types of traffic. > > ########################################### > # REDIRECTING PORT 80 TRAFFIC TO SQUID > ########################################### > ACCEPT $FW net tcp 80 > REDIRECT loc 4044 tcp 80Those rules are correct assuming that you have configured Squid to be a transparent proxy listening on port 4044. In squid.conf: http_port 4044 transparent It also assumes that your Squid ACLs are correct.> This is my policy file > #SOURCE DEST POLICY LOG LIMIT:BURST > # > #all all ACCEPT > net fw ACCEPTThat''s a foolish policy. fw net ACCEPT fw loc ACCEPT loc fw ACCEPT loc net REJECT net loc REJECT all all REJECT #LAST LINE -- DO NOT REMOVE> fw net ACCEPTThat policy makes your first ACCEPT rule above unnecessary.> fw loc ACCEPT > loc fw ACCEPT > loc net REJECT > net loc REJECT > all all REJECTI recommend that you specify a LOG level on those REJECT policies. That way, you will KNOW when the Shorewall-configured ruleset rejects a connection request.> #LAST LINE -- DO NOT REMOVEThe only thing that could possibly shed any more light on this would be the output of ''shorewall dump'' collected while users are having issues with transparent proxy. One other suggestion; go back in your kernel logs to time periods when users were experiencing issues. Look for any unusual messages, especially those having to do with ''conntrack''. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon''s best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev
Tom Eastep
2009-Dec-17 17:59 UTC
Re: Shorewall 4.4.4.-1 with squid 3.0Stable20-1 as non-transparnt proxy
On Thu, 17 Dec 2009 06:50:13 -0800 Tom Eastep <teastep@shorewall.net> wrote:> Asim Ahmed Khan wrote: > > yes i believe that it is true what you are saying. I really suspect > > the glitch is somewhere around the rule that redirects traffic to > > squid in transparent mode. > > Then there is nothing more that I can do to help you. Because your > configuration is correct. I''ll repeat one more time -- if the rule > redirects one request from the loc zone to tcp port 80, then it will > redirect all requests. There is nothing that can cause the rule to > work on some requests and to then fail for 30 seconds to several > minutes.I should clarify that there is no way to mis-configure the rule so that this behavior happens.> > One other suggestion; go back in your kernel logs to time periods when > users were experiencing issues. Look for any unusual messages, > especially those having to do with ''conntrack''. >Running out of conntrack table entries *can* cause the symptoms that you describe. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon''s best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev