Hi,
I''m having a strange problem with my SMTP services.
My setup consists of a shorewall gateway with an SMTP
service on it accessible via Internet through eth0.
It''s a multi-ISP setup and I have two other net zones
from which remote hosts can connect via SMTP but to
another mail server behind another internal shorewall
server setup as a bridge.
During months now, I haven''t had issues and we could
receive email messages from anyone and on any one of
our mail servers.
However, I just recently found an exception with one
single remote host (single machine load balanced on:
194.179.55.129 and 194.179.55.135).
This host can connect without issues to $FW''s SMTP
service (net1/eth0) but cannot connect to
10.215.144.7''s SMTPD (net2/net3).
Please have a look at the tcpdumps below.
I tried setting CLAMPMSS to Yes but that did not
change anything, apparently.
I never specified the addresses 194.179.55.129 and
194.179.55.135 in any shorewall config file (eg.
tcrules).
What could I try to do to locate and fix the source of
the problem?
Thank you for your help.
On gateway shorewall:
ACCEPT net1 $FW tcp 25 -
- 10/min:20
DNAT net2 loc:10.215.144.7 tcp 25 -
- 10/min:20
DNAT net3 loc:10.215.144.7 tcp 25
(net1 is on eth0)
# tcpdump -i eth0 host 194.179.55.129 or host
194.179.55.135
10:31:14.174241 IP giss7.seg-social.es.43579 >
192.168.92.2.smtp: S 2854491711:2854491711(0) win
24820 <nop,wscale 0,nop,nop,sackOK,mss 1460>
10:31:14.177544 IP 192.168.92.2.smtp >
giss7.seg-social.es.43579: S 124784606:124784606(0)
ack 2854491712 win 5840 <mss
1460,nop,nop,sackOK,nop,wscale 2>
10:31:14.236810 IP giss7.seg-social.es.43579 >
192.168.92.2.smtp: . ack 1 win 24820
10:31:14.254276 IP 192.168.92.2.smtp >
giss7.seg-social.es.43579: P 1:31(30) ack 1 win 1460
10:31:14.316864 IP giss7.seg-social.es.43579 >
192.168.92.2.smtp: . ack 31 win 24820
(in this case SMTP works fine)
(net3 is on eth3)
# tcpdump -i eth3 host 194.179.55.129 or host
194.179.55.135
10:28:17.930425 IP giss7.seg-social.es.43479 >
192.168.101.2.smtp: S 3220972402:3220972402(0) win
24820 <nop,wscale 0,nop,nop,sackOK,mss 1452>
10:28:17.930781 IP 192.168.101.2.smtp >
giss7.seg-social.es.43479: S 1046102548:1046102548(0)
ack 3220972403 win 32767 <mss
1460,nop,nop,sackOK,nop,wscale 14>
10:28:21.482136 IP 192.168.101.2.smtp >
giss7.seg-social.es.43479: S 1046102548:1046102548(0)
ack 3220972403 win 32767 <mss
1460,nop,nop,sackOK,nop,wscale 14>
10:28:27.482519 IP 192.168.101.2.smtp >
giss7.seg-social.es.43479: S 1046102548:1046102548(0)
ack 3220972403 win 32767 <mss
1460,nop,nop,sackOK,nop,wscale 14>
10:28:39.484056 IP 192.168.101.2.smtp >
giss7.seg-social.es.43479: S 1046102548:1046102548(0)
ack 3220972403 win 32767 <mss
1460,nop,nop,sackOK,nop,wscale 14>
(in this case SMTP fails)
On internal firewall/bridge:
ACCEPT net loc:10.215.144.7 tcp 25
# tcpdump -i br0 host 194.179.55.129 or host
194.179.55.135
11:31:46.182383 IP giss7.seg-social.es.50845 >
10.215.144.7.smtp: S 3631023300:3631023300(0) win
24820 <nop,wscale 0,nop,nop,sackOK,mss 1452>
11:31:46.182550 IP 10.215.144.7.smtp >
giss7.seg-social.es.50845: S 1085960616:1085960616(0)
ack 3631023301 win 32767 <mss
1460,nop,nop,sackOK,nop,wscale 14>
11:31:49.512665 IP 10.215.144.7.smtp >
giss7.seg-social.es.50845: S 1085960616:1085960616(0)
ack 3631023301 win 32767 <mss
1460,nop,nop,sackOK,nop,wscale 14>
11:31:55.711813 IP 10.215.144.7.smtp >
giss7.seg-social.es.50845: S 1085960616:1085960616(0)
ack 3631023301 win 32767 <mss
1460,nop,nop,sackOK,nop,wscale 14>
Please find the shorewall dumps for both systems
(gateway and bridge):
http://213.96.91.201/shorewall/
The SMTP connections are from 194.179.55.129 or
194.179.55.135 to 192.168.92.2 and 192.168.101.2 or
192.168.100.2 (connections to 192.168.92.2 work: SMTPD
is on $FW; the others fail).
____________________________________________________________________________________
Pinpoint customers who are looking for what you sell.
http://searchmarketing.yahoo.com/
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
Vieri Di Paola wrote:> > > Please have a look at the tcpdumps below. >If you want me to look at the output of tcpdump, please use the "-n" option. It''s impossible to be certain what one is looking at when tcpdump is printing the result of reverse DNS lookups rather than the raw IP addresses. Thanks, -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep wrote:> Vieri Di Paola wrote: >> >> Please have a look at the tcpdumps below. >> > > If you want me to look at the output of tcpdump, please use the "-n" option. > It''s impossible to be certain what one is looking at when tcpdump is > printing the result of reverse DNS lookups rather than the raw IP addresses.That having been said, I would next do the tcpdump on eth0 on the bridge (make sure the SYN,ACKs are being sent from the bridge) then on eth1 on the outer firewall. I see this conntrack entry on the bridge: tcp 6 35 SYN_RECV src=194.179.55.129 dst=10.215.144.7 sport=61911 dport=25 packets=1 bytes=52 src=10.215.144.7 dst=194.179.55.129 sport=25 dport=61911 packets=5 bytes=260 mark=0 use=1 This indicates that Netfilter connection tracking on the bridge is not seeing or not recognizing the SYN,ACK response. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
--- Tom Eastep <teastep@shorewall.net> wrote:> > If you want me to look at the output of tcpdump, > please use the "-n" option.tcpdump on eth1 on the gateway (remote tried to connect twice): 13:18:57.029836 IP 194.179.55.135.49114 > 10.215.144.7.25: S 3140736196:3140736196(0) win 24820 <nop,wscale 0,nop,nop,sackOK,mss 1452> 13:18:57.030181 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:19:01.430928 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:19:07.432794 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:19:19.437405 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:19:43.440156 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:19:52.789009 IP 194.179.55.129.39005 > 10.215.144.7.25: S 1836765456:1836765456(0) win 24820 <nop,wscale 0,nop,nop,sackOK,mss 1452> 13:19:52.789345 IP 10.215.144.7.25 > 194.179.55.129.39005: S 45091514:45091514(0) ack 1836765457 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:19:56.246909 IP 10.215.144.7.25 > 194.179.55.129.39005: S 45091514:45091514(0) ack 1836765457 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:20:02.246117 IP 10.215.144.7.25 > 194.179.55.129.39005: S 45091514:45091514(0) ack 1836765457 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> tcpdump on eth0 on the bridge: 13:13:45.253046 IP 194.179.55.135.49114 > 10.215.144.7.25: S 3140736196:3140736196(0) win 24820 <nop,wscale 0,nop,nop,sackOK,mss 1452> 13:13:45.253265 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:13:49.653649 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:13:55.655017 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:14:07.658590 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:14:31.659389 IP 10.215.144.7.25 > 194.179.55.135.49114: S 4284445628:4284445628(0) ack 3140736197 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:14:41.007658 IP 194.179.55.129.39005 > 10.215.144.7.25: S 1836765456:1836765456(0) win 24820 <nop,wscale 0,nop,nop,sackOK,mss 1452> 13:14:41.007803 IP 10.215.144.7.25 > 194.179.55.129.39005: S 45091514:45091514(0) ack 1836765457 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:14:44.465082 IP 10.215.144.7.25 > 194.179.55.129.39005: S 45091514:45091514(0) ack 1836765457 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:14:50.463779 IP 10.215.144.7.25 > 194.179.55.129.39005: S 45091514:45091514(0) ack 1836765457 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> 13:15:02.466701 IP 10.215.144.7.25 > 194.179.55.129.39005: S 45091514:45091514(0) ack 1836765457 win 32767 <mss 1460,nop,nop,sackOK,nop,wscale 14> Please disregard the time stamps as the two systems'' clocks are skewed. Updated shorewall dumps for both systems (gateway and bridge): http://213.96.91.201/shorewall/ Thank you for your time. ____________________________________________________________________________________ Choose the right car based on your needs. Check out Yahoo! Autos new Car Finder tool. http://autos.yahoo.com/carfinder/ ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Vieri Di Paola wrote:> Please disregard the time stamps as the two systems'' > clocks are skewed. > > Updated shorewall dumps for both systems > (gateway and bridge): > > http://213.96.91.201/shorewall/ >In this dump, I did not see a conntrack entry on the bridge as before but there is one on the outer firewall. tcp 6 0 SYN_RECV src=194.179.55.129 dst=192.168.100.2 sport=39005 dport=25 packets=1 bytes=52 src=10.215.144.7 dst=194.179.55.129 sport=25 dport=39005 packets=6 bytes=312 mark=2 use=1 My advice from yesterday still stands (see my response to my own post) -- you need to understand where the responding SYN,ACK is being lost. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
--- Tom Eastep <teastep@shorewall.net> wrote:> I would next do the tcpdump > on eth0 on the bridge > (make sure the SYN,ACKs are being sent from the > bridge) then on eth1 on the > outer firewall. > > I see this conntrack entry on the bridge: > > tcp 6 35 SYN_RECV src=194.179.55.129 > dst=10.215.144.7 sport=61911 > dport=25 packets=1 bytes=52 src=10.215.144.7 > dst=194.179.55.129 sport=25 > dport=61911 packets=5 bytes=260 mark=0 use=1 > > This indicates that Netfilter connection tracking on > the bridge is not > seeing or not recognizing the SYN,ACK response. > > > http://213.96.91.201/shorewall/ > > In this dump, I did not see a conntrack entry on the > bridge as before but > there is one on the outer firewall. > > tcp 6 0 SYN_RECV src=194.179.55.129 > dst=192.168.100.2 sport=39005 > dport=25 packets=1 bytes=52 src=10.215.144.7 > dst=194.179.55.129 sport=25 > dport=39005 packets=6 bytes=312 mark=2 use=1 > > My advice from yesterday still stands > you need to understand where the responding SYN,ACK > is being lost.I wish I knew how to do that efficiently. Should I `watch` /proc/net/ip_conntrack on both shorewall systems and grep the remote host''s public IP address and log the SYN_* states? I suppose that a "successful" connection would go through stages SYN_SENT, SYN_RECV, ESTABLISHED on both shorewall systems? What really puzzles me is that this is happening only with this particular host (at least according to user feedback), ie. 1 case out of "so many". Anyway, what can be deduced from your two previous findings related to SYN_RECV that appeared on the bridge in yesterday''s dump and on the gateway in today''s dump? Shouldn''t SYN_RECV appear on both? (I did notice that shorewall dump took *a lot* more time to complete on the bridge so maybe the conntrack entry got removed?) I didn''t grasp the meaning of the fact that if SYN_RECV is found on the bridge then that means that conntrack did not see/recognize the SYN/ACK response. ( the network is: WORLD --- GATEWAY --- EXTERNAL LAN --- BRIDGE --- INTERNAL NETWORK WITH SMTP SERVER ) Well, I have a lot of TCP/IP homework now (forgive my ignorance). Thanks for putting me on the right track. ____________________________________________________________________________________ Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. http://mobile.yahoo.com/go?refer=1GNXIC ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
--- Vieri Di Paola <vieridipaola@yahoo.com> wrote:> I didn''t grasp the meaning of the fact that if > SYN_RECV is found on the bridge then that means that > conntrack did not see/recognize the SYN/ACK > response. > ( the network is: WORLD --- GATEWAY --- EXTERNAL LAN > --- BRIDGE --- INTERNAL NETWORK WITH SMTP SERVER )What I seem to understand is that my TCP server receives the connection request from the client and sends out an "ACK request" but never receives it back from the client. Maybe I should update/recompile the same kernel on both systems before making further tests. ____________________________________________________________________________________ Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. http://mobile.yahoo.com/go?refer=1GNXIC ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Vieri Di Paola wrote:> --- Tom Eastep <teastep@shorewall.net> wrote: > >> I would next do the tcpdump >> on eth0 on the bridge >> (make sure the SYN,ACKs are being sent from the >> bridge) then on eth1 on the >> outer firewall. >> >> I see this conntrack entry on the bridge: >> >> tcp 6 35 SYN_RECV src=194.179.55.129 >> dst=10.215.144.7 sport=61911 >> dport=25 packets=1 bytes=52 src=10.215.144.7 >> dst=194.179.55.129 sport=25 >> dport=61911 packets=5 bytes=260 mark=0 use=1 >> >> This indicates that Netfilter connection tracking on >> the bridge is not >> seeing or not recognizing the SYN,ACK response. >> >>> http://213.96.91.201/shorewall/ >> In this dump, I did not see a conntrack entry on the >> bridge as before but >> there is one on the outer firewall. >> >> tcp 6 0 SYN_RECV src=194.179.55.129 >> dst=192.168.100.2 sport=39005 >> dport=25 packets=1 bytes=52 src=10.215.144.7 >> dst=194.179.55.129 sport=25 >> dport=39005 packets=6 bytes=312 mark=2 use=1 >> >> My advice from yesterday still stands >> you need to understand where the responding SYN,ACK >> is being lost. > > I wish I knew how to do that efficiently. Should I > `watch` /proc/net/ip_conntrack on both shorewall > systems and grep the remote host''s public IP address > and log the SYN_* states? > I suppose that a "successful" connection would go > through stages SYN_SENT, SYN_RECV, ESTABLISHED on both > shorewall systems?I think you should run tcpdump on the various interfaces like I suggested.> > What really puzzles me is that this is happening only > with this particular host (at least according to user > feedback), ie. 1 case out of "so many". >The SMTP server doesn''t have any extra routes defined on it that is redirecting replies to that client does it?> Anyway, what can be deduced from your two previous > findings related to SYN_RECV that appeared on the > bridge in yesterday''s dump and on the gateway in > today''s dump?http://iptables-tutorial.frozentux.net/iptables-tutorial.html#TCPCONNECTIONS> Shouldn''t SYN_RECV appear on both? (I did notice that > shorewall dump took *a lot* more time to complete on > the bridge so maybe the conntrack entry got removed?) >Might be -- does your bridge have an enormous log? If so, /sbin/shorewall can take a long time wading through the entire log so it can print the last 20 ''Shorewall'' messages.> I didn''t grasp the meaning of the fact that if > SYN_RECV is found on the bridge then that means that > conntrack did not see/recognize the SYN/ACK response. > ( the network is: WORLD --- GATEWAY --- EXTERNAL LAN > --- BRIDGE --- INTERNAL NETWORK WITH SMTP SERVER )Again, please see http://iptables-tutorial.frozentux.net/iptables-tutorial.html#TCPCONNECTIONS> > Well, I have a lot of TCP/IP homework now (forgive my > ignorance). Thanks for putting me on the right track. >-Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep wrote:> > >> I didn''t grasp the meaning of the fact that if >> SYN_RECV is found on the bridge then that means that >> conntrack did not see/recognize the SYN/ACK response. >> ( the network is: WORLD --- GATEWAY --- EXTERNAL LAN >> --- BRIDGE --- INTERNAL NETWORK WITH SMTP SERVER ) > > Again, please see > http://iptables-tutorial.frozentux.net/iptables-tutorial.html#TCPCONNECTIONS > >> Well, I have a lot of TCP/IP homework now (forgive my >> ignorance). Thanks for putting me on the right track. >>My bad. I guess I need to refer to the above URL more often myself as I was confusing SYN_SENT and SYN_RECV state. From the "shorewall dump" on the gateway: tcp 6 0 SYN_RECV src=194.179.55.129 dst=192.168.100.2 sport=39005 dport=25 packets=1 bytes=52 src=10.215.144.7 dst=194.179.55.129 sport=25 dport=39005 packets=6 bytes=312 mark=2 use=1 This indicates that the gateway has received a response from the SMTP server and has sent it on to the client. And, from the tcpdump on eth3 that you included in your original problem report (that I hadn''t looked at because of the lack of the -n flag), it looks like the response is leaving your firewall. So -- my next suggestion would be to capture a packet trace on the client system (see if the SYN,ACK is being lost somewhere between the gateway and the client) or if the client is finding something wrong with it and discarding it. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On Tue, Aug 14, 2007 at 11:16:44AM -0700, Tom Eastep wrote:> > Shouldn''t SYN_RECV appear on both? (I did notice that > > shorewall dump took *a lot* more time to complete on > > the bridge so maybe the conntrack entry got removed?) > > > > Might be -- does your bridge have an enormous log? If so, /sbin/shorewall > can take a long time wading through the entire log so it can print the last > 20 ''Shorewall'' messages.That seems correctable. Pass the log through tac, use head instead of tail, and pass the output of head through tac again. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On 8/14/07, Andrew Suffield <asuffield@suffields.me.uk> wrote:> That seems correctable. Pass the log through tac, use head instead of > tail, and pass the output of head through tac again.''tail'' is much faster. Even with a huge file (the one I tested with is 26 gigabytes) it simply mmap()s the file and then scans backwards until it''s got enough newlines - it took about 1/20th of a second on that 26GB file. Perhaps shorewall is using a different, less efficient method? Will ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Andrew Suffield wrote:> On Tue, Aug 14, 2007 at 11:16:44AM -0700, Tom Eastep wrote: >>> Shouldn''t SYN_RECV appear on both? (I did notice that >>> shorewall dump took *a lot* more time to complete on >>> the bridge so maybe the conntrack entry got removed?) >>> >> Might be -- does your bridge have an enormous log? If so, /sbin/shorewall >> can take a long time wading through the entire log so it can print the last >> 20 ''Shorewall'' messages. > > That seems correctable. Pass the log through tac, use head instead of > tail, and pass the output of head through tac again.With the current implementation: gateway:~/test # time shorewall show log > /dev/null real 0m0.783s user 0m0.904s sys 0m0.132s gateway:~/test # time shorewall show log > /dev/null real 0m0.776s user 0m1.000s sys 0m0.068s gateway:~/test # time shorewall show log > /dev/null real 0m0.786s user 0m0.888s sys 0m0.112s gateway:~/test # time shorewall show log > /dev/null real 0m0.770s user 0m0.888s sys 0m0.068s gateway:~/test # time shorewall show log > /dev/null real 0m0.182s user 0m0.104s sys 0m0.060s gateway:~/test # With Andrew''s proposed change: gateway:~/test # time shorewall show log > /dev/null real 0m0.188s user 0m0.156s sys 0m0.052s gateway:~/test # time shorewall show log > /dev/null real 0m0.187s user 0m0.132s sys 0m0.060s gateway:~/test # time shorewall show log > /dev/null real 0m0.182s user 0m0.088s sys 0m0.076s gateway:~/test # time shorewall show log > /dev/null real 0m0.195s user 0m0.136s sys 0m0.068s gateway:~/test # Log file was ~2MB and contained nothing but Shorewall messages. Thanks, Andrew. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Will Murnane wrote:> On 8/14/07, Andrew Suffield <asuffield@suffields.me.uk> wrote: >> That seems correctable. Pass the log through tac, use head instead of >> tail, and pass the output of head through tac again. > ''tail'' is much faster. Even with a huge file (the one I tested with > is 26 gigabytes) it simply mmap()s the file and then scans backwards > until it''s got enough newlines - it took about 1/20th of a second on > that 26GB file. Perhaps shorewall is using a different, less > efficient method?The log file can have other record types besides netfilter messages. So Shorewall was piping the output of ''grep'' through ''tail'' which doesn''t permit that optimization. As I reported in the earlier post, switching to Andrew''s algorithm does improve performance. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
On Tue, Aug 14, 2007 at 03:07:37PM -0400, Will Murnane wrote:> On 8/14/07, Andrew Suffield <asuffield@suffields.me.uk> wrote: > > That seems correctable. Pass the log through tac, use head instead of > > tail, and pass the output of head through tac again. > ''tail'' is much faster. Even with a huge file (the one I tested with > is 26 gigabytes) it simply mmap()s the file and then scans backwards > until it''s got enough newlines - it took about 1/20th of a second on > that 26GB file. Perhaps shorewall is using a different, less > efficient method?It''s (source) | grep | sed | sed | tail. I can''t see any straightforward way to improve it other than (source) | tac | grep | head | tac | sed | sed. Note that the problem is of the form "find the last 20 (lines matching the regexp)"; if it was "(find the last 20 lines) matching the regexp" then using tail directly on the file would indeed be faster. ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep wrote:> > With the current implementation: > > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.783s > user 0m0.904s > sys 0m0.132s > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.776s > user 0m1.000s > sys 0m0.068s > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.786s > user 0m0.888s > sys 0m0.112s > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.770s > user 0m0.888s > sys 0m0.068sWith Andrew''s proposed change (previous email had copy/paste error):> gateway:~/test # time shorewall show log > /dev/null > > real 0m0.182s > user 0m0.104s > sys 0m0.060s > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.188s > user 0m0.156s > sys 0m0.052s > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.187s > user 0m0.132s > sys 0m0.060s > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.182s > user 0m0.088s > sys 0m0.076s > gateway:~/test # time shorewall show log > /dev/null > > real 0m0.195s > user 0m0.136s > sys 0m0.068s > gateway:~/test # > > Log file was ~2MB and contained nothing but Shorewall messages.-Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
--- Tom Eastep <teastep@shorewall.net> wrote:> > What really puzzles me is that this is happening > only > > with this particular host (at least according to > user > > feedback), ie. 1 case out of "so many". > > > > The SMTP server doesn''t have any extra routes > defined on it that is > redirecting replies to that client does it?No, route -n on the SMTP server lists: Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 10.215.144.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 10.215.144.91 0.0.0.0 UG 0 0 0 eth0 where 10.215.144.91 is the "shorewall bridge".> From the "shorewall dump" on the gateway: > > tcp 6 0 SYN_RECV src=194.179.55.129 > dst=192.168.100.2 sport=39005 > dport=25 packets=1 bytes=52 src=10.215.144.7 > dst=194.179.55.129 sport=25 > dport=39005 packets=6 bytes=312 mark=2 use=1 > > This indicates that the gateway has received a > response from the SMTP server > and has sent it on to the client. > And, from the tcpdump on eth3 that you included in > your original problem > report (that I hadn''t looked at because of the lack > of the -n flag), it > looks like the response is leaving your firewall.That''s what I thought. So in theory all seems fine on "my side".> So -- my next suggestion would be to capture a > packet trace on the client > system (see if the SYN,ACK is being lost somewhere > between the gateway and > the client) or if the client is finding something > wrong with it and > discarding it.That''s the point I wanted to get to. I thought that the "problem" had to be on the remote client''s side but I needed some kind of proof so as to convince the remote system''s administrator. I now have the data to sustain that a trace on the client side should be performed. I have no idea what their systems are running (soft-hardware) and if they may drop our packets for some reason. Thank you for helping me out. ____________________________________________________________________________________ Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. http://mobile.yahoo.com/go?refer=1GNXIC ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/