Hi, I have a very simple server setup, using shorewall as my firewall. I have a line like this at the top of my rules file to allow ssh connections, but limited to 3 connection per minute with a burst rate of 3: SSH/ACCEPT net $FW - - - - 3/min:3 - Now when I have that in place, and from a remote machine run scp server:/some/file ., I find that the scp stalls after a few kb reproducibly. Altering the above line to SSH/ACCEPT net $FW i.e. with no connection rate limiting fixes the issue and all is well. My understanding of scp is that it opens a connection and uses that for the whole file, and so it shouldn''t be exceeding the connection limit. So what am I doing wrong? I really would like to limit ssh connections. Thanks Jonathan ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 02:54:52AM +0100, Jonathan Underwood wrote:> > SSH/ACCEPT net $FW - - - > - 3/min:3 - > > Now when I have that in place, and from a remote machine run scp > server:/some/file ., I find that the scp stalls after a few kb > reproducibly. Altering the above line to >Could you give an *exact* command line that produces this error? Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote:> On Fri, May 25, 2007 at 02:54:52AM +0100, Jonathan Underwood wrote: > > > > SSH/ACCEPT net $FW - - - > > - 3/min:3 - > > > > Now when I have that in place, and from a remote machine run scp > > server:/some/file ., I find that the scp stalls after a few kb > > reproducibly. Altering the above line to > > > Could you give an *exact* command line that produces this error?scp servername:/home/username/filename . ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 04:39:10PM +0100, Jonathan Underwood wrote:> On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > On Fri, May 25, 2007 at 02:54:52AM +0100, Jonathan Underwood wrote: > > > > > > SSH/ACCEPT net $FW - - - > > > - 3/min:3 - > > > > > > Now when I have that in place, and from a remote machine run scp > > > server:/some/file ., I find that the scp stalls after a few kb > > > reproducibly. Altering the above line to > > > > > Could you give an *exact* command line that produces this error? > > scp servername:/home/username/filename . >You completely missed the point. I meant something like this: roberto@miami:~/tmp$ scp santiago:tmp/*.deb . openswan-modules-2.4.27-3-386_2.4.6+dfsg-1~bp 100% 140KB 139.8KB/s 00:00 As in, *don''t* obfuscate what you are doing. Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote:> On Fri, May 25, 2007 at 04:39:10PM +0100, Jonathan Underwood wrote: > > On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > > On Fri, May 25, 2007 at 02:54:52AM +0100, Jonathan Underwood wrote: > > > > > > > > SSH/ACCEPT net $FW - - - > > > > - 3/min:3 - > > > > > > > > Now when I have that in place, and from a remote machine run scp > > > > server:/some/file ., I find that the scp stalls after a few kb > > > > reproducibly. Altering the above line to > > > > > > > Could you give an *exact* command line that produces this error?$ scp withnail.phys.ucl.ac.uk:/home/jgu/220107.tar.bz2 . Enter passphrase for key ''/home/jgu/.ssh/id_dsa'': 220107.tar.bz2 0% 9464KB 492.9KB/s - stalled - When it stalls, it periodically has bursts of a few KB/s and then returns to zero. Normally on this link without that line in the rules file, I can get 5 MB/s. I''m still not sure I answered your questions though.> > > > scp servername:/home/username/filename . > > > You completely missed the point. > > I meant something like this: > > roberto@miami:~/tmp$ scp santiago:tmp/*.deb . > openswan-modules-2.4.27-3-386_2.4.6+dfsg-1~bp 100% 140KB 139.8KB/s 00:00 > > As in, *don''t* obfuscate what you are doing. > > Regards, > > -Roberto > > -- > Roberto C. Sánchez > http://people.connexer.com/~roberto > http://www.connexer.com > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iD8DBQFGVwaB5SXWIKfIlGQRAsWeAJ9u+8qL9cKUssnOL3GhPXhHlmUZvwCeKNXK > /H5x6Z9Zcoti20R9Uip56IQ> =85xI > -----END PGP SIGNATURE----- > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > >------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Jonathan Underwood <jonathan.underwood@gmail.com> wrote:> On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > On Fri, May 25, 2007 at 04:39:10PM +0100, Jonathan Underwood wrote: > > > On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > > > On Fri, May 25, 2007 at 02:54:52AM +0100, Jonathan Underwood wrote: > > > > > > > > > > SSH/ACCEPT net $FW - - - > > > > > - 3/min:3 - > > > > > > > > > > Now when I have that in place, and from a remote machine run scp > > > > > server:/some/file ., I find that the scp stalls after a few kb > > > > > reproducibly. Altering the above line to > > > > > > > > > Could you give an *exact* command line that produces this error? > > $ scp withnail.phys.ucl.ac.uk:/home/jgu/220107.tar.bz2 . > Enter passphrase for key ''/home/jgu/.ssh/id_dsa'': > 220107.tar.bz2 0% 9464KB 492.9KB/s - stalled - > > When it stalls, it periodically has bursts of a few KB/s and then > returns to zero. Normally on this link without that line in the rules > file, I can get 5 MB/s. > > I''m still not sure I answered your questions though.I should also add that, if when the scp is in the stalled state as described above, I log into the server (withnail) and comment out the limited SSH/ACCEPT line in rules and replace it with the non limited line and restart shorewall, the scp will then resume at full speed. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 05:08:09PM +0100, Jonathan Underwood wrote:> > 220107.tar.bz2 0% 9464KB 492.9KB/s - stalled - > > When it stalls, it periodically has bursts of a few KB/s and then > returns to zero. Normally on this link without that line in the rules > file, I can get 5 MB/s. > > I''m still not sure I answered your questions though. >That helps. If it is stalled, that means that scp (ssh, in fact) still thinks that the connection is open. That must mean that shorewall is in fact stopping the packets. Of course, this is strange, since I also have ssh rate limited on my machine and have not seen such a thing before. What happens if you use sftp and get the file instead of scp? Also, what version of ssh are you running on the server and also on the client? Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 05:25:15PM +0100, Jonathan Underwood wrote:> > I should also add that, if when the scp is in the stalled state as > described above, I log into the server (withnail) and comment out the > limited SSH/ACCEPT line in rules and replace it with the non limited > line and restart shorewall, the scp will then resume at full speed. >Wow. What does shorewall log just before and during the stall of scp? Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote:> That helps. If it is stalled, that means that scp (ssh, in fact) still > thinks that the connection is open. That must mean that shorewall is in > fact stopping the packets. Of course, this is strange, since I also > have ssh rate limited on my machine and have not seen such a thing > before. What happens if you use sftp and get the file instead of scp? >sftp seems to work fine.> Also, what version of ssh are you running on the server and also on the > client? >On both the server and the client I have openssh 4.3p2 J. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote:> On Fri, May 25, 2007 at 05:25:15PM +0100, Jonathan Underwood wrote: > > > > I should also add that, if when the scp is in the stalled state as > > described above, I log into the server (withnail) and comment out the > > limited SSH/ACCEPT line in rules and replace it with the non limited > > line and restart shorewall, the scp will then resume at full speed. > > > Wow. What does shorewall log just before and during the stall of scp?Well, oddly, nothing is logged. I will look into turning up what is logged. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Jonathan Underwood <jonathan.underwood@gmail.com> wrote:> On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > That helps. If it is stalled, that means that scp (ssh, in fact) still > > thinks that the connection is open. That must mean that shorewall is in > > fact stopping the packets. Of course, this is strange, since I also > > have ssh rate limited on my machine and have not seen such a thing > > before. What happens if you use sftp and get the file instead of scp? > > > > sftp seems to work fine.Oh, spoke too soon. I am now seeing sftp stalling as well. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 06:07:17PM +0100, Jonathan Underwood wrote:> On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > That helps. If it is stalled, that means that scp (ssh, in fact) still > > thinks that the connection is open. That must mean that shorewall is in > > fact stopping the packets. Of course, this is strange, since I also > > have ssh rate limited on my machine and have not seen such a thing > > before. What happens if you use sftp and get the file instead of scp? > > > > sftp seems to work fine. > > > Also, what version of ssh are you running on the server and also on the > > client? > > > > On both the server and the client I have openssh 4.3p2 >I forget exactly how it is configured, but you could you investigate whether persistent or cached connections (I forget the exact term that is used by the ssh folks) have been enabled/disabled? Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote:> On Fri, May 25, 2007 at 06:07:17PM +0100, Jonathan Underwood wrote: > > On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > > That helps. If it is stalled, that means that scp (ssh, in fact) still > > > thinks that the connection is open. That must mean that shorewall is in > > > fact stopping the packets. Of course, this is strange, since I also > > > have ssh rate limited on my machine and have not seen such a thing > > > before. What happens if you use sftp and get the file instead of scp? > > > > > > > sftp seems to work fine. > > > > > Also, what version of ssh are you running on the server and also on the > > > client? > > > > > > > On both the server and the client I have openssh 4.3p2 > > > I forget exactly how it is configured, but you could you investigate > whether persistent or cached connections (I forget the exact term that > is used by the ssh folks) have been enabled/disabled? >Do you mean the ControlMaster stuff? (http://www.torchbox.com/blog/ssh_tips_2.html). I''m not making any use of that, there are no configuration entries in /etc/ssh_config or my user ssh settings in .ssh/config so I don''t think that stuff is involved. J.> Regards, > > -Roberto > > -- > Roberto C. Sánchez > http://people.connexer.com/~roberto > http://www.connexer.com > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iD8DBQFGVyI45SXWIKfIlGQRAi8HAKCzeTY2C1sr0SHetC7z0qx+6DylUQCcDfJ/ > jXJNqclAWzoWOh2F6YXKawk> =2ZfJ > -----END PGP SIGNATURE----- > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > >------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Jonathan Underwood wrote:> SSH/ACCEPT net $FW - - - - 3/min:3I would add logging to that statement and see what happens. eg: SSH/ACCEPT:info net $FW - - - - 3/min:3 ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Simon Hobson <linux@thehobsons.co.uk> wrote:> Jonathan Underwood wrote: > > SSH/ACCEPT net $FW - - - - 3/min:3 > > I would add logging to that statement and see what happens. > eg: > > SSH/ACCEPT:info net $FW - - - - 3/min:3This results in these messages (with a couple of obfuscations in IP addresses: Shorewall:net2fw:ACCEPT:IN=eth0 OUTMAC=00:xx:xx:xx:c2:49:00:d0:79:xx:98:00:08:00 SRC=130.xx.69.87 DST=128.xx.2.35 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=26203 DF PROTO=TCP SPT=53708 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 Shorewall:net2fw:ACCEPT:IN=eth0 OUTMAC=00:xx:xx:xx:xx:49:00:xx:79:xx:98:00:08:00 SRC=130.xx.69.87 DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26293 DF PROTO=TCP SPT=53708 DPT=22 WINDOW=2485 RES=0x00 ACK URGP=0 Shorewall:net2fw:ACCEPT:IN=eth0 OUTMAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87 DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26324 DF PROTO=TCP SPT=53708 DPT=22 WINDOW=3995 RES=0x00 ACK URGP=0 Shorewall:net2fw:ACCEPT:IN=eth0 OUTMAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87 DST=128.xx.2.35 LEN=72 TOS=0x00 PREC=0x00 TTL=53 ID=27573 DF PROTO=TCP SPT=53708 DPT=22 WINDOW=6036 RES=0x00 ACK URGP=0> > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users >------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Jonathan Underwood <jonathan.underwood@gmail.com> wrote:> On 25/05/07, Simon Hobson <linux@thehobsons.co.uk> wrote: > > Jonathan Underwood wrote: > > > SSH/ACCEPT net $FW - - - - 3/min:3 > > > > I would add logging to that statement and see what happens. > > eg: > > > > SSH/ACCEPT:info net $FW - - - - 3/min:3 > This results in these messages (with a couple of obfuscations in IP addresses: > > Shorewall:net2fw:ACCEPT:IN=eth0 OUT> MAC=00:xx:xx:xx:c2:49:00:d0:79:xx:98:00:08:00 SRC=130.xx.69.87 > DST=128.xx.2.35 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=26203 DF PROTO=TCP > SPT=53708 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 > Shorewall:net2fw:ACCEPT:IN=eth0 OUT> MAC=00:xx:xx:xx:xx:49:00:xx:79:xx:98:00:08:00 SRC=130.xx.69.87 > DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26293 DF PROTO=TCP > SPT=53708 DPT=22 WINDOW=2485 RES=0x00 ACK URGP=0 > Shorewall:net2fw:ACCEPT:IN=eth0 OUT> MAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87 > DST=128.xx.2.35 LEN=64 TOS=0x00 PREC=0x00 TTL=53 ID=26324 DF PROTO=TCP > SPT=53708 DPT=22 WINDOW=3995 RES=0x00 ACK URGP=0 > Shorewall:net2fw:ACCEPT:IN=eth0 OUT> MAC=00:xx:xx:xx:xx:49:00:d0:79:95:98:00:08:00 SRC=130.xx.69.87 > DST=128.xx.2.35 LEN=72 TOS=0x00 PREC=0x00 TTL=53 ID=27573 DF PROTO=TCP > SPT=53708 DPT=22 WINDOW=6036 RES=0x00 ACK URGP=0oh. Duh. I''m dumb - they''re obviously the messages corresponding to the ssh session I have open to examine the logs on the remote server :) So it seems the stalled scp transfer isn''t causing anything to be logged. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 08:24:00PM +0100, Jonathan Underwood wrote:> > oh. Duh. I''m dumb - they''re obviously the messages corresponding to > the ssh session I have open to examine the logs on the remote server > :) > > So it seems the stalled scp transfer isn''t causing anything to be logged. >I''m completely baffled. Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote:> On Fri, May 25, 2007 at 08:24:00PM +0100, Jonathan Underwood wrote: > > > > oh. Duh. I''m dumb - they''re obviously the messages corresponding to > > the ssh session I have open to examine the logs on the remote server > > :) > > > > So it seems the stalled scp transfer isn''t causing anything to be logged. > > > I''m completely baffled.Me too :) ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, 2007-05-25 at 22:34 +0100, Jonathan Underwood wrote:> On 25/05/07, Roberto C. Sánchez <roberto@connexer.com> wrote: > > On Fri, May 25, 2007 at 08:24:00PM +0100, Jonathan Underwood wrote: > > > > > > oh. Duh. I''m dumb - they''re obviously the messages corresponding to > > > the ssh session I have open to examine the logs on the remote server > > > :) > > > > > > So it seems the stalled scp transfer isn''t causing anything to be logged. > > > > > I''m completely baffled. > > Me too :)Maybe a silly question, and maybe covered at the start of the thread, but does this all work without shorewall installing a ruleset? i.e. if you do a "shorewall clear" does everything magically work again? This smacks of an MSS/PPPoE type problem where only full TCP segments get dropped. Or there is that sub-protocol that does probes of the connection that gets fouled up by routers doing ICMP blackholing. I forget what that sub-protocol was though. Anyone remember? b. -- My other computer is your Microsoft Windows server. Brian J. Murrell ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 05:48:15PM -0400, Brian J. Murrell wrote:> > Maybe a silly question, and maybe covered at the start of the thread, > but does this all work without shorewall installing a ruleset? i.e. if > you do a "shorewall clear" does everything magically work again? >He did mention that upon removing the rate limit part of the rule and restarting shorewall, everything works correctly. Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, 2007-05-25 at 17:48 -0400, Brian J. Murrell wrote:> > This smacks of an MSS/PPPoE type problem where only full TCP segments > get dropped.I.e. the type of problem that "clamp_mss" was invented to solve.> Or there is that sub-protocol that does probes of the > connection that gets fouled up by routers doing ICMP blackholing. I > forget what that sub-protocol was though. Anyone remember?I do. :-) PMTU discovery. Both of the above problems should exist with or without Shorewall''s rules installed in iptables. b. -- My other computer is your Microsoft Windows server. Brian J. Murrell ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 25/05/07, Brian J. Murrell <brian@interlinx.bc.ca> wrote:> Maybe a silly question, and maybe covered at the start of the thread, > but does this all work without shorewall installing a ruleset? i.e. if > you do a "shorewall clear" does everything magically work again? >Yes, issuing a shorewall clear, or alternatively removing the limit part of the SSH rule fixes the problem reproducibly.> This smacks of an MSS/PPPoE type problem where only full TCP segments > get dropped. Or there is that sub-protocol that does probes of the > connection that gets fouled up by routers doing ICMP blackholing. I > forget what that sub-protocol was though. Anyone remember?Hm. This is a bit over my head I''m afraid. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, 2007-05-25 at 23:04 +0100, Jonathan Underwood wrote:> Yes, issuing a shorewall clear, or alternatively removing the limit > part of the SSH rule fixes the problem reproducibly.I''m afraid my day was too busy to pay attention to the start of this thread, but that you can install rules with shorewall and just not install the limit part suggests to me that it''s not either of the problems below.> > This smacks of an MSS/PPPoE type problem where only full TCP segments > > get dropped. Or there is that sub-protocol that does probes of the > > connection that gets fouled up by routers doing ICMP blackholing. I > > forget what that sub-protocol was though. Anyone remember? > > Hm. This is a bit over my head I''m afraid.Given the above, it''s probably not relevant, so you probably needed worry about it. b. -- My other computer is your Microsoft Windows server. Brian J. Murrell ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Fri, May 25, 2007 at 05:17:09PM -0400, Roberto C. S?nchez wrote:> On Fri, May 25, 2007 at 08:24:00PM +0100, Jonathan Underwood wrote: > > > > oh. Duh. I''m dumb - they''re obviously the messages corresponding to > > the ssh session I have open to examine the logs on the remote server > > :) > > > > So it seems the stalled scp transfer isn''t causing anything to be logged. > > > I''m completely baffled.Capturing the traffic with tcpdump -s 4096 -w (simultaneously on all involved hosts) may be more informative. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 26/05/07, Andrew Suffield <asuffield@suffields.me.uk> wrote:> On Fri, May 25, 2007 at 05:17:09PM -0400, Roberto C. S?nchez wrote: > > On Fri, May 25, 2007 at 08:24:00PM +0100, Jonathan Underwood wrote: > > > > > > oh. Duh. I''m dumb - they''re obviously the messages corresponding to > > > the ssh session I have open to examine the logs on the remote server > > > :) > > > > > > So it seems the stalled scp transfer isn''t causing anything to be logged. > > > > > I''m completely baffled. > > Capturing the traffic with tcpdump -s 4096 -w (simultaneously on all > involved hosts) may be more informative.I''ve never managed to climb the tcpdump learning curve, but I just fired up wireshark on the client machine and set it to filter all packets to/from the server machine. What jumps out at me is this: a) When I hit a stall in the scp transfer, the first suspect packet I see contains "A segment before this frame was lost" in the TCP analysis flags. This packet has source being the server and destination being the local machine. b) Then the next packet''s TCP analysis flags contains "Duplicate to the ACK in frame 299". This packet has source being the local client machine, and destination being the server. Frame 299 is the packet before the packet flagged as "A segment before this frame was lost" that I mentioned in (a) above. c) The I see a continual sequence of packets of type (b), followed by a packet from the server to the client with protocol SSHv2 and seemingly being an ACK to the previous packet. d) After a while of (c), I see packets tagged as retransmission. I''m afraid I don''t know enough to interpret all this. If it''s useful to send over the wireshark capture file, let me know. J ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Another thing that may help as a sanity check is that at the point where an scp is stalling, on the server there are no entries under /proc/net/ipt_recent ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Sat, May 26, 2007 at 11:46:27AM +0100, Jonathan Underwood wrote:> On 26/05/07, Andrew Suffield <asuffield@suffields.me.uk> wrote: > > On Fri, May 25, 2007 at 05:17:09PM -0400, Roberto C. S?nchez wrote: > > > On Fri, May 25, 2007 at 08:24:00PM +0100, Jonathan Underwood wrote: > > > > > > > > oh. Duh. I''m dumb - they''re obviously the messages corresponding to > > > > the ssh session I have open to examine the logs on the remote server > > > > :) > > > > > > > > So it seems the stalled scp transfer isn''t causing anything to be logged. > > > > > > > I''m completely baffled. > > > > Capturing the traffic with tcpdump -s 4096 -w (simultaneously on all > > involved hosts) may be more informative. > > I''ve never managed to climb the tcpdump learning curve, but I just > fired up wireshark on the client machine and set it to filter all > packets to/from the server machine.> I''m afraid I don''t know enough to interpret all this. If it''s useful > to send over the wireshark capture file, let me know.tcpdump -w just saves the traffic to a file. Saving the wireshark capture does exactly the same thing, it''s just easier to install tcpdump; either way will work fine. Posting the captures so we can look at it is probably the only thing left to do at this point, given how bizarre this problem is. Remember - it''s important to get a capture of the *same* session from all the interesting points (at least the server, client, and both interfaces of the firewall). We''ll also need the output of ''shorewall dump'' (I don''t think you posted that yet). Follow #3 on http://shorewall.net/support.htm> a) When I hit a stall in the scp transfer, the first suspect packet I > see contains "A segment before this frame was lost" in the TCP > analysis flags. This packet has source being the server and > destination being the local machine.This part tells you that something is going wrong at a point closer to the server than the one where you captured. The next thing you would want to do is to look at the packet that was dropped and figure out what made it different from all the other ones, so you would need to compare this to a capture from the other end - hence why you need to capture at multiple points (one to tell you which packet is dropped, the other to show you that packet''s contents). ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Hi,
i have a setup where the firewall creates four pptp tunnels (austrian
provider setup for multiple ip addresses) and i have no idea how to
setup traffic shaping in this configuration
if i shape the encapsulated packets on the link to the modem, the
firewall shapes only GRE packets, so this doesnt work.
is there a possibility like to put all pptp tunnels to a bridge and
reroute them via traffic shaping or can i use a common limit on a bunch
of interfaces so the count together, or am i stuck ?
Thanks,
Felix
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
Felix Erkinger
2007-May-26 12:20 UTC
inside access of external address port back to internal address
Hi, i have a setup with a xen server and to interfaces, One to the public net, and one a direct link to a other xen server used for intra domain communication. dom0 is setup with to bridges, netbr and clubr (cluster bridge), where the dom0 only has a ip address on eth1 on the clubr bridge. There is a domu with two virtual interfaces bound to both clubr and netbr, which makes the firewalling, routing and natting. Every domu has a interfaces connected to the clubr, so intra cluster communication works, and outside access goes via the gateway domu. There is a mailserver which gets the smtp/ssmtp ports from the public interfaces natted to his internal address. Im using shorewall 3.0.5. Alltough a fancy setup, it works. The problem with that is, if a internal domu (except the firewall domu where it is working) wants to access the public mailserver via his official address it doesnt get routed back to the internal address. "picture": web domu (1.2.3.3 -> smtp mail.mysetup.org (external) ->via-> gateway domu (1.2.3.1) ->via-> mail domu (1.2.3.2) (doesnt work) gateway domu rules: # Route external firewall address:25 to internal mailserver DNAT net clu:1.2.3.2:25 tcp 25 # experimental rule (doesnt work, currently disabled and makes strange things) DNAT clu clu:1.2.3.2:25 tcp 25 gateway domu interfaces: net eth0 detect tcpflags,norfc1918 clu eth1 detect routeback Thanks for any idea, Felix ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Tom Eastep
2007-May-26 12:40 UTC
Re: inside access of external address port back to internal address
Felix Erkinger wrote:> Hi, > > i have a setup with a xen server and to interfaces, One to the public > net, and one a direct link to a other xen server used for intra domain > communication. > > dom0 is setup with to bridges, netbr and clubr (cluster bridge), where > the dom0 only has a ip address on eth1 on the clubr bridge. > > There is a domu with two virtual interfaces bound to both clubr and > netbr, which makes the firewalling, routing and natting. > > Every domu has a interfaces connected to the clubr, so intra cluster > communication works, and outside access goes via the gateway domu. > > There is a mailserver which gets the smtp/ssmtp ports from the public > interfaces natted to his internal address. > Im using shorewall 3.0.5. > Alltough a fancy setup, it works. > > The problem with that is, if a internal domu (except the firewall domu > where it is working) wants to access the public mailserver via his > official address it doesnt get routed back to the internal address. > > "picture": > web domu (1.2.3.3 -> smtp mail.mysetup.org (external) > ->via-> gateway domu (1.2.3.1) ->via-> mail domu (1.2.3.2) > (doesnt work) > > gateway domu rules: > # Route external firewall address:25 to internal mailserver > DNAT net clu:1.2.3.2:25 tcp 25 > # experimental rule (doesnt work, currently disabled and makes strange > things) > DNAT clu clu:1.2.3.2:25 tcp 25 > > gateway domu interfaces: > net eth0 detect tcpflags,norfc1918 > clu eth1 detect routebackThis sounds like Shorewall FAQ #2. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Felix Erkinger wrote:> Hi, > > i have a setup where the firewall creates four pptp tunnels (austrian > provider setup for multiple ip addresses) and i have no idea how to > setup traffic shaping in this configuration > > if i shape the encapsulated packets on the link to the modem, the > firewall shapes only GRE packets, so this doesnt work. > > is there a possibility like to put all pptp tunnels to a bridge and > reroute them via traffic shaping or can i use a common limit on a bunch > of interfaces so the count together, or am i stuck ?I believe you are stuck. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Felix Erkinger
2007-May-26 13:02 UTC
Re: inside access of external address port back to internal address
Tom Eastep schrieb:> This sounds like Shorewall FAQ #2.Aehhm, sorry, i think this solves it .... (shame on me, for not searching enough) Thank you, Felix ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 26/05/07, Andrew Suffield <asuffield@suffields.me.uk> wrote:> tcpdump -w just saves the traffic to a file. Saving the wireshark > capture does exactly the same thing, it''s just easier to install > tcpdump; either way will work fine. Posting the captures so we can > look at it is probably the only thing left to do at this point, given > how bizarre this problem is. > > Remember - it''s important to get a capture of the *same* session from > all the interesting points (at least the server, client, and both > interfaces of the firewall). >OK, I''ll need a bit of time to do this...> We''ll also need the output of ''shorewall dump'' (I don''t think you > posted that yet). Follow #3 on http://shorewall.net/support.htm >But this bit I have just done. I restarted shorewall with rate limiting in the ssh rule, on the server, and on my local machine tried to scp a file from the server to local machine, which stalled. While it was stalled (i.e. I didn''t ctrl-c out) i did a dump, the result of which is attached. I''ll work on getting useful tcpdump/wireshark output from the server. J. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Jonathan Underwood wrote:> On 26/05/07, Andrew Suffield <asuffield@suffields.me.uk> wrote: > >> tcpdump -w just saves the traffic to a file. Saving the wireshark >> capture does exactly the same thing, it''s just easier to install >> tcpdump; either way will work fine. Posting the captures so we can >> look at it is probably the only thing left to do at this point, given >> how bizarre this problem is. >> >> Remember - it''s important to get a capture of the *same* session from >> all the interesting points (at least the server, client, and both >> interfaces of the firewall). >> > > OK, I''ll need a bit of time to do this... > >> We''ll also need the output of ''shorewall dump'' (I don''t think you >> posted that yet). Follow #3 on http://shorewall.net/support.htm >> > > But this bit I have just done. I restarted shorewall with rate > limiting in the ssh rule, on the server, and on my local machine tried > to scp a file from the server to local machine, which stalled. While > it was stalled (i.e. I didn''t ctrl-c out) i did a dump, the result of > which is attached. > > I''ll work on getting useful tcpdump/wireshark output from the server.A couple of things. a) You are using the RATE LIMIT column of the rules file to limit SSH. That is *not* recommended. Rather, we prefer the ''Limit'' built-in action. The former limits the total number of connections from all sources while the latter is per-IP. So your observation that there was nothing in /proc/.../ipt_recent is correct; with your ruleset, there will *never* be anything there. b) You are getting a lot of INVALID state packets. You might try adding this to your /etc/shorewall/init file: echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Tom Eastep wrote:> Jonathan Underwood wrote: >> On 26/05/07, Andrew Suffield <asuffield@suffields.me.uk> wrote: >> >>> tcpdump -w just saves the traffic to a file. Saving the wireshark >>> capture does exactly the same thing, it''s just easier to install >>> tcpdump; either way will work fine. Posting the captures so we can >>> look at it is probably the only thing left to do at this point, given >>> how bizarre this problem is. >>> >>> Remember - it''s important to get a capture of the *same* session from >>> all the interesting points (at least the server, client, and both >>> interfaces of the firewall). >>> >> OK, I''ll need a bit of time to do this... >> >>> We''ll also need the output of ''shorewall dump'' (I don''t think you >>> posted that yet). Follow #3 on http://shorewall.net/support.htm >>> >> But this bit I have just done. I restarted shorewall with rate >> limiting in the ssh rule, on the server, and on my local machine tried >> to scp a file from the server to local machine, which stalled. While >> it was stalled (i.e. I didn''t ctrl-c out) i did a dump, the result of >> which is attached. >> >> I''ll work on getting useful tcpdump/wireshark output from the server. > > A couple of things. > > a) You are using the RATE LIMIT column of the rules file to limit SSH. > That is *not* recommended. Rather, we prefer the ''Limit'' built-in > action. The former limits the total number of connections from all > sources while the latter is per-IP. So your observation that there was > nothing in /proc/.../ipt_recent is correct; with your ruleset, there > will *never* be anything there. > > b) You are getting a lot of INVALID state packets. You might try adding > this to your /etc/shorewall/init file: > > echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal >You can cause Netfilter to display invalid state packets that it drops with this command echo 255 >/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid modprobe ipt_LOG Warning: This will log to all terminal console sessions! -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Felix Erkinger
2007-May-26 13:56 UTC
Re: multiple pptp client on firewall and traffic shaping
Tom Eastep wrote:> Felix Erkinger wrote: >> Hi, >> >> i have a setup where the firewall creates four pptp tunnels (austrian >> provider setup for multiple ip addresses) and i have no idea how to >> setup traffic shaping in this configuration >> >> if i shape the encapsulated packets on the link to the modem, the >> firewall shapes only GRE packets, so this doesnt work. >> >> is there a possibility like to put all pptp tunnels to a bridge and >> reroute them via traffic shaping or can i use a common limit on a bunch >> of interfaces so the count together, or am i stuck ? > > I believe you are stuck.Hmm, what about setting up a bridge (eg. "tonet"), adding a private ip address to that bridge, setting the default route to the ip address of that bridge, mark all traffic for the corresponding pptp tunnels with fwmark x create four ip rules that sorts packets with fwmark x to table ppp-x if coming from dev tonet, add four ip routes for the default gw of dev ppp-x and table ppp-x Could this work ? ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Felix Erkinger wrote:> Tom Eastep wrote: >> Felix Erkinger wrote: >>> Hi, >>> >>> i have a setup where the firewall creates four pptp tunnels (austrian >>> provider setup for multiple ip addresses) and i have no idea how to >>> setup traffic shaping in this configuration >>> >>> if i shape the encapsulated packets on the link to the modem, the >>> firewall shapes only GRE packets, so this doesnt work. >>> >>> is there a possibility like to put all pptp tunnels to a bridge and >>> reroute them via traffic shaping or can i use a common limit on a bunch >>> of interfaces so the count together, or am i stuck ? >> I believe you are stuck. > > Hmm, what about setting up a bridge (eg. "tonet"), > adding a private ip address to that bridge, > setting the default route to the ip address of that bridge, > mark all traffic for the corresponding pptp tunnels with fwmark x > create four ip rules that sorts packets with fwmark x to table ppp-x if > coming from dev tonet, > add four ip routes for the default gw of dev ppp-x and table ppp-x > > Could this work ?I don''t understand your proposal. What devices would you add to the bridge (can''t be PPtP devices)? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Felix Erkinger
2007-May-26 14:19 UTC
Re: multiple pptp client on firewall and traffic shaping
Tom Eastep schrieb:> Felix Erkinger wrote: >> Tom Eastep wrote: >>> Felix Erkinger wrote: >>>> Hi, >>>> >>>> i have a setup where the firewall creates four pptp tunnels (austrian >>>> provider setup for multiple ip addresses) and i have no idea how to >>>> setup traffic shaping in this configuration >>>> >>>> if i shape the encapsulated packets on the link to the modem, the >>>> firewall shapes only GRE packets, so this doesnt work. >>>> >>>> is there a possibility like to put all pptp tunnels to a bridge and >>>> reroute them via traffic shaping or can i use a common limit on a bunch >>>> of interfaces so the count together, or am i stuck ? >>> I believe you are stuck. >> Hmm, what about setting up a bridge (eg. "tonet"), >> adding a private ip address to that bridge, >> setting the default route to the ip address of that bridge, >> mark all traffic for the corresponding pptp tunnels with fwmark x >> create four ip rules that sorts packets with fwmark x to table ppp-x if >> coming from dev tonet, >> add four ip routes for the default gw of dev ppp-x and table ppp-x >> >> Could this work ? > > I don''t understand your proposal. What devices would you add to the > bridge (can''t be PPtP devices)?hmm, none, i just poked arround, and added a bridge, with "brctl add tonet" and assigned an ip address to it with "ifconfig tonet 10.10.0.1" after that i can ping it, and assign the default gw route to it, so there must be a ip stack answering the ping, so there could be a possibility to reroute ? if everything that want to get outside, gets rerouted via the bridge, so my thinking, i could do traffic shapping on that bridge ? and while poking arround> -Tom > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > > > ------------------------------------------------------------------------ > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Felix Erkinger wrote:> > Tom Eastep schrieb: >> Felix Erkinger wrote: >>> Tom Eastep wrote: >>>> Felix Erkinger wrote: >>>>> Hi, >>>>> >>>>> i have a setup where the firewall creates four pptp tunnels (austrian >>>>> provider setup for multiple ip addresses) and i have no idea how to >>>>> setup traffic shaping in this configuration >>>>> >>>>> if i shape the encapsulated packets on the link to the modem, the >>>>> firewall shapes only GRE packets, so this doesnt work. >>>>> >>>>> is there a possibility like to put all pptp tunnels to a bridge and >>>>> reroute them via traffic shaping or can i use a common limit on a bunch >>>>> of interfaces so the count together, or am i stuck ? >>>> I believe you are stuck. >>> Hmm, what about setting up a bridge (eg. "tonet"), >>> adding a private ip address to that bridge, >>> setting the default route to the ip address of that bridge, >>> mark all traffic for the corresponding pptp tunnels with fwmark x >>> create four ip rules that sorts packets with fwmark x to table ppp-x if >>> coming from dev tonet, >>> add four ip routes for the default gw of dev ppp-x and table ppp-x >>> >>> Could this work ? >> I don''t understand your proposal. What devices would you add to the >> bridge (can''t be PPtP devices)? > > hmm, none, i just poked arround, and added a bridge, with > "brctl add tonet" > and assigned an ip address to it with > "ifconfig tonet 10.10.0.1" > after that i can ping it, and assign the default gw route to it, so > there must be a ip stack answering the ping, so there could be a > possibility to reroute ?Not that I can see. What this calls for is an Intermediate Queuing Device (IQD) attached to your internal interface. Unfortunately, I''ve never been able to make one work (but I haven''t tried very hard either). Warning: It requires kernel patching. See the LARTC Howto. YMMV -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On Sat, May 26, 2007 at 06:43:26AM -0700, Tom Eastep wrote:> >> We''ll also need the output of ''shorewall dump'' (I don''t think you > >> posted that yet). Follow #3 on http://shorewall.net/support.htm > >> > > > > But this bit I have just done. I restarted shorewall with rate > > limiting in the ssh rule, on the server, and on my local machine tried > > to scp a file from the server to local machine, which stalled. While > > it was stalled (i.e. I didn''t ctrl-c out) i did a dump, the result of > > which is attached. > > > > I''ll work on getting useful tcpdump/wireshark output from the server.> b) You are getting a lot of INVALID state packets.Which leads me to suspect that we''re looking at a clusterfuck here. Hypothesis: something *else* is wrong, and is breaking TCP connections at intervals. Under normal circumstances, some kind of error recovery manages to get the connection going again, and the problem is not so pronounced that you''ve noticed it before (particularly given scp''s highly inaccurate reporting of the transfer rate, which tends to hide jitter). However, with the rate limit in place, it''s somehow blocking that from happening. There''s no evidence to back this up, but it''s the first thing I''ve been able to think of which explains what could be going on. I do notice this, which is interesting: Chain net2fw (1 references) pkts bytes target prot opt in out source destination 216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3 Why *four* new ssh connections? That''ll certainly have hit the rate limit, but where did the other three come from? ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Felix Erkinger
2007-May-26 14:50 UTC
Re: multiple pptp client on firewall and traffic shaping
Tom Eastep schrieb:> Felix Erkinger wrote: >> Tom Eastep schrieb: >>> Felix Erkinger wrote: >>>> Tom Eastep wrote: >>>>> Felix Erkinger wrote: >>>>>> Hi, >>>>>> >>>>>> i have a setup where the firewall creates four pptp tunnels (austrian >>>>>> provider setup for multiple ip addresses) and i have no idea how to >>>>>> setup traffic shaping in this configuration >>>>>> >>>>>> if i shape the encapsulated packets on the link to the modem, the >>>>>> firewall shapes only GRE packets, so this doesnt work. >>>>>> >>>>>> is there a possibility like to put all pptp tunnels to a bridge and >>>>>> reroute them via traffic shaping or can i use a common limit on a bunch >>>>>> of interfaces so the count together, or am i stuck ? >>>>> I believe you are stuck. >>>> Hmm, what about setting up a bridge (eg. "tonet"), >>>> adding a private ip address to that bridge, >>>> setting the default route to the ip address of that bridge, >>>> mark all traffic for the corresponding pptp tunnels with fwmark x >>>> create four ip rules that sorts packets with fwmark x to table ppp-x if >>>> coming from dev tonet, >>>> add four ip routes for the default gw of dev ppp-x and table ppp-x >>>> >>>> Could this work ? >>> I don''t understand your proposal. What devices would you add to the >>> bridge (can''t be PPtP devices)? >> hmm, none, i just poked arround, and added a bridge, with >> "brctl add tonet" >> and assigned an ip address to it with >> "ifconfig tonet 10.10.0.1" >> after that i can ping it, and assign the default gw route to it, so >> there must be a ip stack answering the ping, so there could be a >> possibility to reroute ? > > Not that I can see. > > What this calls for is an Intermediate Queuing Device (IQD) attached to > your internal interface. Unfortunately, I''ve never been able to make one > work (but I haven''t tried very hard either). Warning: It requires kernel > patching. See the LARTC Howto.i will investigate a little bit further regarding your information, thank you for your help and trying to make sense out of my ideas :-) , Felix ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Andrew Suffield wrote:> On Sat, May 26, 2007 at 06:43:26AM -0700, Tom Eastep wrote: >>>> We''ll also need the output of ''shorewall dump'' (I don''t think you >>>> posted that yet). Follow #3 on http://shorewall.net/support.htm >>>> >>> But this bit I have just done. I restarted shorewall with rate >>> limiting in the ssh rule, on the server, and on my local machine tried >>> to scp a file from the server to local machine, which stalled. While >>> it was stalled (i.e. I didn''t ctrl-c out) i did a dump, the result of >>> which is attached. >>> >>> I''ll work on getting useful tcpdump/wireshark output from the server. > >> b) You are getting a lot of INVALID state packets. > > Which leads me to suspect that we''re looking at a clusterfuck > here. Hypothesis: something *else* is wrong, and is breaking TCP > connections at intervals. Under normal circumstances, some kind of > error recovery manages to get the connection going again, and the > problem is not so pronounced that you''ve noticed it before > (particularly given scp''s highly inaccurate reporting of the transfer > rate, which tends to hide jitter). However, with the rate limit in > place, it''s somehow blocking that from happening. There''s no evidence > to back this up, but it''s the first thing I''ve been able to think of > which explains what could be going on. > > I do notice this, which is interesting: > > Chain net2fw (1 references) > pkts bytes target prot opt in out source destination > 216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED > 4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3 > > Why *four* new ssh connections? That''ll certainly have hit the rate > limit, but where did the other three come from?They are INVALID (both NEW and INVALID go through the rules) and are silently dropped in Drop. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Tom Eastep wrote:> Andrew Suffield wrote: >> >> I do notice this, which is interesting: >> >> Chain net2fw (1 references) >> pkts bytes target prot opt in out source destination >> 216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED >> 4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3 >> >> Why *four* new ssh connections? That''ll certainly have hit the rate >> limit, but where did the other three come from? > > They are INVALID (both NEW and INVALID go through the rules) and are > silently dropped in Drop.Note that if the ACCEPT rule has no ''limit'' then the INVALID packets are accepted and the problem magically goes away. But because these packets occur regularly, they eventually exhaust any imposed ''limit'' and the connection then stalls. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Tom Eastep wrote:> Tom Eastep wrote: >> Andrew Suffield wrote: >>> I do notice this, which is interesting: >>> >>> Chain net2fw (1 references) >>> pkts bytes target prot opt in out source destination >>> 216 17116 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED >>> 4 276 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 limit: avg 3/min burst 3 >>> >>> Why *four* new ssh connections? That''ll certainly have hit the rate >>> limit, but where did the other three come from? >> They are INVALID (both NEW and INVALID go through the rules) and are >> silently dropped in Drop. > > Note that if the ACCEPT rule has no ''limit'' then the INVALID packets are > accepted and the problem magically goes away. But because these packets > occur regularly, they eventually exhaust any imposed ''limit'' and the > connection then stalls.Until Jonathan finds and corrects the root cause, a workaround would be to precede his Limit/(ACCEPT...limit) rule with: allowInvalid net fw tcp 22 -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 26/05/07, Tom Eastep <teastep@shorewall.net> wrote:> A couple of things. > > a) You are using the RATE LIMIT column of the rules file to limit SSH. > That is *not* recommended. Rather, we prefer the ''Limit'' built-in > action. The former limits the total number of connections from all > sources while the latter is per-IP. So your observation that there was > nothing in /proc/.../ipt_recent is correct; with your ruleset, there > will *never* be anything there. >Ok. so, acting on this piece of information alone, I changed the ssh rule to read Limit:info:SSHA,3,60 net $FW tcp 22 That alone did not help the situation in that when I try and scp from the server the scp stalls. If I stop the scp (Ctrl-C) and then repeat the command, no connection is allowed and shorewall on the server logs that it is dropping the packets from my local machine: May 26 21:41:20 withnail kernel: Shorewall:SSHA:DROP:IN=eth0 OUTMAC=00:19:b9:00:c2:49:00:d0:79:95:98:00:08:00 SRC=90.193.1 82.62 DST=128.40.2.35 LEN=72 TOS=0x00 PREC=0x00 TTL=51 ID=17706 DF PROTO=TCP SPT=34948 DPT=22 WINDOW=3555 RES=0x00 ACK FIN U RGP=0 Of course, these two commands, which should equate to only two connections, should not be triggering the limit of (3,60) to be kicking in. This behaviour is reproducible.> b) You are getting a lot of INVALID state packets. You might try adding > this to your /etc/shorewall/init file: > > echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal >Next I did this (in combination with the change above), and everything works just as it should. I tried one other thing - I reverted this last change (echo 0 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal) and turned off tcp window scaling on the server (i.e. echo 0 > /proc/sys/net/ipv4/tcp_window_scaling). That also made everything work as it should, with no stalling. So, Tom, it seems you''re correct in your analysis. [Aside: Just for kicks, I reverted to the original rate limited ssh rule, and that too works as expected if I do echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal.] So if I understand you (Tom) correctly, this is an issue with the network the server is on, or the firewall it is behind that is causing problems with certain server firewall configurations? [Just in case that''s a confusing sentence, here we have been debugging a shorewall generated firewall, but the machine that firewall is on is behind a LAN firewall allowing ssh connections through to the server in question] I realize at this point it is probably clear that this is no bug with shorewall, but I wonder if you might have any suggestion about how I would go about finding what is causing all of those INVALID packets. Thanks to Tom, Andrew, Brian, Roberto and Simon for your suggestions, it really has been much appreciated. jonathan.> -Tom > -- > Tom Eastep \ Nothing is foolproof to a sufficiently talented fool > Shoreline, \ http://shorewall.net > Washington USA \ teastep@shorewall.net > PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > > >------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 26/05/07, Tom Eastep <teastep@shorewall.net> wrote:> Note that if the ACCEPT rule has no ''limit'' then the INVALID packets are > accepted and the problem magically goes away. But because these packets > occur regularly, they eventually exhaust any imposed ''limit'' and the > connection then stalls.Just to make sure I understand this correctly - do you mean that INVALID packets are "counted" as NEW packets as far as the limit is concerned? J. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Jonathan Underwood wrote:> On 26/05/07, Tom Eastep <teastep@shorewall.net> wrote: >> Note that if the ACCEPT rule has no ''limit'' then the INVALID packets are >> accepted and the problem magically goes away. But because these packets >> occur regularly, they eventually exhaust any imposed ''limit'' and the >> connection then stalls. > > Just to make sure I understand this correctly - do you mean that > INVALID packets are "counted" as NEW packets as far as the limit is > concerned?Sort of. Shorewall sends both INVALID and NEW packets through the rules generated by /etc/shorewall/rules. This gives users the choice of adding ''allowInvalid'' and ''dropInvalid'' rules to control INVALID separately from NEW. If you only want to deal with ''NEW'' connections in your rules file, then just add this as the first entry in the file: dropInvalid all all In the absence of any acceptInvalid/dropInvalid rules, then any rule in your /etc/shorewall/rules applies equally to NEW and INVALID packets. In many cases, this approach allows Shorewall to mask problems like you are experiencing. It doesn''t mask those problems when rate limiting is applied, however. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Jonathan Underwood wrote:> > So if I understand you (Tom) correctly, this is an issue with the > network the server is on, or the firewall it is behind that is causing > problems with certain server firewall configurations? [Just in case > that''s a confusing sentence, here we have been debugging a shorewall > generated firewall, but the machine that firewall is on is behind a > LAN firewall allowing ssh connections through to the server in > question] > > I realize at this point it is probably clear that this is no bug with > shorewall, but I wonder if you might have any suggestion about how I > would go about finding what is causing all of those INVALID packets.The problem is caused by ''out-of-window'' packets. So to totally analyze the problem, you may have to capture: a) The SCP stream on the outer interface of the other firewall. b) The SCP stream on the outer interface of the Shorewall box. c) Invalid connection state packets (I sent instructions earlier). Once you find out which packets are being dropped (c), then you can compare those packets in (a) and (b) to see if the other firewall is mangling them. If not, then you need to send (b) and (c) to the netfilter developers for analysis. tcp_be_liberal (which you are setting) turns off Netfilter window-tracking except for RST packets so you will need to turn of tcp_be_liberal while tracking this down. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
On 27/05/07, Tom Eastep <teastep@shorewall.net> wrote:> The problem is caused by ''out-of-window'' packets. So to totally analyze > the problem, you may have to capture: > > a) The SCP stream on the outer interface of the other firewall. > b) The SCP stream on the outer interface of the Shorewall box. > c) Invalid connection state packets (I sent instructions earlier). > > Once you find out which packets are being dropped (c), then you can > compare those packets in (a) and (b) to see if the other firewall is > mangling them. If not, then you need to send (b) and (c) to the > netfilter developers for analysis.OK, thanks for the info. Alas, (a) isn''t accesible to me, but I''ll spend some time seeing if I can see anything suspect in (b).> > tcp_be_liberal (which you are setting) turns off Netfilter > window-tracking except for RST packets so you will need to turn of > tcp_be_liberal while tracking this down. >OK. Thanks once again for your help and patience. Jonathan. ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
Jonathan Underwood wrote:> On 27/05/07, Tom Eastep <teastep@shorewall.net> wrote: >> The problem is caused by ''out-of-window'' packets. So to totally analyze >> the problem, you may have to capture: >> >> a) The SCP stream on the outer interface of the other firewall. >> b) The SCP stream on the outer interface of the Shorewall box. >> c) Invalid connection state packets (I sent instructions earlier). >> >> Once you find out which packets are being dropped (c), then you can >> compare those packets in (a) and (b) to see if the other firewall is >> mangling them. If not, then you need to send (b) and (c) to the >> netfilter developers for analysis. > > OK, thanks for the info. Alas, (a) isn''t accesible to meYou could also capture on the client system. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/