thr3ads.net - freebsd stable - Weird problems with 'pf' (on both 5.x and 6.x) [Jul 2006]

If this information is useful, please help other people find it:
Share via:

Garance A Drosihn

2006-Jul-28 01:07 UTC

Weird problems with 'pf' (on both 5.x and 6.x)

It happens that I noticed two odd networking problems recently.
One of them is easily reproducible, and I have it tracked down
to one innocuous-looking line in my /etc/pf.conf.  The other
is a problem in a chat server that I run, with a few hundred
people on it, and is much more of a hassle to reproduce.  But
turning off 'pf' to solve the first problem seems to have
also solved the second problem, so I assume both problems
come from the same culprit.

Once I figured out how to reproduce the problem, it seems so
easy to reproduce that I find it odd that no one else has
run into it.  But I also do not notice any PR's that seemed
to describe the problem.  I'd appreciate it if people would
try to duplicate the problem on some other machines.

This problem has been seen on:
      5.x-stable as built on Mon Jul 24
      6.x-stable as built on Mon Jul 17
(as well as several earlier snapshots of both 5.x and 6.x).

I have a freebsd box which is the server for a print queue
named 'bill', and is running pf.  I have other machines which
reference that queue.  It seems that machines on the same
subnet as the server-box do not exhibit the problem.  But
for other machines, if I do 'lpq -Pbill' twice in rapid
succession, then the second one will hang.

After some futzing around, I determined that if my pf.conf
has only the lines:

     # Filtering: the implicit first two rules are
     #pass in all
     #pass out all

then I can do many many lpq's in a row, without any trouble.
But if I restart pf after adding these lines to pf.conf:

     #   Allow all outgoing tcp and udp connections and keep state
     pass out quick proto { tcp, udp } all keep state

then I have the problem where the second 'lpq' from a remote
host will hang, if it is done right after the first one.
That's right.  I add a rule which just does "quick passing"
for *outbound* connections, and somehow that screws up
(blocks?) *incoming* connections.  I have no rules which
should block any packets at all, so my guess is that some
packets are getting lost, delayed, or corrupted somewhere.

-- 
Garance Alistair Drosehn            =   gad@gilead.netel.rpi.edu
Senior Systems Programmer           or  gad@freebsd.org
Rensselaer Polytechnic Institute    or  drosih@rpi.edu

Garance A Drosihn

2006-Jul-28 01:18 UTC

head link

Weird problems with 'pf' (on both 5.x and 6.x)

At 9:07 PM -0400 7/27/06, Garance A Drosihn wrote:>
>But if I restart pf after adding these lines to pf.conf:
>
>     #   Allow all outgoing tcp and udp connections and keep state
>     pass out quick proto { tcp, udp } all keep state
>
>then I have the problem where the second 'lpq' from a remote
>host will hang, if it is done right after the first one.
The client-machine which is doing the lpq is a solaris
machine, so here is the 'snoop' output from that side
of things.  Disclaimer:  I'm not a networking expert,
so I'm hoping someone else will find this a lot more
obvious than I do.

Here's the packets from the first 'lpq', with various
names changed to protect the innocent (and to reduce
the wrapping a little bit...):

________________________________
   1   0.00000 lpq-client -> print-serv ETHER Type=0800 (IP), size = 62 bytes
   1   0.00000 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=48, ID=13267
   1   0.00000 lpq-client -> print-serv TCP D=515 S=1023 Syn 
Seq=1503722122 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
   1   0.00000 lpq-client -> print-serv PRINTER C port=1023
________________________________
   2   0.00068 print-serv -> lpq-client ETHER Type=0800 (IP), size = 62 bytes
   2   0.00068 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=48, ID=4007
   2   0.00068 print-serv -> lpq-client TCP D=1023 S=515 Syn 
Ack=1503722123 Seq=1874442309 Len=0 Win=65535 Options=<mss 
1460,sackOK,eol>
   2   0.00068 print-serv -> lpq-client PRINTER R port=1023
________________________________
   3   0.00072 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
   3   0.00072 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=13268
   3   0.00072 lpq-client -> print-serv TCP D=515 S=1023 
Ack=1874442310 Seq=1503722123 Len=0 Win=24820
   3   0.00072 lpq-client -> print-serv PRINTER C port=1023
________________________________
   4   0.00088 lpq-client -> print-serv ETHER Type=0800 (IP), size = 63 bytes
   4   0.00088 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=49, ID=13269
   4   0.00088 lpq-client -> print-serv TCP D=515 S=1023 
Ack=1874442310 Seq=1503722123 Len=9 Win=24820
   4   0.00088 lpq-client -> print-serv PRINTER C port=1023 \3bill\n
________________________________
   5   0.03003 print-serv -> lpq-client ETHER Type=0800 (IP), size = 132
bytes
   5   0.03003 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=118, ID=4045
   5   0.03003 print-serv -> lpq-client TCP D=1023 S=515 
Ack=1503722132 Seq=1874442310 Len=78 Win=65535
   5   0.03003 print-serv -> lpq-client PRINTER R port=1023 Warning: bill is
________________________________
   6   0.03014 print-serv -> lpq-client ETHER Type=0800 (IP), size = 60 bytes
   6   0.03014 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=40, ID=4046
   6   0.03014 print-serv -> lpq-client TCP D=1023 S=515 Fin 
Ack=1503722132 Seq=1874442388 Len=0 Win=65535
   6   0.03014 print-serv -> lpq-client PRINTER R port=1023
________________________________
   7   0.03020 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
   7   0.03020 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=13270
   7   0.03020 lpq-client -> print-serv TCP D=515 S=1023 
Ack=1874442388 Seq=1503722132 Len=0 Win=24820
   7   0.03020 lpq-client -> print-serv PRINTER C port=1023
________________________________
   8   0.03022 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
   8   0.03022 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=13271
   8   0.03022 lpq-client -> print-serv TCP D=515 S=1023 
Ack=1874442389 Seq=1503722132 Len=0 Win=24820
   8   0.03022 lpq-client -> print-serv PRINTER C port=1023
________________________________
   9   0.03074 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
   9   0.03074 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=13272
   9   0.03074 lpq-client -> print-serv TCP D=515 S=1023 Fin 
Ack=1874442389 Seq=1503722132 Len=0 Win=24820
   9   0.03074 lpq-client -> print-serv PRINTER C port=1023
________________________________
  10   0.03132 print-serv -> lpq-client ETHER Type=0800 (IP), size = 60 bytes
  10   0.03132 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=40, ID=4047
  10   0.03132 print-serv -> lpq-client TCP D=1023 S=515 
Ack=1503722133 Seq=1874442389 Len=0 Win=65534
  10   0.03132 print-serv -> lpq-client PRINTER R port=1023
________________________________


and then here is the packets from the second 'lpq', done
right after the first one.  It looks like the problem is
in the initial handshaking to get the connection started:

________________________________
  11   7.19194 lpq-client -> print-serv ETHER Type=0800 (IP), size = 62 bytes
  11   7.19194 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=48, ID=13273
  11   7.19194 lpq-client -> print-serv TCP D=515 S=1023 Syn 
Seq=1505511645 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
  11   7.19194 lpq-client -> print-serv PRINTER C port=1023
________________________________
  12  10.55769 lpq-client -> print-serv ETHER Type=0800 (IP), size = 62 bytes
  12  10.55769 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=48, ID=13274
  12  10.55769 lpq-client -> print-serv TCP D=515 S=1023 Syn 
Seq=1505511645 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
  12  10.55769 lpq-client -> print-serv PRINTER C port=1023
________________________________
  13  17.30771 lpq-client -> print-serv ETHER Type=0800 (IP), size = 62 bytes
  13  17.30771 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=48, ID=13275
  13  17.30771 lpq-client -> print-serv TCP D=515 S=1023 Syn 
Seq=1505511645 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
  13  17.30771 lpq-client -> print-serv PRINTER C port=1023
________________________________
  14  30.80785 lpq-client -> print-serv ETHER Type=0800 (IP), size = 62 bytes
  14  30.80785 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=48, ID=56013
  14  30.80785 lpq-client -> print-serv TCP D=515 S=1023 Syn 
Seq=1505511645 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
  14  30.80785 lpq-client -> print-serv PRINTER C port=1023
________________________________
  15  57.80771 lpq-client -> print-serv ETHER Type=0800 (IP), size = 62 bytes
  15  57.80771 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=48, ID=56014
  15  57.80771 lpq-client -> print-serv TCP D=515 S=1023 Syn 
Seq=1505511645 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
  15  57.80771 lpq-client -> print-serv PRINTER C port=1023
________________________________
  16 111.80771 lpq-client -> print-serv ETHER Type=0800 (IP), size = 62 bytes
  16 111.80771 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=48, ID=56015
  16 111.80771 lpq-client -> print-serv TCP D=515 S=1023 Syn 
Seq=1505511645 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
  16 111.80771 lpq-client -> print-serv PRINTER C port=1023
________________________________
  17 111.80842 print-serv -> lpq-client ETHER Type=0800 (IP), size = 62 bytes
  17 111.80842 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=48, ID=4050
  17 111.80842 print-serv -> lpq-client TCP D=1023 S=515 Syn 
Ack=1505511646 Seq=3101688498 Len=0 Win=65535 Options=<mss 
1460,sackOK,eol>
  17 111.80842 print-serv -> lpq-client PRINTER R port=1023
________________________________
  18 111.80845 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
  18 111.80845 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=56016
  18 111.80845 lpq-client -> print-serv TCP D=515 S=1023 
Ack=3101688499 Seq=1505511646 Len=0 Win=24820
  18 111.80845 lpq-client -> print-serv PRINTER C port=1023
________________________________
  19 111.80868 lpq-client -> print-serv ETHER Type=0800 (IP), size = 63 bytes
  19 111.80868 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=49, ID=56017
  19 111.80868 lpq-client -> print-serv TCP D=515 S=1023 
Ack=3101688499 Seq=1505511646 Len=9 Win=24820
  19 111.80868 lpq-client -> print-serv PRINTER C port=1023 \3bill\n
________________________________
  20 111.83771 print-serv -> lpq-client ETHER Type=0800 (IP), size = 132
bytes
  20 111.83771 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=118, ID=4088
  20 111.83771 print-serv -> lpq-client TCP D=1023 S=515 
Ack=1505511655 Seq=3101688499 Len=78 Win=65535
  20 111.83771 print-serv -> lpq-client PRINTER R port=1023 Warning: bill is
________________________________
  21 111.83782 print-serv -> lpq-client ETHER Type=0800 (IP), size = 60 bytes
  21 111.83782 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=40, ID=4089
  21 111.83782 print-serv -> lpq-client TCP D=1023 S=515 Fin 
Ack=1505511655 Seq=3101688577 Len=0 Win=65535
  21 111.83782 print-serv -> lpq-client PRINTER R port=1023
________________________________
  22 111.83786 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
  22 111.83786 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=56018
  22 111.83786 lpq-client -> print-serv TCP D=515 S=1023 
Ack=3101688577 Seq=1505511655 Len=0 Win=24820
  22 111.83786 lpq-client -> print-serv PRINTER C port=1023
________________________________
  23 111.83787 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
  23 111.83787 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=56019
  23 111.83787 lpq-client -> print-serv TCP D=515 S=1023 
Ack=3101688578 Seq=1505511655 Len=0 Win=24820
  23 111.83787 lpq-client -> print-serv PRINTER C port=1023
________________________________
  24 111.83851 lpq-client -> print-serv ETHER Type=0800 (IP), size = 54 bytes
  24 111.83851 lpq-client -> print-serv IP  D=128.113.000.001 
S=128.113.002.002 LEN=40, ID=56020
  24 111.83851 lpq-client -> print-serv TCP D=515 S=1023 Fin 
Ack=3101688578 Seq=1505511655 Len=0 Win=24820
  24 111.83851 lpq-client -> print-serv PRINTER C port=1023
________________________________
  25 111.83911 print-serv -> lpq-client ETHER Type=0800 (IP), size = 60 bytes
  25 111.83911 print-serv -> lpq-client IP  D=128.113.002.002 
S=128.113.000.001 LEN=40, ID=4090
  25 111.83911 print-serv -> lpq-client TCP D=1023 S=515 
Ack=1505511656 Seq=3101688578 Len=0 Win=65534
  25 111.83911 print-serv -> lpq-client PRINTER R port=1023
________________________________

All I have to do is '/etc/rc.d/pf stop' on the print-server
machine, and immediately these long delays will go away.

-- 
Garance Alistair Drosehn            =   gad@gilead.netel.rpi.edu
Senior Systems Programmer           or  gad@freebsd.org
Rensselaer Polytechnic Institute    or  drosih@rpi.edu

freebsd stable - Jul 2006 - Weird problems with 'pf' (on both 5.x and 6.x)

Weird problems with 'pf' (on both 5.x and 6.x)

Weird problems with 'pf' (on both 5.x and 6.x)