Hello I got a FreeBSD 7 machine running mail services (among other things). This machine recently replaced a FreeBSD 6.2 machine doing the same tasks. Now and then I need to send alot of mail to customers (mailing list), and one thing i've noticed now after the change is that when I use a lot of connections subsequently (high connection rate, even if they are very shortlived) inside a jail (dunno if that has anything to do with it though), I start to get Operation not permitted in return to connect(). I've seen this in the PHP app that sends mail, when it tried to connect to localhost, as well as from postfix when it have been trying to connect to amavisd on localhost, but also from postfix when it has tried to connect to remote SMTP servers. I do have PF for filtering, but there are no max-src-conn-rate limits enabled for any rules that is used for this. However, from one of the jail I do have a hfsc queue limiting the outgoing mail traffic from one jailed IP. But I'm not sure that this would be the problem, since I've also seen the problem when doing localhost connects in the jail, and also in other jails on an entierly different IP that is not affected. Does anyone have any clues about what I can look at and tune to fix this? Thanks! -- Johan Str?m Stromnet johan@stromnet.se http://www.stromnet.se/
On Sat, May 17, 2008 at 04:33:20PM +0200, Johan Str?m wrote:> Hello > > I got a FreeBSD 7 machine running mail services (among other things). This > machine recently replaced a FreeBSD 6.2 machine doing the same tasks. > Now and then I need to send alot of mail to customers (mailing list), and > one thing i've noticed now after the change is that when I use a lot of > connections subsequently (high connection rate, even if they are very > shortlived) inside a jail (dunno if that has anything to do with it > though), I start to get Operation not permitted in return to connect(). > I've seen this in the PHP app that sends mail, when it tried to connect to > localhost, as well as from postfix when it have been trying to connect to > amavisd on localhost, but also from postfix when it has tried to connect to > remote SMTP servers. > > I do have PF for filtering, but there are no max-src-conn-rate limits > enabled for any rules that is used for this. However, from one of the jail > I do have a hfsc queue limiting the outgoing mail traffic from one jailed > IP. But I'm not sure that this would be the problem, since I've also seen > the problem when doing localhost connects in the jail, and also in other > jails on an entierly different IP that is not affected. > > Does anyone have any clues about what I can look at and tune to fix this?Operation not permitted is most commonly seen on machines using pf(4), where there are rules blocking certain outbound traffic. I believe this has nothing to do with max-src-conn-rate. Chances are some of your pf(4) rules are wrong. There is also the possibility that jails are causing your problem. I have no experience with jails, so I cannot comment on that. I'd consider re-posting your problem to freebsd-pf@freebsd.org, and include your entire pf ruleset, so people could analyse it. Output from "pfctl -s info" would also be benefitial. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Hi Johan and List,
In my case a few months ago it was pahu. Don't give that fine fellow an
account on your precious system !
But seriously, I had a pf-firewalled jail being being used for DNS
testing, with large numbers of udp "connections" hanging around in pf
state. While the default udp timeout settings in PF are lower than those
of the tcp timeouts, it is was still too high for it to to remove the
states in time before hitting the default 10k state limit!
If this is the case with you - run 'pfctl -s state | wc -l' - when there
is traffic load you may see that hitting 10k states if you've not tuned
that variable.
What to do next - up the state limit or lower the state timeouts. I did
both, to be safe.
in /etc/pf.conf these must be at the very top of the file:
# options
# 10k is insanely low, lets raise it..
set limit { frags 16384, states 32768 }
# timeouts - see 'pfctl -s timeouts' for options - you will want to
# change the tcp ones rather than the udp ones for your smtp setup.
# but these are mine, I set them for the dns traffic.
set timeout { udp.first 15, udp.single 5, udp.multiple 30 }
don't forget to:
$ /etc/rc.d/pf check && ?/etc/rc.d/pf reload
HTH,
Alex
On Sat, 2008-05-17 at 16:33 +0200, Johan Str?m wrote:> Hello
>
> I got a FreeBSD 7 machine running mail services (among other things).
> This machine recently replaced a FreeBSD 6.2 machine doing the same
> tasks.
> Now and then I need to send alot of mail to customers (mailing list),
> and one thing i've noticed now after the change is that when I use a
> lot of connections subsequently (high connection rate, even if they
> are very shortlived) inside a jail (dunno if that has anything to do
> with it though), I start to get Operation not permitted in return to
> connect().
> I've seen this in the PHP app that sends mail, when it tried to
> connect to localhost, as well as from postfix when it have been trying
> to connect to amavisd on localhost, but also from postfix when it has
> tried to connect to remote SMTP servers.
>
> I do have PF for filtering, but there are no max-src-conn-rate limits
> enabled for any rules that is used for this. However, from one of the
> jail I do have a hfsc queue limiting the outgoing mail traffic from
> one jailed IP. But I'm not sure that this would be the problem, since
> I've also seen the problem when doing localhost connects in the jail,
> and also in other jails on an entierly different IP that is not
> affected.
>
> Does anyone have any clues about what I can look at and tune to fix
> this?
>
> Thanks!
>
> --
> Johan Str?m
> Stromnet
> johan@stromnet.se
> http://www.stromnet.se/
>
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080517/32b9a1b0/attachment.pgp