freebsd stable - Oct 2020 - pf and hnX interfaces

Hello,

I'm running a FreeBSD 12.1 server as a VM under Hyper-V. And although 
this letter will make an impression of another lame post blaming FreeBSD 
for all of the issues while the author should blame himselm, I'm atm out 
of another explanation. The thing is: I'm getting loads of sendmail 
errors like:


===Cut==
Oct 13 13:49:33 gw1 sm-mta[95760]: 09D8mN2P092173: SYSERR(root): 
putbody: write error: Permission denied
Oct 13 13:49:33 gw1 sm-mta[95760]: 09D8mN2P092173: SYSERR(root): timeout 
writing message to <whatever>.mail.protection.outlook.com.: Permission 
denied

===Cut==

The relay address is just random. The thing is, I can successfully 
connect to it via telnet. Even send some commands. But when this is done 
by senamil - and when it's actually sending messages, I get random 
errors. Firstly I was blaming myself and trying to get the rule that 
actually blocks something. I ended up having none of the block rules 
without log clause, and in the same time tcpdump -netti pflog0 shows no 
droppen packets, but sendmail still eventually complains.

If it matters, I have relatively high rps on this interface, about 25 Kpps.

I've also found several posting mentionsing that hnX is badly handling 
the TSO and LRO mode, so I switched it off. No luck however, with 
vlanhwtag and vlanmtu, which for some reason just cannot be switched 
off. the if_hn also lacks a man page for some reason, so it's unclear 
how to tweak it right.

And the most mysterious part? - when I switch the pf off, the errors 
stops to appear. This would clearly mean that pf blocks some packets, 
but then again, this way the pflog0 would show them up, right (and yes - 
it's "UP" )?

Is there some issue with pf and hn interfaces that I'm unaware about?

Are these symptoms of a bug ?


Thanks.

Eugene.

Hi,

Am 13.10.20 um 10:58 schrieb Eugene M. Zheganin:
> 
> And the most mysterious part? - when I switch the pf off, the errors
> stops to appear. This would clearly mean that pf blocks some packets,
> but then again, this way the pflog0 would show them up, right (and yes -
> it's "UP" )?
maybe you run into state limits?

Try pfctl -si and look into the value for "memory". If this is greater
than 0, you have exausted the global state limit some time. "PF states
limit reached" is also logged in the system message buffer, then.

regards,
Robert Schulze

On 13 Oct 2020, at 10:58, Eugene M. Zheganin wrote:
> I'm running a FreeBSD 12.1 server as a VM under Hyper-V. And although 
> this letter will make an impression of another lame post blaming 
> FreeBSD for all of the issues while the author should blame himselm, 
> I'm atm out of another explanation. The thing is: I'm getting loads
of
> sendmail errors like:
>
>
> ===Cut==>
> Oct 13 13:49:33 gw1 sm-mta[95760]: 09D8mN2P092173: SYSERR(root): 
> putbody: write error: Permission denied
> Oct 13 13:49:33 gw1 sm-mta[95760]: 09D8mN2P092173: SYSERR(root): 
> timeout writing message to <whatever>.mail.protection.outlook.com.: 
> Permission denied
>
> ===Cut==>
A ?Permission denied? on outbound packets can indeed happen when pf 
decides to block the packet.
> The relay address is just random. The thing is, I can successfully 
> connect to it via telnet. Even send some commands. But when this is 
> done by senamil - and when it's actually sending messages, I get 
> random errors. Firstly I was blaming myself and trying to get the rule 
> that actually blocks something. I ended up having none of the block 
> rules without log clause, and in the same time tcpdump -netti pflog0 
> shows no droppen packets, but sendmail still eventually complains.
>
> If it matters, I have relatively high rps on this interface, about 25 
> Kpps.
>
> I've also found several posting mentionsing that hnX is badly handling 
> the TSO and LRO mode, so I switched it off. No luck however, with 
> vlanhwtag and vlanmtu, which for some reason just cannot be switched 
> off. the if_hn also lacks a man page for some reason, so it's unclear 
> how to tweak it right.
>
While it?s possible that there are issues with TSO/LRO those 
wouldn?t look like this. (As an aside, I am interested in any 
reproducible setups where pf has issues with TSO/LRO. As far as I?ve 
been able to see all such issues have been resolved.)
> And the most mysterious part? - when I switch the pf off, the errors 
> stops to appear. This would clearly mean that pf blocks some packets, 
> but then again, this way the pflog0 would show them up, right (and yes 
> - it's "UP" )?
>
It?s possible for pf to drop packets without triggering log rules. For 
example, if pf decides to drop the packet before it matches any rule 
(e.g. it?s a corrupt packet) it won?t show up in pflog.
> Is there some issue with pf and hn interfaces that I'm unaware about?
>
There?s no interface specific code in pf, so it wouldn?t be specific 
to hn interfaces.
> Are these symptoms of a bug ?
>
Perhaps. It can also be a symptom of resource exhaustion.
Are there any signs of memory allocation failures, or incrementing error 
counters (in netstat or in pfctl)?

Best regards,
Kristof