freebsd stable - Aug 2020 - running out of ports: every client port is used only once in outgoing connection

Hello,


I have a situation where I'm running out of client ports on a huge 
reverse-proxy.

Say I have an nginx upstream like this:

upstream geoplatform {
 ??????? hash $hashkey consistent;
 ??????? server 127.0.0.1:4079 fail_timeout=10s;
 ??????? server 127.0.0.1:4080 fail_timeout=10s;
 ??????? server 10.100.34.5:4079 fail_timeout=10s;
 ??????? server 10.100.34.5:4080 fail_timeout=10s;
 ??????? server 10.100.34.7:4079 fail_timeout=10s;
 ??????? server 10.100.34.7:4080 fail_timeout=10s;
 ??????? server 10.100.34.8:4079 fail_timeout=10s;
 ??????? server 10.100.34.8:4080 fail_timeout=10s;
}

And as soon as I'm switching to it from DNS RR I'm starting to get get 
"Can't assign outgoing address when connecting to ...". The usual 
approach would be to assign multiple IP aliases to the destination 
backends, so I will get more of socket tuples. So I did this:

upstream geoplatform {
 ??????? hash $hashkey consistent;
 ??????? server 127.0.0.1:4079 fail_timeout=10s;
 ??????? server 127.0.0.1:4080 fail_timeout=10s;
 ??????? server 127.0.0.2:4079 fail_timeout=10s;
 ??????? server 127.0.0.2:4080 fail_timeout=10s;
 ??????? server 127.0.0.3:4079 fail_timeout=10s;
 ??????? server 127.0.0.3:4080 fail_timeout=10s;
 ??????? server 10.100.34.5:4079 fail_timeout=10s;
 ??????? server 10.100.34.5:4080 fail_timeout=10s;
 ??????? server 10.100.33.8:4079 fail_timeout=10s;
 ??????? server 10.100.33.8:4080 fail_timeout=10s;
 ??????? server 10.100.33.9:4079 fail_timeout=10s;
 ??????? server 10.100.33.9:4080 fail_timeout=10s;
 ??????? server 10.100.33.10:4079 fail_timeout=10s;
 ??????? server 10.100.33.10:4080 fail_timeout=10s;
 ??????? server 10.100.34.7:4079 fail_timeout=10s;
 ??????? server 10.100.34.7:4080 fail_timeout=10s;
 ??????? server 10.100.34.8:4079 fail_timeout=10s;
 ??????? server 10.100.34.8:4080 fail_timeout=10s;
 ??????? server 10.100.34.10:4079 fail_timeout=10s;
 ??????? server 10.100.34.10:4080 fail_timeout=10s;
 ??????? server 10.100.34.11:4079 fail_timeout=10s;
 ??????? server 10.100.34.11:4080 fail_timeout=10s;
 ??????? server 10.100.34.12:4079 fail_timeout=10s;
 ??????? server 10.100.34.12:4080 fail_timeout=10s;

}

Surprisingly, this didn't work. So... I just checked if I really have 
that much of connections. Seems like I'm starting to get troubles on 
130K of connections, but even on the initial upstream configuration I 
should be able to handle 65535 - 10K (since net.inet.ip.portrange.first 
is 10K) = 55535, 55535 * 8 ~ 450K of connections. Looks like the client 
port is not reused at all in socket tuples !

Indeed it does not: the below line is taken when there's no free ports, 
since the nearby console window is flooded with "Can't assign requested
address", so I assume I should already have 10.100.34.6.57026 (local 
IP-port pair) used in as many connection, as many servers I have. But it 
occurs only once:

# netstat -an | grep 10.100.34.6.57026
tcp4?????? 0????? 0 10.100.34.6.57026????? 10.100.34.5.4079 ESTABLISHED

[root at geo2ng:vhost.d/balancer]#

Second test: lets count how many times each port is used in netstat -an:


# netstat -an -p tcp | grep -v LISTEN | grep 10.100 | awk '{print $4}' |
sort | uniq -c | more | grep -v 1\

 ?(none)

So, seems like FreeBSD isn't reusing client ports out-of-the-box.

Linux, on the other hand, does reuse ports for client connection, as 
long as the socket tuple stays unique. How do I get the same behavior on 
FreeBSD ?


Thanks.

Eugene.

Hello,

27.08.2020 23:01, Eugene M. Zheganin wrote:
>
> And as soon as I'm switching to it from DNS RR I'm starting to get
get
> "Can't assign outgoing address when connecting to ...". The
usual
> approach would be to assign multiple IP aliases to the destination 
> backends, so I will get more of socket tuples.
> So, seems like FreeBSD isn't reusing client ports out-of-the-box.
>
> Linux, on the other hand, does reuse ports for client connection, as 
> long as the socket tuple stays unique. How do I get the same behavior 
> on FreeBSD ?
>
>
>
Found this:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=174087

and this:

https://svnweb.freebsd.org/base?view=revision&revision=361228

How do I determine if the latter is merged into STABLE (is it ?) ?


Thanks.

Eugene.