openssh unix dev - Oct 2025 - IPQoS EF packets blocked by some provider(s) / Login to OpenSSH >= 10.1 servers stuck in preauth

Dear fellow OpenSSH friends,

this is not so much a bug report, but a troubleshooting report in the hope it
might be helpful to others using OpenSSH in a similar environment.


TL;DR: Some providers seem to block incoming packets with the QoS marking
"Expedited Forwarding", effectively blocking login to OpenSSH servers
>= 10.1 with default configuration.
        "IPQoS none" in sshd_config (server-side) works around this.


Full story:
I recently upgraded my first server to OpenSSH 10.1_p1 (and subsequently
10.2_p1) and found that starting with 10.1_p1 I was unable to log in from my
home network (neither via IPv4 nor IPv6), independent of the client version.
From other network locations, it worked fine, even carrying a mobile client from
home into another network.

Checking debug logs, my client at home got stuck right before receiving the
banner from the server, and the server session got stuck in pre-auth.
tcpdumping / Wiresharking revealed that the packet with the banner and the first
KEX packet sent by the server did not arrive with the client. Neither the
original packets nor subsequent retransmissions arrived, leading to a
"connection stuck in pre-auth".

As I did not have an immediate idea on which change caused this (network packets
looked "similar enough" between 10.0 and later), I did a bisect and
found that the new IPQoS functionality (
https://github.com/openssh/openssh-portable/commit/289239046b2c4b0076c14394ae9703a879e78706
) triggered this.
And indeed, I could confirm that the packets which never arrived at my client
had Expedited Forwarding / EF set, i.e. DSCP 46.
I could have grasped that new feature from the release notes, but sometimes you
have to learn the hard way...

Indeed, setting:
  IPQoS none
in sshd_config on the server and restarting it fixes things for me. I can again
log in from home to this (self-managed) SSH server, but of course I will remain
locked out from other > 10.0 servers in my current setup.

And indeed, running:
  ping -Q 184
(this is 46 << 2) from "outside" against the WAN IP of my CPE /
home router (directly connected to a fibre) yields full packet loss.
tcpdumping on the external interface of the router shows that the packets do not
arrive there, and I tried from various locations (which can ping each other that
way), so I deduce my provider is silently dropping these packets (in my case, I
have CG-NAT for IPv4 and native IPv6, they drop this for both, while of course
for IPv4 I can only test via a tracked connection through their NAT).

Since my provider serves >1.5 million customers here in Germany, I presume
this might hit others out there (maybe even with other providers). Sadly,
getting through their first level support is almost impossible, so I have only a
small hope of them fixing it.
In case it hits others, I do hope this troubleshooting here may prove useful to
get a temporary workaround or maybe help others to escalate to their provider in
case this is possible for them.


I'm not sure OpenSSH can improve on this situation. Reading through RFC 3246
section 3 [0], maybe it should, as it states:
  "If two adjacent domains have not negotiated an EF rate, the downstream
domain SHOULD use 0 as the rate (i.e., drop all EF marked packets)."
This could indicate that this might happen in other cases, which then
essentially blocks interactive SSH in default server configurations.


Cheers and hope this helps,
	Oliver

[0] https://www.rfc-editor.org/rfc/rfc3246#section-3

Hi,

On Mon, Oct 13, 2025 at 09:56:36PM +0200, Oliver Freyermuth via openssh-unix-dev
wrote:
> Since my provider serves >1.5 million customers here in Germany, I
presume this might hit others out there (maybe even with other providers).
Sadly, getting through their first level support is almost impossible, so I have
only a small hope of them fixing it.
Name and shame... so who is this?  There's enough people on the list
that have good ties into the ISP world, so maybe the message can be 
relayed to "proper level support" :-)

gert
-- 
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             gert at
greenie.muc.de