openssh bugs - Oct 2025 - [Bug 3872] New: Interactive input excessively buffered when using a mux socket if ControlPersist is set

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

            Bug ID: 3872
           Summary: Interactive input excessively buffered when using a
                    mux socket if ControlPersist is set
           Product: Portable OpenSSH
           Version: 10.0p2
          Hardware: Other
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: grawity at gmail.com

With 10.0, connecting to an SSH server with a mux socket active results
in all interactive input being *heavily* buffered. For example, hitting
'Enter' once shows no response; hitting 'Enter' a second time
sends
both keypresses. Sometimes 3-4 keypresses are buffered up.

But this only occurs if ControlPersist is set to something other than
'no'. (I have it set to "5s" in my config, due to historical
reasons.)

good: -S none
good: -S /tmp/sock -o ControlPersist=no
bad: -S /tmp/sock -o ControlPersist=5s

Git bisect points towards 289239046b2c "upstream: Make ssh(1) and
sshd(8) set IP QoS (aka IP_TOS, IPV6_TCLASS)". I'm pretty sure it is
not the usual "DSCP intolerant network" issue since it also happens
with `ssh localhost`.

Though just in case, I tried "-o IPQoS=none" which is documented as
using the OS default, and got:

debug2: client_loop: session QoS is now interactive
debug2: fd 3 setting TCP_NODELAY
debug1: set_sock_tos: invalid TOS 2147483647

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

brunni at netestate.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |brunni at netestate.de

--- Comment #1 from brunni at netestate.de ---
I have the same problem.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

lfleischer at lfos.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lfleischer at lfos.de

--- Comment #2 from lfleischer at lfos.de ---
Several members of Arch Linux staff are impacted too. A downstream bug
report can be found here [1]. I believe this only happens with 10.1
(and not 10.0 as suggested in the original report).

[1]
https://gitlab.archlinux.org/archlinux/packaging/packages/openssh/-/issues/20

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #3 from brunni at netestate.de ---
I have the problem with 10.1p1 (upgraded from 9.9p2). Linux Kernel
5.10.245, glibc 2.42 and gcc 12.5.0.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #4 from Damien Miller <djm at mindrot.org> ---
Created attachment 3904
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3904&action=edit
unconditionally set nodelay

Please attach a full debug trace from a working vs affected session.

Also, please try the attached patch (after attaching the traces)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

selmo2000 at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |selmo2000 at gmail.com

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Christian Hesse <mail at eworm.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mail at eworm.de

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #5 from Mantas M. [grawity] <grawity at gmail.com> ---
Created attachment 3905
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3905&action=edit
ssh -vvv (good commit before IPQoS change)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #6 from Mantas M. [grawity] <grawity at gmail.com> ---
Created attachment 3906
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3906&action=edit
ssh -vvv (bad release commit)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #7 from Mantas M. [grawity] <grawity at gmail.com> ---
The patch doesn't seem to make any difference (for me at least). It
seems that NODELAY was already being set.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3905|text/x-log                  |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3906|text/x-log                  |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #8 from Darren Tucker <dtucker at dtucker.net> ---
Minimal repro case derived from the Arch bug tracker (in this case, on
Fedora):

$ rm -f /tmp/newsocket; time ssh -o ControlMaster=auto -o
ControlPersist=60s -o ControlPath=/tmp/newsocket -t -- yourserver true

real    0m5.566s
user    0m0.002s
sys     0m0.004s

The timing is dependent on how long you wait to hit enter a couple of
times.  Without the -t it does not exhibit the problem.

My first guess was ObscureKeystrokeTiming, but setting it to "no"
doesn't seem to help.  My second guess, since it takes two hits on
enter, was EscapeChar processing, but setting that to "none"
doesn't
help either.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #9 from Damien Miller <djm at mindrot.org> ---
I think the problem is that reads from the client tty are blocking in
channels.c:channel_handle_rfd()

I'd expect reads from the TTY to be non-blocking, but that doesn't seem
to be happening. No idea why yet and I don't understand how
289239046b2c could cause this.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #10 from Darren Tucker <dtucker at dtucker.net> ---
Given there's a reproducer and a relatively small range of changes, I
bisected it down to one of the following (the SIGINFO changes don't
compile until the ifdef wrapping):

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
dc5147028ff19213a32281dad07bba02e58da3fa
289239046b2c4b0076c14394ae9703a879e78706
f807a598c96be683d97810481e954ec9db6b0027
b7ee13fbbb4ebafcf71f29685f053ecb97d1bcef
We cannot bisect more!

commit b7ee13fbbb4ebafcf71f29685f053ecb97d1bcef
Author: Damien Miller <djm at mindrot.org>
Date:   Mon Aug 18 14:22:18 2025 +1000

    wrap SIGINFO in ifdef

commit 289239046b2c4b0076c14394ae9703a879e78706 (HEAD)
Author: djm at openbsd.org <djm at openbsd.org>
Date:   Mon Aug 18 03:43:01 2025 +0000

    upstream: Make ssh(1) and sshd(8) set IP QoS (aka IP_TOS,
IPV6_TCLASS)

    continually at runtime based on what sessions/channels are open.

    Previously, ssh(1) and sshd(8) would pick a QoS value when they
    were started and use it for the whole connection. This could
    produce suboptimal choices for the QoS value, e.g. for multiplexed
    sessions that started interactive but picked up a sftp client,
    or sessions that moved large amounts of data via port forwarding.

    Now the QoS value will change to the non-interactive IPQoS whenever
    a "non-interactive" channel is open; basically any channel that
lacks
    a tty other than agent forwarding.

    This is important now that the default interactive IPQoS is EF
    (Expedited Forwarding), as many networks are configured to allow
    only relatively small amounts of traffic of this class and they
will
    aggressively deprioritise the entire connection if this is
exceeded.

    NB. because ssh(1) and sshd(8) now change IP_TOS/IPV6_TCLASS
    continually via setsockopt(), this commit requires a recent
pledge(2)
    change that landed recently in the OpenBSD kernel. Please ensure
    you have updated to a kernel from within the last two weeks before
    updating OpenSSH.

    with job@ deraadt@

    OpenBSD-Commit-ID: 325fc41717eecdf5e4b534bfa8d66817425b840f

commit dc5147028ff19213a32281dad07bba02e58da3fa
Author: djm at openbsd.org <djm at openbsd.org>
Date:   Mon Aug 18 03:29:11 2025 +0000

    upstream: SIGINFO handler for sshd(8) to dump active

    channels/sessions ok deraadt@

    OpenBSD-Commit-ID: 9955cb6d157c6d7aa23a819e8ef61b1edabc8b7d

commit f807a598c96be683d97810481e954ec9db6b0027
Author: djm at openbsd.org <djm at openbsd.org>
Date:   Mon Aug 18 03:28:36 2025 +0000

    upstream: SIGINFO handler for ssh(1) to dump active

    channels/sessions ok deraadt@

    OpenBSD-Commit-ID: 12f88a5044bca40ef5f41ff61b1755d0e25df901

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned-bugs at mindrot.org |djm at mindrot.org

--- Comment #11 from Damien Miller <djm at mindrot.org> ---
Created attachment 3907
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3907&action=edit
don't overload c->isatty

I think this fixes the problem.

The root cause was overloading the channel c->isatty variable.

Previously this meant "this channel is connected to a TTY" but
289239046b2c overloaded this to also mean "this channel's remote side
is connected to a PTY".

Unfortunately, c->isatty has some side effects. This includes sometimes
forcing reads on the local ttyfd even of poll() tells us the fd isn't
ready.

We only ever want to do that in sshd when we're on the ptymaster side;
doing in the client caused this problem.

The attached patch uses a separate channel struct member to track the
remote PTY status instead of c->isatty. AFAIK it should fix this bug.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #12 from Mantas M. [grawity] <grawity at gmail.com> ---
(In reply to Darren Tucker from comment #10)
> Given there's a reproducer and a relatively small range of changes,
> I bisected it down to one of the following (the SIGINFO changes
> don't compile until the ifdef wrapping):
I did a s/SIGINFO/SIGBUS/g to get those to build during bisect, in
order to narrow it down to the single commit.

(In reply to Damien Miller from comment #11)
> Created attachment 3907 [details]
> don't overload c->isatty
This patch seems to work here.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

--- Comment #13 from Damien Miller <djm at mindrot.org> ---
attachment 3907 has been committed and cherrypicked to the V_10_1
stable branch.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net
             Blocks|                            |3876


Referenced Bugs:

https://bugzilla.mindrot.org/show_bug.cgi?id=3876
[Bug 3876] Tracking bug for OpenSSH 10.2
-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

wgh at torlan.ru changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wgh at torlan.ru

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Sam James <sam at gentoo.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo.org

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

fzacarias3k at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fzacarias3k at gmail.com

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #14 from Damien Miller <djm at mindrot.org> ---
This has been committed and is in openssh-10.2 - thanks

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3872

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jacobenders1 at gmail.com

--- Comment #15 from Darren Tucker <dtucker at dtucker.net> ---
*** Bug 3890 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.