openssh unix dev - Aug 2018 - sshd 7.8p1 close connection from VMware Fusion NAT Port Forwarding

On Mon, 27 Aug 2018, Stuart Henderson wrote:
> On 2018-08-27, Zach Cheung <kuroro.zhang at gmail.com> wrote:
> > After upgrading my VMware Fusion (10.1.3) Arch Guest to the latest
with
> > OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS Sierra
> > (10.12.6) host to Arch guest via local NAT port forwarding failed, but
via
> > Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed the
> > problem.
> >
> > Any idea about this bug?
> 
> I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
Do you have any insight into what is breaking here? I don't believe
changing the default DSCP values should break connections...

-d

On Tue, 28 Aug 2018 at 06:24, Damien Miller <djm at mindrot.org> wrote:
> On Mon, 27 Aug 2018, Stuart Henderson wrote:
>
> > On 2018-08-27, Zach Cheung <kuroro.zhang at gmail.com> wrote:
> > > After upgrading my VMware Fusion (10.1.3) Arch Guest to the
latest with
> > > OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS
Sierra
> > > (10.12.6) host to Arch guest via local NAT port forwarding
failed, but
> via
> > > Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed
the
> > > problem.
> > >
> > > Any idea about this bug?
> >
> > I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
>
> Do you have any insight into what is breaking here? I don't believe
> changing the default DSCP values should break connections...


I suspect VMWare Fusion has a very broken NAT implementation, where they
seem to hash packets to identify flows on (part of) the DSCP field.

Kind regards,

Job

On 2018/08/28 14:17, Damien Miller wrote:
> On Mon, 27 Aug 2018, Stuart Henderson wrote:
>
> > On 2018-08-27, Zach Cheung <kuroro.zhang at gmail.com> wrote:
> > > After upgrading my VMware Fusion (10.1.3) Arch Guest to the
latest with
> > > OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from macOS
Sierra
> > > (10.12.6) host to Arch guest via local NAT port forwarding
failed, but via
> > > Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1 fixed
the
> > > problem.
> > >
> > > Any idea about this bug?
> >
> > I bet it is the QoS change. Try "IPQoS lowdelay,throughput".
>
> Do you have any insight into what is breaking here? I don't believe
> changing the default DSCP values should break connections...
I think it's probably a NAT bug in VMware Fusion. tcpdump might
give more clues as to how it's broken (maybe it's mangling packets,
maybe it's just rejecting them) but actually fixing it would need
VMware's involvement.

Short description: OpenSSH 7.8 started marking packets with DSCP
(af21 for interactive, cs1 for bulk) instead of IP TOS ("lowdelay"
for interactive, "throughput" for bulk). VMware Fusion with NAT
port-forwarding to sshd in the guest fails with OpenSSH 7.8.
It should be possible to replicate this failure with older OpenSSH
(6.0 or newer) by using "IPQoS af21 cs1" in sshd_config in the guest.

Unless any VMware people are reading this, it's probably best if one
of their customers reports it as a bug, I can't imagine it would be
that complicated to fix, the problem will be getting the report past
front-line support and on to the right person.

confirmed that sshd 7.7p1 with "IPQoS af21 cs1" also closed
connection.

On Tue, Aug 28, 2018 at 6:19 PM Stuart Henderson <stu at spacehopper.org>
wrote:
> On 2018/08/28 14:17, Damien Miller wrote:
> > On Mon, 27 Aug 2018, Stuart Henderson wrote:
> >
> > > On 2018-08-27, Zach Cheung <kuroro.zhang at gmail.com>
wrote:
> > > > After upgrading my VMware Fusion (10.1.3) Arch Guest to the
latest
> with
> > > > OpenSSH upgraded from 7.7p1 to 7.8p1, found that ssh from
macOS
> Sierra
> > > > (10.12.6) host to Arch guest via local NAT port forwarding
failed,
> but via
> > > > Arch LAN IP worked, downgraded OpenSSH from 7.8p1 to 7.7p1
fixed the
> > > > problem.
> > > >
> > > > Any idea about this bug?
> > >
> > > I bet it is the QoS change. Try "IPQoS
lowdelay,throughput".
> >
> > Do you have any insight into what is breaking here? I don't
believe
> > changing the default DSCP values should break connections...
>
> I think it's probably a NAT bug in VMware Fusion. tcpdump might
> give more clues as to how it's broken (maybe it's mangling packets,
> maybe it's just rejecting them) but actually fixing it would need
> VMware's involvement.
>
> Short description: OpenSSH 7.8 started marking packets with DSCP
> (af21 for interactive, cs1 for bulk) instead of IP TOS
("lowdelay"
> for interactive, "throughput" for bulk). VMware Fusion with NAT
> port-forwarding to sshd in the guest fails with OpenSSH 7.8.
> It should be possible to replicate this failure with older OpenSSH
> (6.0 or newer) by using "IPQoS af21 cs1" in sshd_config in the
guest.
>
> Unless any VMware people are reading this, it's probably best if one
> of their customers reports it as a bug, I can't imagine it would be
> that complicated to fix, the problem will be getting the report past
> front-line support and on to the right person.
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>