openssh unix dev - Aug 2024 - Debian openssh option review: considering splitting out GSS-API key exchange

[I've CCed openssh-unix-dev for awareness, but set Mail-Followup-To to
just debian-devel and debian-ssh to avoid potentially spamming them with
a long discussion.  If you choose to override this then that's your
call, but please be mindful of upstream's time.]

Following the xz-utils backdoor, I'm reconsidering some choices in
Debian's OpenSSH packaging.  Please note that significant rearchitecture
of the upstream code is out of scope for the Debian packaging, so I'm
going to disregard comments of the form "maybe there should be a module
loader so all these things can just be plug-ins" or other such blue-sky
things; from my point of view this is just about configuring things a
bit more wisely within more or less our current constraints.


libsystemd
=========
This is the obvious thing on everyone's mind right now.  At the time I
merged that patch, "not NIHing code that's in a perfectly good
library"
seemed like a reasonable trade-off, but we do seem to have ended up on
the wrong side of history on this one.  There's work in progress to land
readiness protocol notification upstream without libsystemd (thanks
Damien and Luca!), and I expect to cherry-pick this into Debian once
it's agreed, so we'll get rid of that linkage and reduce our patch load
a bit.

We also have a patch from Ubuntu to support the systemd socket
activation protocol.  I've rewritten this to avoid using libsystemd, and
I'll submit it upstream once the readiness notification work is sorted
out.  But it's not particularly invasive once the libsystemd linkage is
removed, so it's not the end of the world if this ends up staying in our
patch queue.


GSS-API key exchange
===================
Way back in 2005, I merged the GSS-API key exchange patch into Debian's
main openssh package (https://bugs.debian.org/275472).  At the time it
seemed like a sensible overall reduction in maintenance burden (if I
remember correctly, the openssh-krb5 package often ended up lagging a
fair bit behind openssh).  While the patch is fairly large, it hasn't
generally been too hard to forward-port to newer versions of OpenSSH,
and Fedora carries it too so there's some sharing of work.

However, OpenSSH upstream has long rejected it, mainly on the basis that
they don't like adding new pre-authentication attack surface, and this
week seems like a good one to reconsider what patches we're shipping by
default.  gssapi.patch is the largest patch in our openssh package by an
order of magnitude, and easily the most intrusive in terms of complexity
and exposure, so I've somewhat regretted my choice to merge it a few
times over the years.

All the same, I'm aware that some people now depend on having this
facility in Debian's main openssh package: I get enough occasional bug
reports to convince me that it's still in use.  So, if I decide to split
it back out, I'd want to arrange for a somewhat graceful transition.
We've had it for nearly 20 years now, so we can take the time to do a
proper job that at least tries not to leave users in the lurch.

How does this rough plan sound?

 * for Debian trixie (current testing):

   * add dependency-only packages called something like
     openssh-client-gsskex and openssh-server-gsskex, depending on their
     non-gsskex alternatives
   * add NEWS.Debian entry saying that people need to install these
     packages if they want to retain GSS-API key exchange support
   * add release note saying the same

 * for Debian trixie+1 (or maybe after the next Ubuntu LTS, depending on
   exact timings):

   * add separate openssh-gsskex source package, carrying gssapi.patch
     in addition to whatever's in openssh, and whose binary packages
     Conflicts/Replaces/Provides the corresponding ones from openssh
   * add some kind of regular CI to warn about openssh-gsskex being out
     of date relative to openssh
   * drop gssapi.patch from openssh, except for small patches to
     configuration file handling to accept the relevant options with
     some kind of informative warning (compare
     https://bugs.debian.org/152657)

I guess we should decide whether the separate packages are to be needed
for GSS-API authentication as well as key exchange, because that affects
the choice of dependency-only package names in trixie.  If we only split
out gssapi.patch (for key exchange; sorry about the slightly misleading
name) but kept --with-kerberos5 (which also controls authentication),
then we'd significantly reduce our patch load but not sshd's linkage.

I've seen the suggestion of using libgssglue here
(https://fosstodon.org/@jas/112194876950058188).  That might be a good
idea and I have no particular objection to it, though I also don't know
much about it and it would probably be better if an expert did the work.
Perhaps it would make continuing to build the default variant using
--with-kerberos5 more palatable, since then the extra non-trivial
linkage would only affect people who turn on those options.


TCP wrappers
===========
We carry a patch to restore support for TCP wrappers, which was dropped
in OpenSSH 6.7 (October 2014); see
https://lists.mindrot.org/pipermail/openssh-unix-dev/2014-April/032497.html
and thread.  That wasn't long before the Debian 8 (jessie) freeze, and
so I patched it back in "temporarily", but then I dropped the ball on
organizing a proper transition.  libwrap links to libgssapi_krb5 (via
libnsl and libtirpc), so if we want to do a proper job of removing that
linkage then we'll have to finish this transition too.  This probably
means a similar timeline, with the addition that people will have to
make sure that they aren't relying on /etc/hosts.deny being effective
for sshd.

At the time, denyhosts was popular, but it was removed from Debian
several years ago.  I remember that, when I dealt with that on my own
systems, fail2ban seemed like the obvious replacement, and my impression
is that it's pretty widely used nowadays; it's very pluggable but it
normally works by adding firewall rules.  Are there any similar popular
systems left that rely on editing /etc/hosts.deny?

Fedora dropped libwrap from sshd in 2018
(https://bugzilla.redhat.com/show_bug.cgi?id=1530163), and
https://fedoraproject.org/wiki/Changes/Deprecate_TCP_wrappers has some
other options here (which would need to be adapted for Debian, but
broadly similar approaches would work).


SELinux
======
The fact that we build using --with-selinux has come up
(https://cybervillains.com/@djm/112192735215160932).  I haven't formed a
complete opinion on this, but I'm less worried about this linkage than
about GSS-API: it doesn't need much in the way of complex OpenSSH
patches, and the idea that it links indirectly to liblzma seems to have
been a mistaken one that turned up early in the discussions around the
xz-utils backdoor.

My feeling on this is that it's probably of about as much concern as
PAM, which we're definitely stuck with enabling, and I'm not
enthusiastic about adding a matrix of variant packages.  We could go for
something like openssh-{client,server}-full, but I'm not clear that
there's much in the way of correlation between people who need GSS-API
key exchange and people who need SELinux support, and I don't want to
force more people than necessary onto the variant that includes an extra
4000-odd-line patch.

For the time being my inclination is to leave this be, but I've seen the
suggestion that pam_selinux is basically all you need
(https://infosec.exchange/@alwayscurious/112192949171400643), so maybe
it would be an option to drop --with-selinux in favour of that?  I've
never used SELinux, so I'd need an expert to weigh on here.


Comments welcome,

-- 
Colin Watson (he/him)                              [cjwatson at debian.org]

On Tue, 2 Apr 2024, Colin Watson wrote:

[I'm not subscribed to the debian-* lists, please Cc me in replies if
you want me to see them]
> [I've CCed openssh-unix-dev for awareness, but set Mail-Followup-To to
> just debian-devel and debian-ssh to avoid potentially spamming them
> with a long discussion. If you choose to override this then that's
> your call, but please be mindful of upstream's time.]
Thanks Colin for considering how to reduce dependency chains for sshd.
I just remembered that this is not the first time that sshd has been
attacked via a transitive library dependency - it has happened before,
about 10 years ago:

https://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/

Attacks like these are impossible for sshd to defend against itself.
Instead we have to look to minimising the number of libraries that end
up in sshd's address space, especially that of the privileged sshd
process.

We are currently exploring splitting sshd into separate binaries for
the listener, privileged monitor, pre- and post-auth network-facing
processes so that each can be reduced in size and functionality to
the minimum possible. This should remove a number of dependencies from
the privileged process. There's a draft of these changes at
https://github.com/djmdjm/openssh-wip/pull/26 but it's OpenBSD-only
at this stage. We're likely to proceed with splitting the listener
process from the rest of sshd hopefully before the next release.

Another thing we're considering in OpenSSH is changing how we integrate
with PAM. PAM's API demands loading modules into the authenticating
process' address space, but obviously we've just been reminded that this
is risky.

I think that I would prefer to move to a model where there PAM auth and
account modules run in a helper process, and only the session module
runs in the unprivileged post-auth sshd process.

This means that PAM auth/account modules and their transitive library
dependencies cannot affect the sshd address space. They would still
likely need to run with privilege, could still fail permissively in
unwanted situations and might still be able to cause problems directly
(e.g. opening a reverse shell from the PAM module itself), but they
would no longer have direct access to the contents of sshd network
traffic, signatures, etc that are extremely useful in building NOBUS
(https://en.wikipedia.org/wiki/NOBUS) backdoors like the xv one.

Where this gets challenging is that some PAM modules make assumptions
that the auth, account and session modules all run in the same address
space. These would break until re-architected to pass things explicitly,
e.g. via environment variables, temp files, etc.

Time permitting, I'll get a prototype of these changes made for wider
experimentation.

-d

On Tue, Apr 02, 2024 at 01:30:11AM +0100, Colin Watson
wrote:
>  * for Debian trixie (current testing):
> 
>    * add dependency-only packages called something like
>      openssh-client-gsskex and openssh-server-gsskex, depending on their
>      non-gsskex alternatives
>    * add NEWS.Debian entry saying that people need to install these
>      packages if they want to retain GSS-API key exchange support
This is now implemented in Debian unstable.  I called the packages
openssh-client-gssapi and openssh-server-gssapi, with the intention of
splitting out both GSS-API authentication and key exchange support
later: that is, in trixie+1 I intend to build openssh without
--with-kerberos5 as well as dropping the key exchange patch from the
main packages, and you'd have to use openssh-*-gssapi for either
function.

-- 
Colin Watson (he/him)                              [cjwatson at debian.org]