openssh unix dev - May 2004 - OpenSSH v3.8p1 fails to interoperate for GSSAPI (Kerberos) and X-Windows

Versions: openssh-3.8p1-33, heimdal-0.6.1rc3-51, XFree86-4.3.99.902-40,
tk-8.4.6-37, all from SuSE 9.1 (unhacked); back-version peers have
openssh-3.5p1, XFree86-4.3.0-115, etc. from SuSE 8.2.

Symptoms:  

1.  When the client and server versions are unequal, the Kerberos ticket
is not accepted for authentication.  All the clients have
PreferredAuthentications gssapi-with-mic, gssapi, others.

2.  When ForwardX11 is true (ssh -X switch), ForwardX11Trusted is at its
default value (false), the client is 3.8p1, and the server is 3.5p1,
most X Window System clients work OK, but the following fail with the
indicated obscure error messages:
    
    xmag        BadDrawable (infinite loop of this error) 
    xwd		BadWindow 
    xcalc       (intermittent; sorry, didn't write down the code) 
    tcl/tk apps BadAtom doing X_GetProperty for 0x1a0 = InterpRegistry.
		To demonstrate, it's sufficient to do "wish" with no
		script.

Opera,  Konqueror, Mozilla and Netscape were not tested but they all
have a feature similar to tcl/tk and most likely would fail similarly.
This is a property on the root window through which one instance passes
messages to another, e.g. so a mail reader's helper app can signal a
browser to open a URL in a new window.

Sometimes but not always, the failure of one of the above apps triggers
a behavior mode where, from then on, the connection from any X app
whatsoever is refused with the message "Invalid MIT-MAGIC-COOKIE-1
key".

With the ssh -Y switch (ForwardX11Trusted true) on a v3.8p1 client and
v3.5p1 server, all the above apps work properly.  When the client is
v3.5p1 and with any server version, all the apps work properly.
In a trinary chain 3.5 -> 3.8 -> 3.5, all the apps work properly.

When both the client and the server are 3.8p1, and tcl/tk is the latest
version, it works but xmag and xwd fail as above.

Analysis: 

Sometime between 2003-09-12 and the present, a draft RFC:
    http://www.vandyke.com/technology/draft-ietf-secsh-gsskeyex.txt
was issued defining gssapi-with-mic, which resists certain "man in the
middle" attacks. v3.8p1 does only gssapi-with-mic; versions up to v3.7
do only old-style gssapi.  There appears to be no ./configure switch to
turn on gssapi-without-mic at compile time in v3.8.  The resulting lack
of interoperability fully explains the symptoms seen.  There is no
workaround.

Web postings suggest that apparently there has been an upgrade to
X-Windows security that involves a configurable security policy that
includes at least two classes of clients, trusted and untrusted, and
something involving access cookies that have a limited lifetime.  From
these hints I can understand the X errors: each failing app tries to
read an object (window, property, etc.) that it did not create, which
fails if it is generically untrusted.  As a workaround, the Debian
developer Colin Wilson has specified ForwardX11Trusted = true as the
default in their distro's ssh_config.

Discussion:  

To get to the conclusions above I spent all of Saturday from morning to
22:00, compiling back versions of OpenSSH and, eventually, reverting
files of v3.8p1 to v3.7p1, one at a time, until I hit a relevant effect.
Once I spotted the change to ForwardX11Trusted in readconf.c, I had the
keyword for a search on Google, which revealed what was happening.  I am
not a happy camper at this moment.

Here at UCLA-Mathnet we had to revert a patch to Solaris, that hackers
are actively exploiting, because of a lack of backward compatibility: it
breaks several important apps and we couldn't get them upgraded
workingly.  For this reason we are going through a lot of disruption and
spending a lot of money to junk a collection of Sun servers and convert
to Linux (SuSE), where our apps do run and which we can keep patched.
Now that we don't have to support some older Solaris boxes we're also
deploying Kerberos (Heimdal).  We intend to provide the excellent
service that our users expect.  In particular, we are part of a
worldwide community and we need to interoperate with peers whose upgrade
policies we can't dictate.

We're going to have to make a decision: by installing or reverting v3.8
we will accept or send out Kerberos tickets over ssh only with peers who
do run v3.8, or only from those who don't.  Given our situation it's
clear that we have to make the second choice: revert from v3.8p1 to some
earlier version, probably v3.7p1.

That decision makes the X-Windows issue moot, but eventually we would
need to get ForwardX11Trusted turned on when needed: likely turn it on
in ssh_config.

For the Future:  

In OpenSSH v3.9 I hope to see both gssapi-with-mic and old-style gssapi
supported by both the client and the server.  I understand the need to
encourage sysops to upgrade to gssapi-with-mic, but a suicide strategy
(upgrading is mandatory and we won't talk to you if you don't) is
suicidal for you, not for us.  A truly paranoid sysop, in an environment
where "man in the middle" attacks are expected, will put only
gssapi-with-mic in PreferredAuthentications and/or do a similar maneuver
on his server using a future separate option.  But imposing that policy
atomically on everyone is not going to happen.  Not even within one
administrative domain, because machines have to be upgraded one at a
time, and if one had the bad judgment to upgrade to v3.8, the
post-upgrade and pre-upgrade machines would be cut off from one another,
for Kerberos and X11 trust.

Helpful for the user would be an "ask" option for
GSSAPIAuthentication,
or even for every style of authentication, triggering a message along
these lines:

    The server refuses gssapi-with-mic (or whatever your most preferred
    method is) but accepts gssapi (or other lesser method).  Is this a
    known obsolescence of the server, or is a man in the middle trying
    to steal your credential?

In his posting the Debian developer points out that users are going to
learn to make X-Windows work by setting ForwardX11Trusted true in their
personal .ssh/config files, or putting -Y in their ssh shell aliases,
where the sysop cannot discover and revert them when the current
confusion on X security is straightened out.  Unfortunately,
ForwardX11Trusted implies ForwardX11, so if I set ForwardX11Trusted
true, every ssh will forward X, which is a waste and a possible security
exposure.  I request that these options be made independent, including 
a -y switch on ssh to turn off trust.  In ssh_config I would set FowardX11
false and ForwardX11Trusted true, so plain ssh does not forward, but if
X were forwarded with the -X switch, it would be trusted, unless the user
overrode for particular hosts with -y or in his private .ssh/config file.

Later, when I had some idea how or whether my apps were going to work
through X11 security, I would change the global ForwardX11Trusted to
false, after notifying the users.

My users and I look forward to getting the best security possible on any
particular connection, even if some connections are superior to others.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc at math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Before I start, 3.8.1p1 is out and fixes a number of bugs with 3.8p1, so 
you should consider using it instead.

Jim Carter wrote:
> Sometime between 2003-09-12 and the present, a draft RFC:
>     http://www.vandyke.com/technology/draft-ietf-secsh-gsskeyex.txt
> was issued defining gssapi-with-mic, which resists certain "man in the
> middle" attacks. v3.8p1 does only gssapi-with-mic; versions up to v3.7
> do only old-style gssapi.  There appears to be no ./configure switch to
> turn on gssapi-without-mic at compile time in v3.8.  The resulting lack
> of interoperability fully explains the symptoms seen.  There is no
> workaround.
Simon Wilkinson published a patch to enable backwards compatibility with 
"gssapi" authentication.
http://marc.theaimsgroup.com/?l=openssh-unix-dev&m=107826289602763
> To get to the conclusions above I spent all of Saturday from morning to
> 22:00, compiling back versions of OpenSSH
Were the release notes unclear?  Both of these were documented in the 
"Changes since OpenSSH 3.7.1" section":

* ssh(1) now uses untrusted cookies for X11-Forwarding.
   Some X11 applications might need full access to the X11 server,
   see ForwardX11Trusted in ssh(1) and xauth(1) for more information.

* The experimental "gssapi" support has been replaced with
   the "gssapi-with-mic" to fix possible MITM attacks.
   The two versions are not compatible.

The full text of the release notes is:
http://www.openssh.com/txt/release-3.8
> In OpenSSH v3.9 I hope to see both gssapi-with-mic and old-style gssapi
> supported by both the client and the server.
Limited interop (with what is an obsolete internet-draft with known 
security problems) is provided by Simon's patch.
> In his posting the Debian developer points out that users are going to
> learn to make X-Windows work by setting ForwardX11Trusted true in their
> personal .ssh/config files, or putting -Y in their ssh shell aliases,
> where the sysop cannot discover and revert them when the current
> confusion on X security is straightened out.  Unfortunately,
> ForwardX11Trusted implies ForwardX11, so if I set ForwardX11Trusted
> true, every ssh will forward X, which is a waste and a possible security
> exposure.
That's only the "-Y" command-line option (which is a substitute
for
"-X"), ForwardX11Trusted does not imply ForwardX11 (at least in the 
current version, I didn't check older ones).

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
     Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

Jim Carter wrote:
> Versions: openssh-3.8p1-33, heimdal-0.6.1rc3-51, XFree86-4.3.99.902-40,
> tk-8.4.6-37, all from SuSE 9.1 (unhacked); back-version peers have
> openssh-3.5p1, XFree86-4.3.0-115, etc. from SuSE 8.2.
> 
> Symptoms:  
> 
> 1.  When the client and server versions are unequal, the Kerberos ticket
> is not accepted for authentication.  All the clients have
> PreferredAuthentications gssapi-with-mic, gssapi, others.
openssh-3.5 never included GSSAPI autentication. You must be using a
distribution or 3rd party patch.
> 2.  When ForwardX11 is true (ssh -X switch), ForwardX11Trusted is at its
> default value (false), the client is 3.8p1, and the server is 3.5p1,
> most X Window System clients work OK, but the following fail with the
> indicated obscure error messages:
>     
>     xmag        BadDrawable (infinite loop of this error) 
>     xwd		BadWindow 
>     xcalc       (intermittent; sorry, didn't write down the code) 
>     tcl/tk apps BadAtom doing X_GetProperty for 0x1a0 = InterpRegistry.
> 		To demonstrate, it's sufficient to do "wish" with no
> 		script.
http://www.openssh.com/faq.html#3.13
> Sometime between 2003-09-12 and the present, a draft RFC:
>     http://www.vandyke.com/technology/draft-ietf-secsh-gsskeyex.txt
> was issued defining gssapi-with-mic, which resists certain "man in the
> middle" attacks. v3.8p1 does only gssapi-with-mic; versions up to v3.7
> do only old-style gssapi.  There appears to be no ./configure switch to
> turn on gssapi-without-mic at compile time in v3.8.  The resulting lack
> of interoperability fully explains the symptoms seen.  There is no
> workaround.
We won't include support for a deprecated exchange with known
security problems. The only release (3.7) that included support for
the "gssapi" (i.e. without MIC) included the following text in the
release notes:
>     - This release contains some GSSAPI user authentication support
>       to replace legacy KerberosV authentication support. At present
>       this code is still considered experimental and SHOULD NOT BE
>       USED.
I'm told that Simon Wilkinson has patches to add the old gssapi
(no MIC) exchange back for people who need it, but I can't see it on
his site: http://www.sxw.org.uk/computing/patches/openssh.html

The GSSAPI issue wouldn't have caused you as much pain if your Linux
vendor hadn't added support for an unfinished protocol. Most other
Linux vendors did the right thing and made the patch available as a
compile time options, or as a clearly labelled separate package.

If this same vendor is not providing you with the necessary support to
retain compatibility with their previous versions, then you probably go
and yell at them :)

-d

Jim Carter wrote:
[snip]
> 2.  When ForwardX11 is true (ssh -X switch), ForwardX11Trusted is at its
> default value (false), the client is 3.8p1, and the server is 3.5p1,
> most X Window System clients work OK, but the following fail with the
> indicated obscure error messages:
> 
>     xmag        BadDrawable (infinite loop of this error)
>     xwd         BadWindow
>     xcalc       (intermittent; sorry, didn't write down the code)
Could you file BUG REPORTS at
http://freedesktop.org/bugzilla/enter_bug.cgi?product=xorg , one bug
report per application, please ?
If you hurry up it's likely that the next release (X11R6.7.1 ?!) may
include the fixes...

----

Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix
programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Thank you all for your replies.  Please accept my apology for a somewhat
intemperate tone, but also please consider where I was coming from: I
had figured out that our Kerberos deployment was going to be derailed
because of the 3.5[SuSE] <-> 3.8 lack of interoperability, and then I
turned to the X-Windows issue: seemingly random error messages that
suggested corruption in the encrypted channel.  I had no idea that it
was deliberate and documented!

Darren Tucker <dtucker at zip.com.au> wrote:
> Simon Wilkinson published a patch to enable backwards compatibility with
> "gssapi" authentication.
> http://marc.theaimsgroup.com/?l=openssh-unix-dev&m=107826289602763
Thank you!  This will be very helpful.  I must have used keywords (like
gssapi-with-mic) in my Google search that missed it.  I have a software
audit script that can deal with locally patched software, rather than
having to slavishly use whatever the distro gives us.  (A big advantage
of using a distro is that most of the time you can automate patches, but
there *is* a downside...)  When all of our systems have been upgraded
and when we're sure that off-site users aren't going to get cut off --
probably we won't have too many that we'll have to bully into upgrading
-- we can decommit gssapi-without-mic.
> That's only the "-Y" command-line option (which is a
substitute for
> "-X"),  ForwardX11Trusted does not imply ForwardX11 (at least in
the
> current version, I didn't check older ones).
OK, that's reasonable.  For the record, I confirm that if you set
ForwardX11Trusted=true and ForwardX11=false in ssh_config, then plain
"ssh" does not forward X11, but "ssh -X" does forward it,
and it is
trusted (the offending apps will run).  (With either setting, ssh -Y
works as expected.)  This is how we've set up our ssh_config for the
machines with openssh v3.8p1, following the Debian guy's suggestion.


James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc at math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Damien Miller <djm at mindrot.org> wrote:
> The GSSAPI issue wouldn't have caused you as much pain if your Linux
> vendor hadn't added support for an unfinished protocol. Most other
> Linux vendors did the right thing and made the patch available as a
> compile time options, or as a clearly labelled separate package.
> If this same vendor is not providing you with the necessary support to
> retain compatibility with their previous versions, then you probably go
> and yell at them :)
I have :-(  I'll append your comments to the ticket file.  

The vendor is SuSE.  Generally they're fairly aggressive, but within the
bounds of reason, in getting new features into their distro.  I see your
comment that GSSAPI first appeared in OpenSSH v3.7.  You are probably
right that SuSE added a circulating patch to v3.5p1, or backported the
3.7 GSSAPI code, both of which they are known to do regularly when a 
feature is important.

In the current climate of hacking, I imagine that Kerberos-capable ssh
is one of the features most often asked for.  Here's a reference for
"current climate of hacking", which is worth reading:

http://securecomputing.stanford.edu/alerts/multiple-unix-6apr2004.html

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc at math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)