openssh unix dev - Oct 2017 - Status of OpenSSL 1.1 support - Thoughts

Hi Ingo,

On Wed, Oct 18, 2017 at 4:15 PM, Ingo Schwarze <schwarze at usta.de>
wrote:
> Hi,
>
> jpbion at jfwest.com wrote on Wed, Oct 18, 2017 at 05:53:21AM -0700:
>
>> 4) As a first result, with no judgement on anyone, just looking at the
>> data - the root cause of this issue seems to be the split of LibreSSL
>> from OpenSSL
>
> No, you are totally misrepresenting the situation.  The root cause
> is the split of OpenSSL-1.1 from OpenSSL-1.0, and that OpenSSL
> dumped the large and dangerous work of dealing with the large-scale
> API change on each and every application project instead of providing
> an official transition path that can be taken seriously.
Important API change between major releases is something to be
expected. Sometimes the changes are limited, sometimes thay aren't.

The structure of the changes themselves is the reason why the openssl
folks could not provide any migration path (the existance of this path
would have negated the changes themselves). Anyway, this is done, and
I'm not sure we have the power to go back in time to fix that (and I'm
not even sure it's necessary to fix it).

Anyway, I still agree with you: the problem is not the split, it's the
API change.
> LibreSSL has almost nothing to do with the problem.  Even if LibreSSL
> had never happened, the same problem would still exists.
>
> Oh, wait, LibreSSL has to do with it in one sense.  It is available
> as one possible way to *solve* the problem.  Either temporarily or
> for good, whichever you like.
This is not a good idea IMHO. One want to be able to write
security-related code easily, and the old OpenSSL API (and thus, the
current LibreSSL interface) does not make that simple at all. As a
result, we got many incorrect use of this very API in the past which
led to security issues in many softwares (CVE-2009-0021). While some
degree of API complexity is inherent to the role of the library, I'm
pretty sure you can agree that the simpler it is, the better.

Given the fact that the knowledge of the structures of LibreSSL is
needed to correclty use the library, I hope you'll understand my
point: using LibreSSL as it is today instead of porting to OpenSSL is
not a good idea :)

(Not to mention that more and more software implement compatibility
with OpenSSL 1.1, so asking them to go the other way is not going to
work well).
>> OpenSSL and LibreSSL, given the fact neither seems to have a desire
>> to maintain compatibility with the other (again, as far as I can see).
>
> That is an unfounded allegation.  Of course LibreSSL has a desire
> to eventually integrate the 1.1 API.  Joel has said so long ago,
> in public, that in principle, opaque structs are a good concept
> [for example citation 1: Dec 30, 2016], and i have heard repeated
> discussions inside the LibreSSL project on how to get there.  It
> is just a lot of work, it is made harder by the lack of a clear
> migration path, and it is of limited usefulness as long as application
> programs must still support the OpenSSL-1.0 API.  That's what
> prevented it from getting done so far.
There is no easy migration path. More precisely, there is only one
migration path, and it's not pretty. To allow applications to do the
jump at their pace, you have to propose both interfaces and mark the
old one as deprecated (and remove it in a subsequent release).

I'd volonteer to do this but given the gigantism of the task, any help
would be appreciated (not to mention that even if Joel agrees with the
idea of having the 1.1 API in LibreSSL, that does not mean he would
agree with my strategy).
> Given that you got the facts wrong, your conclusions are misleading
> as well.  All this was explained already, so your mail sounds almost
> trollish: It should already be well-known that the central design
> goal of LibreSSL is to be a compatible drop-in replacement for
> OpenSSL - at the time of the fork, that was OpenSSL-1.0.  If, after
> the fork, OpenSSL breaks its own API and leaves users in the rain,
> blaming that on LibreSSL is quite dishonest.  Even if the API break
> is so severe that it takes LibreSSL substantially more than a year
> to deal with it, even if LibreSSL hasn't yet solved the problem for
> its own purposes.
I agree with you on this one. The LibreSSL folks are not to be blamed
for this situation.
> The real problem is:  How is OpenSSH supposed to support OpenSSL-1.0
> and OpenSSL-1.1 at the same time, given that the API break is so
> severe that switching from one to the other requires a 3000+ line
> diff?
But at the same time, some ditributions are phasing out OpenSSL 1.0.2
and are integrating OpenSSL 1.1 while at the same time some supported
LTS distributions still rely on older OpenSSL versions. So for a
period of time you'll have to be compatible with both, whether you
like it or not. The question is then: is it up to you to be compatible
with both, or is it up to distributors to provide compatibilty? When
more and more softwares propose their own shim it makes the later less
and less understandable.
> Yours,
>   Ingo
BR,

-- Emmanuel Deloget

Hi Emmanuel,

Emmanuel Deloget wrote on Wed, Oct 18, 2017 at 05:45:40PM +0200:
> Important API change between major releases is something to be
> expected. Sometimes the changes are limited, sometimes thay aren't.
>
> The structure of the changes themselves is the reason why the openssl
> folks could not provide any migration path (the existance of this path
> would have negated the changes themselves).
True, but only in the 1.0 release series, while still attaining the
goal of making the transition manageable for application projects.
When you implement a major redesign, you can never expect to reap
the benefit in the old stable branch.  The way OpenSSL did it doesn't
achieve that either.
> Ingo Schwarze wrote:
>> Oh, wait, LibreSSL has to do with it in one sense.  It is available
>> as one possible way to *solve* the problem.  Either temporarily or
>> for good, whichever you like.
> This is not a good idea IMHO. One want to be able to write
> security-related code easily, and the old OpenSSL API (and thus, the
> current LibreSSL interface) does not make that simple at all. As a
> result, we got many incorrect use of this very API in the past which
> led to security issues in many softwares (CVE-2009-0021). While some
> degree of API complexity is inherent to the role of the library, I'm
> pretty sure you can agree that the simpler it is, the better.
> 
> Given the fact that the knowledge of the structures of LibreSSL is
> needed to correclty use the library, I hope you'll understand my
> point: using LibreSSL as it is today instead of porting to OpenSSL is
> not a good idea :)
For the short term, it is of the same order of quality as using
the officially maintained OpenSSL-1.0 branch, or maybe even a bit
better because it may have fewer bugs.  Of course, neither gets
the benefits of opaque structs, but both allow working around the
transition problem.

In the long term, LibreSSL will hopefully transition to opaque structs
as well, only less rushed and with a better transition plan, at which
point your argument will become moot.
> More precisely, there is only one migration path, and it's not pretty.
> To allow applications to do the jump at their pace, you have to propose
> both interfaces and mark the old one as deprecated (and remove it in
> a subsequent release).
Precisely.  That is how you do a large-scale change that is
incompatible both ways in a widely-used library:

 1. Say you are at version 41.0.
 2. Add the new interfaces in version 41.1 (minor bump).
 3. Invite some application projects to switch to the new interfaces.
 4. Help them with the switch and learn from the experience.
 5. Quite probably, you will have to publish bug fix releases
    41.1.1 and 41.1.2 to improve what you did in steps 2 to 4.
 6. Invite *all* users to switch.
 7. Once many of your users have switched, announce future deprecation.
 8. Wait some time.
 9. Announce end of life for a specific date.
 10. Remove the old interfaces in version 42.0 (major bump).

OpenSSL jumped the shark and skipped steps 3 to 9, doing 1 and 10
in one single leap.

Of course, you can't get the full benefit until step 10, in particular
when the point is to improve encapsulation.  But that's no excuse
for putting users in an impossible situation.

If, for whatever reason, you realize too late that you completely
forgot about steps 3 to 9, you can still mitigate part of the havoc
done by doing step 2 after step 10, enabling the forgotten migration
path.  Better late than never.
> I'd volonteer to do this but given the gigantism of the task, any help
> would be appreciated (not to mention that even if Joel agrees with the
> idea of having the 1.1 API in LibreSSL, that does not mean he would
> agree with my strategy).
You could offer your help for providing that migration path to the
OpenSSL project, if you feel qualified and willing to provide such
help.  I think you should definitely use the expertise of the OpenSSL
developers to make sure that providing the migration path doesn't
introduce bugs and vulnerabilities, and avoid working as a lone
wolf if possible.
> But at the same time, some ditributions are phasing out OpenSSL 1.0.2
> and are integrating OpenSSL 1.1 while at the same time some supported
> LTS distributions still rely on older OpenSSL versions. So for a
> period of time you'll have to be compatible with both, whether you
> like it or not. The question is then: is it up to you to be compatible
> with both,
The OpenSSH project simply doesn't have the manpower to maintain
compatibility with both at the same time, and even if it had, doing
so would compromise the central project goal of security through
simplicity and diligence.
> or is it up to distributors to provide compatibilty?
That will be beyond the means of many distributors, in which case
they either need to use OpenSSL-1.0 or LibreSSL for OpenSSH.

Even for those distributors who can afford the required manpower,
i consider attempting to maintain a 3000+ line vendor patch risky,
and maintaining a non-official compat layer risky as well.  Quite
respectable vendors have screwed up with three-line patches in the
past, how can anybody feel confident about a 3000-line vendor patch?
How sure can you feel that none of these 3000 lines causes non-obvious
problems?  That also answers the question Jakub Jelen asked why i
would feel uncomfortable about using Fedora.

Yours,
  Ingo

OpenSSL developers believed that there was a need for a significant change. A
part of that change was a conscious choice to break (some of) the existing API.
They considered that pain unavoidable. So far I happen to agree with their
rationale and approach. Move from visible internal structures to accessor
functions is a good thing, regardless of what you may think of it. And the new
API *is* better, again like it or not.

I understand the frustration with lack of a ?migration library?, but how to you
see a ?shim? that allows code that relies on being able to directly access
members of structures, run unmodified (just recompiled)?

In my opinion, a reasonable thing for OpenSSH to do would be to port their code
to using accessor functions, and write a shim library to the ?old? way (exactly
as was proposed here before).

LibreSSL would have to do the same eventually. The pain of such migration can be
postponed, but not avoided indefinitely.
--
Regards,
Uri Blumenthal

On 10/18/17, 12:38, "openssh-unix-dev on behalf of Ingo Schwarze"
<openssh-unix-dev-bounces+uri=ll.mit.edu at mindrot.org on behalf of schwarze
at usta.de> wrote:

    Hi Emmanuel,
    
    Emmanuel Deloget wrote on Wed, Oct 18, 2017 at 05:45:40PM +0200:
    
    > Important API change between major releases is something to be
    > expected. Sometimes the changes are limited, sometimes thay aren't.
    >
    > The structure of the changes themselves is the reason why the openssl
    > folks could not provide any migration path (the existance of this path
    > would have negated the changes themselves).
    
    True, but only in the 1.0 release series, while still attaining the
    goal of making the transition manageable for application projects.
    When you implement a major redesign, you can never expect to reap
    the benefit in the old stable branch.  The way OpenSSL did it doesn't
    achieve that either.
    
    > Ingo Schwarze wrote:
    
    >> Oh, wait, LibreSSL has to do with it in one sense.  It is available
    >> as one possible way to *solve* the problem.  Either temporarily or
    >> for good, whichever you like.
    
    > This is not a good idea IMHO. One want to be able to write
    > security-related code easily, and the old OpenSSL API (and thus, the
    > current LibreSSL interface) does not make that simple at all. As a
    > result, we got many incorrect use of this very API in the past which
    > led to security issues in many softwares (CVE-2009-0021). While some
    > degree of API complexity is inherent to the role of the library,
I'm
    > pretty sure you can agree that the simpler it is, the better.
    > 
    > Given the fact that the knowledge of the structures of LibreSSL is
    > needed to correclty use the library, I hope you'll understand my
    > point: using LibreSSL as it is today instead of porting to OpenSSL is
    > not a good idea :)
    
    For the short term, it is of the same order of quality as using
    the officially maintained OpenSSL-1.0 branch, or maybe even a bit
    better because it may have fewer bugs.  Of course, neither gets
    the benefits of opaque structs, but both allow working around the
    transition problem.
    
    In the long term, LibreSSL will hopefully transition to opaque structs
    as well, only less rushed and with a better transition plan, at which
    point your argument will become moot.
    
    > More precisely, there is only one migration path, and it's not
pretty.
    > To allow applications to do the jump at their pace, you have to propose
    > both interfaces and mark the old one as deprecated (and remove it in
    > a subsequent release).
    
    Precisely.  That is how you do a large-scale change that is
    incompatible both ways in a widely-used library:
    
     1. Say you are at version 41.0.
     2. Add the new interfaces in version 41.1 (minor bump).
     3. Invite some application projects to switch to the new interfaces.
     4. Help them with the switch and learn from the experience.
     5. Quite probably, you will have to publish bug fix releases
        41.1.1 and 41.1.2 to improve what you did in steps 2 to 4.
     6. Invite *all* users to switch.
     7. Once many of your users have switched, announce future deprecation.
     8. Wait some time.
     9. Announce end of life for a specific date.
     10. Remove the old interfaces in version 42.0 (major bump).
    
    OpenSSL jumped the shark and skipped steps 3 to 9, doing 1 and 10
    in one single leap.
    
    Of course, you can't get the full benefit until step 10, in particular
    when the point is to improve encapsulation.  But that's no excuse
    for putting users in an impossible situation.
    
    If, for whatever reason, you realize too late that you completely
    forgot about steps 3 to 9, you can still mitigate part of the havoc
    done by doing step 2 after step 10, enabling the forgotten migration
    path.  Better late than never.
    
    > I'd volonteer to do this but given the gigantism of the task, any
help
    > would be appreciated (not to mention that even if Joel agrees with the
    > idea of having the 1.1 API in LibreSSL, that does not mean he would
    > agree with my strategy).
    
    You could offer your help for providing that migration path to the
    OpenSSL project, if you feel qualified and willing to provide such
    help.  I think you should definitely use the expertise of the OpenSSL
    developers to make sure that providing the migration path doesn't
    introduce bugs and vulnerabilities, and avoid working as a lone
    wolf if possible.
    
    > But at the same time, some ditributions are phasing out OpenSSL 1.0.2
    > and are integrating OpenSSL 1.1 while at the same time some supported
    > LTS distributions still rely on older OpenSSL versions. So for a
    > period of time you'll have to be compatible with both, whether you
    > like it or not. The question is then: is it up to you to be compatible
    > with both,
    
    The OpenSSH project simply doesn't have the manpower to maintain
    compatibility with both at the same time, and even if it had, doing
    so would compromise the central project goal of security through
    simplicity and diligence.
    
    > or is it up to distributors to provide compatibilty?
    
    That will be beyond the means of many distributors, in which case
    they either need to use OpenSSL-1.0 or LibreSSL for OpenSSH.
    
    Even for those distributors who can afford the required manpower,
    i consider attempting to maintain a 3000+ line vendor patch risky,
    and maintaining a non-official compat layer risky as well.  Quite
    respectable vendors have screwed up with three-line patches in the
    past, how can anybody feel confident about a 3000-line vendor patch?
    How sure can you feel that none of these 3000 lines causes non-obvious
    problems?  That also answers the question Jakub Jelen asked why i
    would feel uncomfortable about using Fedora.
    
    Yours,
      Ingo
    _______________________________________________
    openssh-unix-dev mailing list
    openssh-unix-dev at mindrot.org
    https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
    
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5211 bytes
Desc: not available
URL:
<http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20171018/fc0a3ab7/attachment-0001.p7s>

Hello,


On Wed, Oct 18, 2017 at 6:36 PM, Ingo Schwarze <schwarze at usta.de>
wrote:
> Hi Emmanuel,
>
> Emmanuel Deloget wrote on Wed, Oct 18, 2017 at 05:45:40PM +0200:
<snip>
>> I'd volonteer to do this but given the gigantism of the task, any
help
>> would be appreciated (not to mention that even if Joel agrees with the
>> idea of having the 1.1 API in LibreSSL, that does not mean he would
>> agree with my strategy).
>
> You could offer your help for providing that migration path to the
> OpenSSL project, if you feel qualified and willing to provide such
> help.  I think you should definitely use the expertise of the OpenSSL
> developers to make sure that providing the migration path doesn't
> introduce bugs and vulnerabilities, and avoid working as a lone
> wolf if possible.
Working as a lone wolf on this project is definitely not what I want
to do. However, I can propose the first (simple) patches and hopefully
someone else will join the effort once it has started :)
>> But at the same time, some ditributions are phasing out OpenSSL 1.0.2
>> and are integrating OpenSSL 1.1 while at the same time some supported
>> LTS distributions still rely on older OpenSSL versions. So for a
>> period of time you'll have to be compatible with both, whether you
>> like it or not. The question is then: is it up to you to be compatible
>> with both,
>
> The OpenSSH project simply doesn't have the manpower to maintain
> compatibility with both at the same time, and even if it had, doing
> so would compromise the central project goal of security through
> simplicity and diligence.
Ok.
>> or is it up to distributors to provide compatibilty?
>
> That will be beyond the means of many distributors, in which case
> they either need to use OpenSSL-1.0 or LibreSSL for OpenSSH.
>
> Even for those distributors who can afford the required manpower,
> i consider attempting to maintain a 3000+ line vendor patch risky,
> and maintaining a non-official compat layer risky as well.  Quite
> respectable vendors have screwed up with three-line patches in the
> past, how can anybody feel confident about a 3000-line vendor patch?
> How sure can you feel that none of these 3000 lines causes non-obvious
> problems?  That also answers the question Jakub Jelen asked why i
> would feel uncomfortable about using Fedora.
>
> Yours,
>   Ingo
Let's restate the full goal here :

* first, LibreSSL should integrate the OpenSSL 1.1 API
* second, it should mark the replaced interface as deprecated
* third, make OpenSSH use the new interface

That sounds like a manageable list of goals -- i.e. one that can prove
to be completed before the EOL of OpenSSL 1.0.2 in Dec. 2019.

First problem to overcome is that LibreSSL is released every 6 month,
so the work is probably going to be split on two or more releases. As
a consequence, the transition work cannot start on OpenSSH before the
next LibreSSL release (i.e. version 2.7.0 that should be released in
May) and will probably take a long time.

For distributors and maintainers out there, that means:

* distributions that decided to phase out OpenSSL 1.0.2 in favor of
OpenSSL 1.1 will have to support an OpenSSH patch that will grow
smaller and smaller at each release.
* distributions that decided to keep OpenSSL 1.0.2 or to link OpenSSH
with LibreSSL have nothing to do, and will be ready to phase out
OpenSSL 1.0.2 at EOL time.

Is it something that apear to be manageable?

BTW, do you know any other software that uses the LibreSSL API the
same way OpenSSH is using it (i.e. LibreSSL is the primary target,
OpenSSL compatibility is of lesser importance)?

BR,

-- Emmanuel Deloget