Hi Ingo, On Wed, Oct 18, 2017 at 4:15 PM, Ingo Schwarze <schwarze at usta.de> wrote:> Hi, > > jpbion at jfwest.com wrote on Wed, Oct 18, 2017 at 05:53:21AM -0700: > >> 4) As a first result, with no judgement on anyone, just looking at the >> data - the root cause of this issue seems to be the split of LibreSSL >> from OpenSSL > > No, you are totally misrepresenting the situation. The root cause > is the split of OpenSSL-1.1 from OpenSSL-1.0, and that OpenSSL > dumped the large and dangerous work of dealing with the large-scale > API change on each and every application project instead of providing > an official transition path that can be taken seriously.Important API change between major releases is something to be expected. Sometimes the changes are limited, sometimes thay aren't. The structure of the changes themselves is the reason why the openssl folks could not provide any migration path (the existance of this path would have negated the changes themselves). Anyway, this is done, and I'm not sure we have the power to go back in time to fix that (and I'm not even sure it's necessary to fix it). Anyway, I still agree with you: the problem is not the split, it's the API change.> LibreSSL has almost nothing to do with the problem. Even if LibreSSL > had never happened, the same problem would still exists. > > Oh, wait, LibreSSL has to do with it in one sense. It is available > as one possible way to *solve* the problem. Either temporarily or > for good, whichever you like.This is not a good idea IMHO. One want to be able to write security-related code easily, and the old OpenSSL API (and thus, the current LibreSSL interface) does not make that simple at all. As a result, we got many incorrect use of this very API in the past which led to security issues in many softwares (CVE-2009-0021). While some degree of API complexity is inherent to the role of the library, I'm pretty sure you can agree that the simpler it is, the better. Given the fact that the knowledge of the structures of LibreSSL is needed to correclty use the library, I hope you'll understand my point: using LibreSSL as it is today instead of porting to OpenSSL is not a good idea :) (Not to mention that more and more software implement compatibility with OpenSSL 1.1, so asking them to go the other way is not going to work well).>> OpenSSL and LibreSSL, given the fact neither seems to have a desire >> to maintain compatibility with the other (again, as far as I can see). > > That is an unfounded allegation. Of course LibreSSL has a desire > to eventually integrate the 1.1 API. Joel has said so long ago, > in public, that in principle, opaque structs are a good concept > [for example citation 1: Dec 30, 2016], and i have heard repeated > discussions inside the LibreSSL project on how to get there. It > is just a lot of work, it is made harder by the lack of a clear > migration path, and it is of limited usefulness as long as application > programs must still support the OpenSSL-1.0 API. That's what > prevented it from getting done so far.There is no easy migration path. More precisely, there is only one migration path, and it's not pretty. To allow applications to do the jump at their pace, you have to propose both interfaces and mark the old one as deprecated (and remove it in a subsequent release). I'd volonteer to do this but given the gigantism of the task, any help would be appreciated (not to mention that even if Joel agrees with the idea of having the 1.1 API in LibreSSL, that does not mean he would agree with my strategy).> Given that you got the facts wrong, your conclusions are misleading > as well. All this was explained already, so your mail sounds almost > trollish: It should already be well-known that the central design > goal of LibreSSL is to be a compatible drop-in replacement for > OpenSSL - at the time of the fork, that was OpenSSL-1.0. If, after > the fork, OpenSSL breaks its own API and leaves users in the rain, > blaming that on LibreSSL is quite dishonest. Even if the API break > is so severe that it takes LibreSSL substantially more than a year > to deal with it, even if LibreSSL hasn't yet solved the problem for > its own purposes.I agree with you on this one. The LibreSSL folks are not to be blamed for this situation.> The real problem is: How is OpenSSH supposed to support OpenSSL-1.0 > and OpenSSL-1.1 at the same time, given that the API break is so > severe that switching from one to the other requires a 3000+ line > diff?But at the same time, some ditributions are phasing out OpenSSL 1.0.2 and are integrating OpenSSL 1.1 while at the same time some supported LTS distributions still rely on older OpenSSL versions. So for a period of time you'll have to be compatible with both, whether you like it or not. The question is then: is it up to you to be compatible with both, or is it up to distributors to provide compatibilty? When more and more softwares propose their own shim it makes the later less and less understandable.> Yours, > IngoBR, -- Emmanuel Deloget
Hi Emmanuel, Emmanuel Deloget wrote on Wed, Oct 18, 2017 at 05:45:40PM +0200:> Important API change between major releases is something to be > expected. Sometimes the changes are limited, sometimes thay aren't. > > The structure of the changes themselves is the reason why the openssl > folks could not provide any migration path (the existance of this path > would have negated the changes themselves).True, but only in the 1.0 release series, while still attaining the goal of making the transition manageable for application projects. When you implement a major redesign, you can never expect to reap the benefit in the old stable branch. The way OpenSSL did it doesn't achieve that either.> Ingo Schwarze wrote:>> Oh, wait, LibreSSL has to do with it in one sense. It is available >> as one possible way to *solve* the problem. Either temporarily or >> for good, whichever you like.> This is not a good idea IMHO. One want to be able to write > security-related code easily, and the old OpenSSL API (and thus, the > current LibreSSL interface) does not make that simple at all. As a > result, we got many incorrect use of this very API in the past which > led to security issues in many softwares (CVE-2009-0021). While some > degree of API complexity is inherent to the role of the library, I'm > pretty sure you can agree that the simpler it is, the better. > > Given the fact that the knowledge of the structures of LibreSSL is > needed to correclty use the library, I hope you'll understand my > point: using LibreSSL as it is today instead of porting to OpenSSL is > not a good idea :)For the short term, it is of the same order of quality as using the officially maintained OpenSSL-1.0 branch, or maybe even a bit better because it may have fewer bugs. Of course, neither gets the benefits of opaque structs, but both allow working around the transition problem. In the long term, LibreSSL will hopefully transition to opaque structs as well, only less rushed and with a better transition plan, at which point your argument will become moot.> More precisely, there is only one migration path, and it's not pretty. > To allow applications to do the jump at their pace, you have to propose > both interfaces and mark the old one as deprecated (and remove it in > a subsequent release).Precisely. That is how you do a large-scale change that is incompatible both ways in a widely-used library: 1. Say you are at version 41.0. 2. Add the new interfaces in version 41.1 (minor bump). 3. Invite some application projects to switch to the new interfaces. 4. Help them with the switch and learn from the experience. 5. Quite probably, you will have to publish bug fix releases 41.1.1 and 41.1.2 to improve what you did in steps 2 to 4. 6. Invite *all* users to switch. 7. Once many of your users have switched, announce future deprecation. 8. Wait some time. 9. Announce end of life for a specific date. 10. Remove the old interfaces in version 42.0 (major bump). OpenSSL jumped the shark and skipped steps 3 to 9, doing 1 and 10 in one single leap. Of course, you can't get the full benefit until step 10, in particular when the point is to improve encapsulation. But that's no excuse for putting users in an impossible situation. If, for whatever reason, you realize too late that you completely forgot about steps 3 to 9, you can still mitigate part of the havoc done by doing step 2 after step 10, enabling the forgotten migration path. Better late than never.> I'd volonteer to do this but given the gigantism of the task, any help > would be appreciated (not to mention that even if Joel agrees with the > idea of having the 1.1 API in LibreSSL, that does not mean he would > agree with my strategy).You could offer your help for providing that migration path to the OpenSSL project, if you feel qualified and willing to provide such help. I think you should definitely use the expertise of the OpenSSL developers to make sure that providing the migration path doesn't introduce bugs and vulnerabilities, and avoid working as a lone wolf if possible.> But at the same time, some ditributions are phasing out OpenSSL 1.0.2 > and are integrating OpenSSL 1.1 while at the same time some supported > LTS distributions still rely on older OpenSSL versions. So for a > period of time you'll have to be compatible with both, whether you > like it or not. The question is then: is it up to you to be compatible > with both,The OpenSSH project simply doesn't have the manpower to maintain compatibility with both at the same time, and even if it had, doing so would compromise the central project goal of security through simplicity and diligence.> or is it up to distributors to provide compatibilty?That will be beyond the means of many distributors, in which case they either need to use OpenSSL-1.0 or LibreSSL for OpenSSH. Even for those distributors who can afford the required manpower, i consider attempting to maintain a 3000+ line vendor patch risky, and maintaining a non-official compat layer risky as well. Quite respectable vendors have screwed up with three-line patches in the past, how can anybody feel confident about a 3000-line vendor patch? How sure can you feel that none of these 3000 lines causes non-obvious problems? That also answers the question Jakub Jelen asked why i would feel uncomfortable about using Fedora. Yours, Ingo
Blumenthal, Uri - 0553 - MITLL
2017-Oct-18 17:27 UTC
Status of OpenSSL 1.1 support - Thoughts
OpenSSL developers believed that there was a need for a significant change. A part of that change was a conscious choice to break (some of) the existing API. They considered that pain unavoidable. So far I happen to agree with their rationale and approach. Move from visible internal structures to accessor functions is a good thing, regardless of what you may think of it. And the new API *is* better, again like it or not. I understand the frustration with lack of a ?migration library?, but how to you see a ?shim? that allows code that relies on being able to directly access members of structures, run unmodified (just recompiled)? In my opinion, a reasonable thing for OpenSSH to do would be to port their code to using accessor functions, and write a shim library to the ?old? way (exactly as was proposed here before). LibreSSL would have to do the same eventually. The pain of such migration can be postponed, but not avoided indefinitely. -- Regards, Uri Blumenthal On 10/18/17, 12:38, "openssh-unix-dev on behalf of Ingo Schwarze" <openssh-unix-dev-bounces+uri=ll.mit.edu at mindrot.org on behalf of schwarze at usta.de> wrote: Hi Emmanuel, Emmanuel Deloget wrote on Wed, Oct 18, 2017 at 05:45:40PM +0200: > Important API change between major releases is something to be > expected. Sometimes the changes are limited, sometimes thay aren't. > > The structure of the changes themselves is the reason why the openssl > folks could not provide any migration path (the existance of this path > would have negated the changes themselves). True, but only in the 1.0 release series, while still attaining the goal of making the transition manageable for application projects. When you implement a major redesign, you can never expect to reap the benefit in the old stable branch. The way OpenSSL did it doesn't achieve that either. > Ingo Schwarze wrote: >> Oh, wait, LibreSSL has to do with it in one sense. It is available >> as one possible way to *solve* the problem. Either temporarily or >> for good, whichever you like. > This is not a good idea IMHO. One want to be able to write > security-related code easily, and the old OpenSSL API (and thus, the > current LibreSSL interface) does not make that simple at all. As a > result, we got many incorrect use of this very API in the past which > led to security issues in many softwares (CVE-2009-0021). While some > degree of API complexity is inherent to the role of the library, I'm > pretty sure you can agree that the simpler it is, the better. > > Given the fact that the knowledge of the structures of LibreSSL is > needed to correclty use the library, I hope you'll understand my > point: using LibreSSL as it is today instead of porting to OpenSSL is > not a good idea :) For the short term, it is of the same order of quality as using the officially maintained OpenSSL-1.0 branch, or maybe even a bit better because it may have fewer bugs. Of course, neither gets the benefits of opaque structs, but both allow working around the transition problem. In the long term, LibreSSL will hopefully transition to opaque structs as well, only less rushed and with a better transition plan, at which point your argument will become moot. > More precisely, there is only one migration path, and it's not pretty. > To allow applications to do the jump at their pace, you have to propose > both interfaces and mark the old one as deprecated (and remove it in > a subsequent release). Precisely. That is how you do a large-scale change that is incompatible both ways in a widely-used library: 1. Say you are at version 41.0. 2. Add the new interfaces in version 41.1 (minor bump). 3. Invite some application projects to switch to the new interfaces. 4. Help them with the switch and learn from the experience. 5. Quite probably, you will have to publish bug fix releases 41.1.1 and 41.1.2 to improve what you did in steps 2 to 4. 6. Invite *all* users to switch. 7. Once many of your users have switched, announce future deprecation. 8. Wait some time. 9. Announce end of life for a specific date. 10. Remove the old interfaces in version 42.0 (major bump). OpenSSL jumped the shark and skipped steps 3 to 9, doing 1 and 10 in one single leap. Of course, you can't get the full benefit until step 10, in particular when the point is to improve encapsulation. But that's no excuse for putting users in an impossible situation. If, for whatever reason, you realize too late that you completely forgot about steps 3 to 9, you can still mitigate part of the havoc done by doing step 2 after step 10, enabling the forgotten migration path. Better late than never. > I'd volonteer to do this but given the gigantism of the task, any help > would be appreciated (not to mention that even if Joel agrees with the > idea of having the 1.1 API in LibreSSL, that does not mean he would > agree with my strategy). You could offer your help for providing that migration path to the OpenSSL project, if you feel qualified and willing to provide such help. I think you should definitely use the expertise of the OpenSSL developers to make sure that providing the migration path doesn't introduce bugs and vulnerabilities, and avoid working as a lone wolf if possible. > But at the same time, some ditributions are phasing out OpenSSL 1.0.2 > and are integrating OpenSSL 1.1 while at the same time some supported > LTS distributions still rely on older OpenSSL versions. So for a > period of time you'll have to be compatible with both, whether you > like it or not. The question is then: is it up to you to be compatible > with both, The OpenSSH project simply doesn't have the manpower to maintain compatibility with both at the same time, and even if it had, doing so would compromise the central project goal of security through simplicity and diligence. > or is it up to distributors to provide compatibilty? That will be beyond the means of many distributors, in which case they either need to use OpenSSL-1.0 or LibreSSL for OpenSSH. Even for those distributors who can afford the required manpower, i consider attempting to maintain a 3000+ line vendor patch risky, and maintaining a non-official compat layer risky as well. Quite respectable vendors have screwed up with three-line patches in the past, how can anybody feel confident about a 3000-line vendor patch? How sure can you feel that none of these 3000 lines causes non-obvious problems? That also answers the question Jakub Jelen asked why i would feel uncomfortable about using Fedora. Yours, Ingo _______________________________________________ openssh-unix-dev mailing list openssh-unix-dev at mindrot.org https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20171018/fc0a3ab7/attachment-0001.p7s>
Hello, On Wed, Oct 18, 2017 at 6:36 PM, Ingo Schwarze <schwarze at usta.de> wrote:> Hi Emmanuel, > > Emmanuel Deloget wrote on Wed, Oct 18, 2017 at 05:45:40PM +0200:<snip>>> I'd volonteer to do this but given the gigantism of the task, any help >> would be appreciated (not to mention that even if Joel agrees with the >> idea of having the 1.1 API in LibreSSL, that does not mean he would >> agree with my strategy). > > You could offer your help for providing that migration path to the > OpenSSL project, if you feel qualified and willing to provide such > help. I think you should definitely use the expertise of the OpenSSL > developers to make sure that providing the migration path doesn't > introduce bugs and vulnerabilities, and avoid working as a lone > wolf if possible.Working as a lone wolf on this project is definitely not what I want to do. However, I can propose the first (simple) patches and hopefully someone else will join the effort once it has started :)>> But at the same time, some ditributions are phasing out OpenSSL 1.0.2 >> and are integrating OpenSSL 1.1 while at the same time some supported >> LTS distributions still rely on older OpenSSL versions. So for a >> period of time you'll have to be compatible with both, whether you >> like it or not. The question is then: is it up to you to be compatible >> with both, > > The OpenSSH project simply doesn't have the manpower to maintain > compatibility with both at the same time, and even if it had, doing > so would compromise the central project goal of security through > simplicity and diligence.Ok.>> or is it up to distributors to provide compatibilty? > > That will be beyond the means of many distributors, in which case > they either need to use OpenSSL-1.0 or LibreSSL for OpenSSH. > > Even for those distributors who can afford the required manpower, > i consider attempting to maintain a 3000+ line vendor patch risky, > and maintaining a non-official compat layer risky as well. Quite > respectable vendors have screwed up with three-line patches in the > past, how can anybody feel confident about a 3000-line vendor patch? > How sure can you feel that none of these 3000 lines causes non-obvious > problems? That also answers the question Jakub Jelen asked why i > would feel uncomfortable about using Fedora. > > Yours, > IngoLet's restate the full goal here : * first, LibreSSL should integrate the OpenSSL 1.1 API * second, it should mark the replaced interface as deprecated * third, make OpenSSH use the new interface That sounds like a manageable list of goals -- i.e. one that can prove to be completed before the EOL of OpenSSL 1.0.2 in Dec. 2019. First problem to overcome is that LibreSSL is released every 6 month, so the work is probably going to be split on two or more releases. As a consequence, the transition work cannot start on OpenSSH before the next LibreSSL release (i.e. version 2.7.0 that should be released in May) and will probably take a long time. For distributors and maintainers out there, that means: * distributions that decided to phase out OpenSSL 1.0.2 in favor of OpenSSL 1.1 will have to support an OpenSSH patch that will grow smaller and smaller at each release. * distributions that decided to keep OpenSSL 1.0.2 or to link OpenSSH with LibreSSL have nothing to do, and will be ready to phase out OpenSSL 1.0.2 at EOL time. Is it something that apear to be manageable? BTW, do you know any other software that uses the LibreSSL API the same way OpenSSH is using it (i.e. LibreSSL is the primary target, OpenSSL compatibility is of lesser importance)? BR, -- Emmanuel Deloget