On Mon, Oct 16, 2017 at 12:40:54AM +0200, Ingo Schwarze wrote:> Colin Watson wrote on Sun, Oct 15, 2017 at 10:51:46PM +0100: > > Is it actually a requirement that an API compatibility layer be > > maintained by the OpenSSL team, or could a hypothetical group of > > external developers interested in breaking this stalemate fork > > openssl-compat.tar.gz, stick it in a git repository somewhere, and start > > making versioned releases and trying to address the other problems you > > describe? > > At the risk of telling you something that you already know, OpenSSL > is both a large and a complicated library, and even though some > parts of the compatibility layer may seem relatively simple on first > sight, that doesn't imply there are no non-obvious pitfalls, and > if somebody who is only partly qualified or only understands parts > of the code makes seemingly simple changes in code related to > cryptography, that's exactly how some serious vulnerabilities in > both OpenSSL and in some non-OpenBSD ports of OpenSSH were caused > in the past. > > So the hypothetical group you wish for would also have to be > qualified, and there are not very many people in the world who > would qualify to lead such a group, i fear. There are certainly > some inside the OpenSSL project; at least they are likely to > know the code, to know how they changed it and what that implies, > and likely already aware of some of the potential pitfalls.OK. So that still basically leaves us at an impasse, because OpenSSH will only use a compatibility layer with a sufficiently good maintenance team, and the OpenSSL team (as far as I can tell from public statements) don't want to be on the hook for that. Which leads me back to my previous question: what conversations have there been between the OpenSSH and OpenSSL developers about this problem? Has OpenSSL upstream actually been told directly by OpenSSH that this is a problem, or are they only hearing about this from users trying to compile OpenSSH against 1.1? I've only found evidence of the latter in public mailing list posts so far.> [ about the option of building OpenSSH against LibreSSL on Debian ] > > > This would be a pretty bad option for me as a distributor - it'd > > mean I'd have to keep track of LibreSSL security updates. > > In the past, that has proven noticeably less stressful than keeping > track of OpenSSL security updates, and i think it is safe to say > that it would be orders of magnitude less stressful and less dangerous > *for an outside party* than engineering and maintaining a compatibility > library.That's as may be, but (a) I don't have to keep track of *either* OpenSSL or LibreSSL updates right now in general, (b) our distribution policy is generally that we strenuously avoid using bundled copies of code. Fedora has the same policy, and so far has opted to ship a ~3600-line patch to OpenSSH to use the 1.1 API. That patch will surely get substantially smaller once they're on 7.6p1 and can drop all the protocol 1 bits, but even so, I *really* dislike the option of using a non-upstreamed patch for that, which is why I'm trying to find a better option that's compatible with our policies before my hand is forced by OpenSSL 1.0 build support being removed from Debian, which is likely to be before the end of the year. If my only other option is to use LibreSSL, then that will mean packaging LibreSSL separately, and https://bugs.debian.org/754513 seems to have petered out a couple of years ago, not to mention being a pile of work I really don't have time for as well as requiring overcoming non-trivial objections. I realise that this is not the OpenSSH team's problem as such, and that as a LibreSSL developer you may well not be super-sympathetic to this argument; but nevertheless, I don't think this is a viable option right now for us as a distributor. -- Colin Watson [cjwatson at debian.org]
Hi Colin, Colin Watson wrote on Mon, Oct 16, 2017 at 10:26:03AM +0100:> Which leads me back to my previous question: what conversations have > there been between the OpenSSH and OpenSSL developers about this > problem? Has OpenSSL upstream actually been told directly by OpenSSH > that this is a problem, or are they only hearing about this from users > trying to compile OpenSSH against 1.1? I've only found evidence of the > latter in public mailing list posts so far.I'm not completely sure, i don't know all the private communications either. But note two things: First, communication problems in cases where OpenSSL was the interested party and OpenBSD developers were trying to help played a part (even though not the only part) in the decision of forking LibreSSL. There had been bug reports from OpenBSD to OpenSSL that went nowhere for considerable times. Similarly, while i committed little code to OpenSSH (mainly in one small corner, UTF-8 safety) and no code to LibreSSL, i have done substantial work on LibreSSL documentation, and much of that could also be useful for OpenSSL, as an OpenSSL developer privately confirmed to me. I spent a bit of time (some hours) preparing a set of patches (not against their code, but applying conflicting documentation patches by hand is certainly easier than applying conflicting code patches) and sent them directly to OpenSSL more than half a year ago. Last time i checked (when doing my latest merge of documentation improvements from OpenSSL to LibreSSL), none of that had been applied yet. In the case at hand, we would be asking them to do substantial work that helps us and that they seem to consider not that important for their own purposes. That is likely to work out even worse than doing some work ourselves that is not really needed for our own purposes and mainly intended to help them. Let's put it this way: As an example, my personal direct communication with OpenSSL members was always polite and friendly, but rarely led to tangible results. Well, maybe there were one or two trivial typo fixes applied some years ago.> (b) our distribution policy is generally that we strenuously avoid > using bundled copies of code.For what it's worth, i actually consider that a good policy in general. The OpenBSD ports tree usually aims for the same goal. Admittedly, in some cases, exceptions are very hard to avoid. For example, at certain times, the porting team couldn't avoid using bundled SQLite in firefox, but so they had to switch back and forth a few times in that respect.> Fedora has the same policy, and so far has opted to ship a ~3600-line > patch to OpenSSH to use the 1.1 API.Frankly, i would feel uncomfortable using OpenSSH on Fedora.> If my only other option is to use LibreSSL, then that will mean > packaging LibreSSL separately, and https://bugs.debian.org/754513 > seems to have petered out a couple of years ago,Reading that thread, my impression is that the main reason is that the question "what is this needed for" was never fully answered. You don't really have to package a library that nothing is using yet. Sure, there were also some technical issues raised, but the thread seems generally constructive to me, even if back then, nobody was in enough of a fix to actually put in the required work. I imagine if you, as the SSH maintainer, spoke up and said: "OpenSSH requires an OpenSSL-1.0 compatible API, so we must have either an OpenSSL-1.0 or a LibreSSL package in Debian" that might carry some weight and may either make people think again about deleting OpenSSL-1.0 or revitalize the thread about LibreSSL. Doesn't Debian have a policy that established APIs supported upstream cannot be deleted while important software still uses them?> I realise that this is not the OpenSSH team's > problem as such, and that as a LibreSSL developer you may well not be > super-sympathetic to this argument; but nevertheless, I don't think this > is a viable option right now for us as a distributor.I completely understand that you are in a difficult situation and that you like none of the options you have: (1) package LibreSSL (2) bundle LibreSSL (3) keep the existing OpenSSL-1.0 package (still supported upstream) Until somebody sufficiently qualified maintains a compat library, *and* LibreSSL gains 1.1-compatible interfaces *and* OpenSSH switches over (three large items lacking volunteers, which consequently seem very unlikely to be completed by the end of the year), these three are the only options i can see. Yours, Ingo
On Mon, Oct 16, 2017 at 05:18:59PM +0200, Ingo Schwarze wrote:> Colin Watson wrote on Mon, Oct 16, 2017 at 10:26:03AM +0100: > > Which leads me back to my previous question: what conversations have > > there been between the OpenSSH and OpenSSL developers about this > > problem? Has OpenSSL upstream actually been told directly by OpenSSH > > that this is a problem, or are they only hearing about this from users > > trying to compile OpenSSH against 1.1? I've only found evidence of the > > latter in public mailing list posts so far. > > I'm not completely sure, i don't know all the private communications > either.Right, that's why I keep asking. :-) I'm not interested in rehashing communication problems, just trying to find out what the state of play is; for example if the answer is that nobody has really properly brought this to the OpenSSL developers then an obvious way in which I could at least try to improve the situation would be to do so! (It might not solve the problem, but it would be better than for example assuming they're aware of it when they aren't.)> > If my only other option is to use LibreSSL, then that will mean > > packaging LibreSSL separately, and https://bugs.debian.org/754513 > > seems to have petered out a couple of years ago, > > Reading that thread, my impression is that the main reason is that > the question "what is this needed for" was never fully answered. > You don't really have to package a library that nothing is using yet. > Sure, there were also some technical issues raised, but the thread > seems generally constructive to me, even if back then, nobody was > in enough of a fix to actually put in the required work. > > I imagine if you, as the SSH maintainer, spoke up and said: "OpenSSH > requires an OpenSSL-1.0 compatible API, so we must have either an > OpenSSL-1.0 or a LibreSSL package in Debian" that might carry some > weight and may either make people think again about deleting > OpenSSL-1.0 or revitalize the thread about LibreSSL.That was indeed on my list of things to do; I've replied to the bug above with a summary of the situation, and copied debian-devel.> Doesn't Debian have a policy that established APIs supported upstream > cannot be deleted while important software still uses them?Well, in general, sure, that sort of thing is enforced by what amounts to distribution-wide CI. But the situation would still amount to a release-critical bug *somewhere*, and I could well end up in a situation where I can't effectively upload new changes until it's sorted out.> I completely understand that you are in a difficult situation and > that you like none of the options you have: > > (1) package LibreSSL > (2) bundle LibreSSL > (3) keep the existing OpenSSL-1.0 package (still supported upstream)That's an accurate summary, yes. For completeness, I do think that (4) apply the patch from Kurt/Fedora is an option. I don't think it's a good option, both for the reasons you have and for the reason that it would be a ton of ongoing maintenance work for me, but it is an option. We'll see how the debian-devel conversation works out, but I would like to know the communication status so that (if necessary) I can also appeal to the OpenSSL folks. -- Colin Watson [cjwatson at debian.org]
Hello everyone, On Mon, Oct 16, 2017 at 5:18 PM, Ingo Schwarze <schwarze at usta.de> wrote:> Hi Colin, > > Colin Watson wrote on Mon, Oct 16, 2017 at 10:26:03AM +0100: >><snip>>> I realise that this is not the OpenSSH team's >> problem as such, and that as a LibreSSL developer you may well not be >> super-sympathetic to this argument; but nevertheless, I don't think this >> is a viable option right now for us as a distributor. > > I completely understand that you are in a difficult situation and > that you like none of the options you have: > > (1) package LibreSSL > (2) bundle LibreSSL > (3) keep the existing OpenSSL-1.0 package (still supported upstream) > > Until somebody sufficiently qualified maintains a compat library, > *and* LibreSSL gains 1.1-compatible interfaces *and* OpenSSH switches > over (three large items lacking volunteers, which consequently seem > very unlikely to be completed by the end of the year), these three > are the only options i can see.Let's restate these in numbered bullet points: (a) somebody sufficiently qualified maintains a compat library (b) LibreSSL gains 1.1-compatible interfaces (c) OpenSSH switches over I'm not sure point (b) is necessary. The goal of the shim is to emulate the OpenSSL 1.1 interface by encapsulating OpenSSL 1.0 / LibreSSL code, so no change is needed in the upstream library (that would make the change really impossible IMHO). So the problem goes down to 2 point: (a) and (c).>From my experience, (a) is not that problematic. The goal is not toadd any cryptographic code in the shim. Doing that would be both quite dangerous and not very productive since downstream users are not using theses (new) features anyway. The goal for now is to use the OpenSSL 1.1 API, not OpenSSL 1.1 itself. Good news: the shim code is quite neutral (w.r.t. security) and often very simple. If you really give a look to the Fedora patch [1] (and I encourage everyone who's serious about porting OpenSSL to OpensSSL 1.1 to do so, without judging the quelity of the patch itself), you'll see that the shim itself is less than 550 lines long. Most of the patch deals with changing the OpenSSH code to match the new API (some of the functions used in this patch are older and are already present in OpenSSL 1.0 and LibreSSL). At a first glance, this shim code seems to be mostly extracted from the OpenSSL 1.1 code and is trivial in many cases. There are parts that require some attention (again, w.r.t. security) but even those are quite simple (most of these if not all are related to memory management). In any case it does not require the mastery of the crypto code in OpenSSL. Thus I would say that the real problematic point is point (c): is the OpenSSL team comfortable with changing the OpenSSH code to support this API? Changing openssh-portable but not openssh means the two versions are likely to diverge at some point, so the wise among us would call for a global change (i.e. make openssh compatible with the OpenSSL 1.1 API and port the changes to openssh-portable). As I understood it, this would be problematic because openssh does not target OpenSSL at all, so that would introduce a number of changes that are not needed into the code base. The change is not as urgent as it may seem: OpenSSL 1.0.2 is to be supported for another 14 monthes so unless distributions want to phase it out earlier, there is still time to do it (I do understand that some distributions are phasing out OpenSSL 1.0).> Yours, > IngoBR, -- Emmanuel Deloget [1] http://pkgs.fedoraproject.org/cgit/rpms/openssh.git/tree/openssh-7.3p1-openssl-1.1.0.patch
On Mon, 16 Oct 2017, Colin Watson wrote:> If my only other option is to use LibreSSL, then that will mean > packaging LibreSSL separately, and https://bugs.debian.org/754513 seems > to have petered out a couple of years ago, not to mention being a pile > of work I really don't have time for as well as requiring overcoming > non-trivial objections. I realise that this is not the OpenSSH team's > problem as such, and that as a LibreSSL developer you may well not be > super-sympathetic to this argument; but nevertheless, I don't think this > is a viable option right now for us as a distributor.I'm sorry to have put you in this situation, but we have an upstream who is LibreSSL exclusively, a need to support LibreSSL and BoringSSL in the portable version and limited time and resources of our own. Even adopting the use of shims that give us the OpenSSL 1.1.x API means considerable additional work for us, because OpenBSD doesn't use that API. I'm willing to do it, but not if I'm going to be fighting the shims themselves along the way. -d
On Mon, 2017-10-16 at 17:18 +0200, Ingo Schwarze wrote:> > Fedora has the same policy, and so far has opted to ship a ~3600- > > line > > patch to OpenSSH to use the 1.1 API. > > Frankly, i would feel uncomfortable using OpenSSH on Fedora.Thank you for the support. Do you have any real reason to say so? Yes, we opted to improve existing patch, implement missing parts, test it and contribute it back to OpenSSH upstream in spite of moving forward with OpenSSL upstream. It takes some effort to do so, but we do not have to think about bundling LibreSSL nor depend on soon-to-by-outdated OpenSSL. As these threads appear all over and over again on this list, Fedora is not the only distro that had this problem and would like to see it resolved in a sensible way, but it is stalled in this point for over a year. Regards, -- Jakub Jelen Software Engineer Security Technologies Red Hat, Inc.
On Tue, Oct 17, 2017 at 09:39:50AM +1100, Damien Miller wrote:> On Mon, 16 Oct 2017, Colin Watson wrote: > > If my only other option is to use LibreSSL, then that will mean > > packaging LibreSSL separately, and https://bugs.debian.org/754513 seems > > to have petered out a couple of years ago, not to mention being a pile > > of work I really don't have time for as well as requiring overcoming > > non-trivial objections. I realise that this is not the OpenSSH team's > > problem as such, and that as a LibreSSL developer you may well not be > > super-sympathetic to this argument; but nevertheless, I don't think this > > is a viable option right now for us as a distributor. > > I'm sorry to have put you in this situation, but we have an upstream who > is LibreSSL exclusively, a need to support LibreSSL and BoringSSL in the > portable version and limited time and resources of our own. > > Even adopting the use of shims that give us the OpenSSL 1.1.x API means > considerable additional work for us, because OpenBSD doesn't use that > API. I'm willing to do it, but not if I'm going to be fighting the shims > themselves along the way.The discussion on debian-devel seemed to indicate that embedding a copy of LibreSSL might actually end up being an approach we could live with for now, since it would mean that we don't have to worry about whether LibreSSL's support cycles align with Debian's. I didn't get unanimity on this, but there was more consensus than I expected. Have you done any more work on https://lists.mindrot.org/pipermail/openssh-unix-dev/2017-October/036346.html as yet? It's probably worth mentioning sooner rather than later that anything that involved fetching something from the network at build time wouldn't work for us; perhaps embedding a copy of (the relevant parts of) LibreSSL would be possible though? On a somewhat separate note, I still need to work out what to do about openssh-ssh1, which is the copy of 7.5p1 that I split out to a separate source package in Debian as described in https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-May/035070.html. We still need to be able to build that even after we stop supporting OpenSSL 1.0. My current thought, reversing my previous opinion, is that it may actually be best to apply the patch set from Kurt and Fedora for OpenSSL 1.1 support *only* for openssh-ssh1. My rationale is: * I can't imagine that there's any appetite among OpenSSH developers for issuing a 7.5p2 with an embedded LibreSSL just for the sake of the obsolete protocol that you explicitly want to stop spending time on. * Distro-patching 7.5p1 to add an embedded copy of LibreSSL would be an even more gigantic patch than the Fedora one, and not clearly less of a headache for me. We could reasonably debate whether it would be more or less prone to failure. * I want to spend as little of my time as possible keeping openssh-ssh1 on life support, and the Fedora patch exists today while other options require more (even if not necessarily much more) work. * The difficulty of accurately forward-porting Fedora's patch to newer upstream versions doesn't apply in the case of openssh-ssh1, as there will be no new upstream versions. * openssh-ssh1 is client-only, reducing the scope of possible problems. * Acknowledging Ingo's views on the Fedora patch in https://lists.mindrot.org/pipermail/openssh-unix-dev/2017-October/036365.html, nobody security-conscious is going to be using protocol 1 on a public network anyway, since it's already known to be broken. The only reasonable way to use it is as a glorified telnet on something like a private management network to talk to devices that don't speak anything else and can't be upgraded. In that context, an error in the OpenSSL 1.1 support patch is not going to have catastrophic consequences. This is an opportunity for people to tell me why that line of reasoning is wrong. -- Colin Watson [cjwatson at debian.org]