llvm dev - Dec 2019 - [RFC] LLVM Security Group and Process

On Tue, Nov 19, 2019 at 10:46 AM JF Bastien <jfbastien at apple.com>
wrote:
> And I do agree that if someone were to come in and put in the significant
> amounts of work to make LLVM directly usable in security-sensitive places,
> then we could support that. But none of that should have anything to do
> with the security group or its membership. All of that work and discussion,
> and the decision to support it in the end, should be done as a project-wide
> discussion and decision, just like anything else that's worked on.
>
>
> Here’s where we disagree: how to get from nothing being security to the
> right things being security.
>
> I want to put that power in the hands of the security group, because
> they’d be the ones with experience handling security issues, defining
> security boundaries, fixing issues in those boundaries, etc. I’m worried
> that the community as a whole would legislate things as needing to be
> secure, without anyone in the security group able or willing to make it so.
> That’s an undesirable outcome because it sets them up for failure.
>
> Of course neither of us is saying that the community should dictate to the
> security group, nor that the security group should dictate to the
> community. It should be a discussion. I agree with you that, in that
> transition period from no security to right security there might be cases
> where the security group disappoints the community, behind temporarily
> closed doors. There might be mistakes, an issue which should have been
> treated as security related won’t be. I would rather trust the security
> group, expect that it’ll do outreach when it feels unqualified to handle an
> issue, and fix any mistakes it makes if it happens. Doing so is better than
> where we are today.
>
My worry is actually the inverse -- that there may be a tendency to treat
more issues as "security" than should be. When some bug is reported
via the
security process, I suspect there will be a default-presumption towards
using the security process to resolve it, with all the downsides that go
along with that.

What I want is for it to be clear that certain kinds of issues are
currently explicitly out-of-scope. E.g. crashes/code-execution/etc
resulting from parsing or compiling untrusted C source-code with Clang, or
parsing/compiling untrusted bitcode with LLVM, or linking untrusted files
with LLD. These sorts of things should not, currently, be treated with a
"security" mindset. They're *bugs*, which should be fixed, but if
something's security depends on llvm being able to securely process
untrusted inputs, sorry, that's not reasonable. (And yes, that's maybe
sad,
but is the reality right now). Until someone is willing to put in the
significant effort to make those processes generally secure for use on
untrusted inputs, handling individual bug-reports of this kind via a
special process is not going to realistically improve security.

Furthermore, if people disagree with the above paragraph, I'd like that
discussion to be had in the open ahead of time, rather than in private
after the first time such an issue is reported via a defined security
process.

It feels like you want the security team to be two different things:
1. A way to privately report security issues to LLVM, and a group of people
to privately work on fixing such issues, for a coordinated release.
2. A group of people working generally on defining or improving security
properties of LLVM.

I don't think these two need or should be linked, though some of the same
people might be involved in both.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191125/2f6ecb65/attachment.html>

On Mon, 25 Nov 2019 at 15:36, James Y Knight via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> What I want is for it to be clear that certain kinds of issues are
currently explicitly out-of-scope. E.g. crashes/code-execution/etc resulting
from parsing or compiling untrusted C source-code with Clang, or
parsing/compiling untrusted bitcode with LLVM, or linking untrusted files with
LLD. These sorts of things should not, currently, be treated with a
"security" mindset. They're bugs, which should be fixed, but if
something's security depends on llvm being able to securely process
untrusted inputs, sorry, that's not reasonable. (And yes, that's maybe
sad, but is the reality right now). Until someone is willing to put in the
significant effort to make those processes generally secure for use on untrusted
inputs, handling individual bug-reports of this kind via a special process is
not going to realistically improve security.
I agree 100% with this. LLVM is not secure in that way. Treating that
kind of report as a serious security issue would just be security
theatre and give the impression the project is developed with
different goals than many people writing the code (including me)
actually have. Moving to a world where it's reasonable to make those
crashes a security issue won't even be substantially helped by that
kind of whack-a-mole approach.

Cheers.

Tim.

> On Nov 25, 2019, at 7:36 AM, James Y Knight <jyknight at google.com>
wrote:
> 
> 
> 
> On Tue, Nov 19, 2019 at 10:46 AM JF Bastien <jfbastien at apple.com
<mailto:jfbastien at apple.com>> wrote:
>> And I do agree that if someone were to come in and put in the
significant amounts of work to make LLVM directly usable in security-sensitive
places, then we could support that. But none of that should have anything to do
with the security group or its membership. All of that work and discussion, and
the decision to support it in the end, should be done as a project-wide
discussion and decision, just like anything else that's worked on.
> 
> Here’s where we disagree: how to get from nothing being security to the
right things being security.
> 
> I want to put that power in the hands of the security group, because they’d
be the ones with experience handling security issues, defining security
boundaries, fixing issues in those boundaries, etc. I’m worried that the
community as a whole would legislate things as needing to be secure, without
anyone in the security group able or willing to make it so. That’s an
undesirable outcome because it sets them up for failure.
> 
> Of course neither of us is saying that the community should dictate to the
security group, nor that the security group should dictate to the community. It
should be a discussion. I agree with you that, in that transition period from no
security to right security there might be cases where the security group
disappoints the community, behind temporarily closed doors. There might be
mistakes, an issue which should have been treated as security related won’t be.
I would rather trust the security group, expect that it’ll do outreach when it
feels unqualified to handle an issue, and fix any mistakes it makes if it
happens. Doing so is better than where we are today.
> 
> My worry is actually the inverse -- that there may be a tendency to treat
more issues as "security" than should be. When some bug is reported
via the security process, I suspect there will be a default-presumption towards
using the security process to resolve it, with all the downsides that go along
with that.
Agreed, that polarity is also a risk. I don’t see how to fix this issue either,
except to trust the security group. Its members will be more competent at doing
the right thing than the general LLVM community because they’ve dealt with this
stuff before.

> What I want is for it to be clear that certain kinds of issues are
currently explicitly out-of-scope.
Yes I want this list, but I don’t think we need it now. Once we’ve got a group
of experts looking at security issues they can incrementally figure out that
list. Do you think that’s acceptable?

> E.g. crashes/code-execution/etc resulting from parsing or compiling
untrusted C source-code with Clang, or parsing/compiling untrusted bitcode with
LLVM, or linking untrusted files with LLD. These sorts of things should not,
currently, be treated with a "security" mindset. They're bugs,
which should be fixed, but if something's security depends on llvm being
able to securely process untrusted inputs, sorry, that's not reasonable.
(And yes, that's maybe sad, but is the reality right now). Until someone is
willing to put in the significant effort to make those processes generally
secure for use on untrusted inputs, handling individual bug-reports of this kind
via a special process is not going to realistically improve security.
> 
> Furthermore, if people disagree with the above paragraph, I'd like that
discussion to be had in the open ahead of time, rather than in private after the
first time such an issue is reported via a defined security process.
> 
> It feels like you want the security team to be two different things:
> 1. A way to privately report security issues to LLVM, and a group of people
to privately work on fixing such issues, for a coordinated release.
> 2. A group of people working generally on defining or improving security
properties of LLVM.
> 
> I don't think these two need or should be linked, though some of the
same people might be involved in both.
I don’t want 1 and 2 linked, though as you say it can be the same group. I’m
saying that 2 should inform 1, they don’t exist in a vacuum.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191125/e494c4ef/attachment.html>

On Mon, Nov 25, 2019 at 5:38 PM JF Bastien <jfbastien at apple.com> wrote:
>
>
> On Nov 25, 2019, at 7:36 AM, James Y Knight <jyknight at google.com>
wrote:
>
>
>
> On Tue, Nov 19, 2019 at 10:46 AM JF Bastien <jfbastien at apple.com>
wrote:
>
>> And I do agree that if someone were to come in and put in the
significant
>> amounts of work to make LLVM directly usable in security-sensitive
places,
>> then we could support that. But none of that should have anything to do
>> with the security group or its membership. All of that work and
discussion,
>> and the decision to support it in the end, should be done as a
project-wide
>> discussion and decision, just like anything else that's worked on.
>>
>>
>> Here’s where we disagree: how to get from nothing being security to the
>> right things being security.
>>
>> I want to put that power in the hands of the security group, because
>> they’d be the ones with experience handling security issues, defining
>> security boundaries, fixing issues in those boundaries, etc. I’m
worried
>> that the community as a whole would legislate things as needing to be
>> secure, without anyone in the security group able or willing to make it
so.
>> That’s an undesirable outcome because it sets them up for failure.
>>
>> Of course neither of us is saying that the community should dictate to
>> the security group, nor that the security group should dictate to the
>> community. It should be a discussion. I agree with you that, in that
>> transition period from no security to right security there might be
cases
>> where the security group disappoints the community, behind temporarily
>> closed doors. There might be mistakes, an issue which should have been
>> treated as security related won’t be. I would rather trust the security
>> group, expect that it’ll do outreach when it feels unqualified to
handle an
>> issue, and fix any mistakes it makes if it happens. Doing so is better
than
>> where we are today.
>>
>
> My worry is actually the inverse -- that there may be a tendency to treat
> more issues as "security" than should be. When some bug is
reported via the
> security process, I suspect there will be a default-presumption towards
> using the security process to resolve it, with all the downsides that go
> along with that.
>
>
> Agreed, that polarity is also a risk. I don’t see how to fix this issue
> either, except to trust the security group. Its members will be more
> competent at doing the right thing than the general LLVM community because
> they’ve dealt with this stuff before.
>
Again, I find it entirely reasonable to place trust in a small subset of
the members of the LLVM community to do the right thing in response to
security issues which must remain temporarily secret. It's infeasible to
allow the entire community to participate. I just don't want to entrust
anything *else* to the Security Group, as an organization, because it's
unnecessary (despite that they would likely be entirely worthy of that
trust).
> What I want is for it to be clear that certain kinds of issues are
> currently explicitly out-of-scope.
>
> Yes I want this list, but I don’t think we need it now. Once we’ve got a
> group of experts looking at security issues they can incrementally figure
> out that list. Do you think that’s acceptable?
>
We know now, even before any issues have been reported through this
process, what some of the areas of concern are going to be. Some have been
mentioned before on this thread, and others likely have not. I would like
to see it explicitly called out, up front, how we expect to treat certain
issues without waiting for them to be reported.

Why do I want that? Because I want the security group's mission statement
and mandate from the community to be clear. If there's disagreement about
which sorts of things should or should not be treated as security issues
(which I suspect there may well be), I'd like that to be hashed out in the
open now, rather than delaying any such debate until such a time as it
*must* be hashed out in private by the Security Group in response to a
concrete private vulnerability report.

However, I agree it's not necessary for you to define this immediately. If
you'd like to attempt to find other volunteers to author those policies,
rather than doing it yourself, I see absolutely no problem with that. But I
would still like to see such a document get proposed and reviewed via the
project's usual open discussion forum (mailing lists, code reviews on new
policy docs, etc), as soon as possible, in order to reduce surprises as
much as possible. (Recognizing that it cannot and should not attempt to
cover every eventuality.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191203/746adc88/attachment-0001.html>