Secure testing team - Oct 2009 - Vulnerability impact on issues

Hi,
I just had a chat with Raphael about the impact levels we currently set for 
vulnerabilities in the tracker. We both came to the conclusion that our 
current way of assigning that is rather sub-optimal.

At the moment we try to judge the impact, the bug type, the availability of 
the issue and our priority which often is not easy to connect and we end up 
with situations where it is very hard (not to say random) to set the impact.

Classifying security issues is a really hard task and known to be flawed. So I 
think it''s time to change what we are currently doing.

What about just setting what priority the issue has for us? We can''t
properly
classify the impact with three levels anyway.

Instead I propose we let the levels like they are but use them with the 
meaning of priority. The tracker already says urgency so we need to change our 
documentation regarding that and maybe optionally displaying the CVSS score 
might be helpful (I know this score is flawed as well but it''s better
than
none).

Opinions?

Cheers
Nico

-- 
Nico Golde - http://www.ngolde.de - nion at jabber.ccc.de - GPG: 0xA0A0AAAA
For security reasons, all text in this mail is double-rot13 encrypted.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20091028/e236c36a/attachment.pgp>

On Wed, Oct 28, 2009 at 06:05:00PM +0100, Nico Golde
wrote:
> Hi,
> I just had a chat with Raphael about the impact levels we currently set for
> vulnerabilities in the tracker. We both came to the conclusion that our 
> current way of assigning that is rather sub-optimal.
> 
> At the moment we try to judge the impact, the bug type, the availability of
> the issue and our priority which often is not easy to connect and we end up
> with situations where it is very hard (not to say random) to set the
impact.
> 
> Classifying security issues is a really hard task and known to be flawed.
So I
> think it''s time to change what we are currently doing.
> 
> What about just setting what priority the issue has for us? We
can''t properly
> classify the impact with three levels anyway.
> 
> Instead I propose we let the levels like they are but use them with the 
> meaning of priority. The tracker already says urgency so we need to change
our
> documentation regarding that and maybe optionally displaying the CVSS score
> might be helpful (I know this score is flawed as well but it''s
better than
> none).
Or let''s simply get rid of them at all.

Cheers,
        Moritz

Moritz Muehlenhoff wrote:
> 
> Or let''s simply get rid of them at all.
Keeping it would allow us (and others) to see what and how should issues be
prioritised.

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net

On Wed, Oct 28, 2009 at 02:58:28PM -0600, Raphael Geissert
wrote:
> Moritz Muehlenhoff wrote:
> > 
> > Or let''s simply get rid of them at all.
> 
> Keeping it would allow us (and others) to see what and how should issues be
> prioritised.
It''s done by current practice; if it''s important people work
on it, if not
it''s postponed. I don''t see it providing any use at all.

Cheers,
        Moritz

On Wed, 28 Oct 2009 14:58:28 -0600, Raphael Geissert
wrote:
> Moritz Muehlenhoff wrote:
> > 
> > Or let''s simply get rid of them at all.
> 
> Keeping it would allow us (and others) to see what and how should issues be
> prioritised.
i agree that prioritizing is useful.  it helps to decide which among
the present issues should be worked first.  it also makes it ok to
leave certain issues untouched for long periods of time (i.e. those with
low-urgency).  if issues were not categorized and left open for long
periods of time, it would look very bad, because without information to
the contrary, human tendency is to assume all things being equal, and
the conclusion drawn would be that many dangerous issues were being
left open.

i don''t think redefining the meaning of low/medium/high to be based
solely on debian priority will make a whole lot of a difference; because
ultimately priority is dependent on the facts currently known about the
issue anyway. i.e. a flaw known to be used in existing exploits should
have a higher priority than one where that fact is not yet known. 

i think that the current system is ok, but we shouldn''t be so worried
about getting the priority exactly right.  we also shouldn''t be too
staunch about keeping the priority in line with the current definitions
just because of the written text in the introduction (all language
is subject to interpretation anyway).

i would however like to encourage initial gut-feeling ratings for all
incoming issues; primarily because it is unclear how you would decide
whether to work on a non-prioritized issue over one of medium
priority.  and as progress is made, if that gut-feeling rating was
wrong, then the triager should feel free to change the rating to
reflect the currently known state.

mike

On Thu, 29 Oct 2009 08:15:41 am Moritz Muehlenhoff
wrote:
> On Wed, Oct 28, 2009 at 02:58:28PM -0600, Raphael Geissert wrote:
> > Moritz Muehlenhoff wrote:
> > > Or let''s simply get rid of them at all.
> >
> > Keeping it would allow us (and others) to see what and how should
issues
> > be prioritised.
> 
> It''s done by current practice; if it''s important people
work on it, if not
> it''s postponed. I don''t see it providing any use at all.
I am with Moritz here, if an issue is in serious need of fixing, it usually 
gets an RT ticket in one of the security queues (and quite often the first 
person that spots it and has the needed spare time fixes it).

Cheers
Steffen

Hi,
* Steffen Joeris <steffen.joeris at skolelinux.de> [2009-10-30
10:22]:
> On Thu, 29 Oct 2009 08:15:41 am Moritz Muehlenhoff wrote:
> > On Wed, Oct 28, 2009 at 02:58:28PM -0600, Raphael Geissert wrote:
> > > Moritz Muehlenhoff wrote:
> > > > Or let''s simply get rid of them at all.
> > >
> > > Keeping it would allow us (and others) to see what and how should
issues
> > > be prioritised.
> > 
> > It''s done by current practice; if it''s important
people work on it, if not
> > it''s postponed. I don''t see it providing any use at
all.
> I am with Moritz here, if an issue is in serious need of fixing, it usually
> gets an RT ticket in one of the security queues (and quite often the first 
> person that spots it and has the needed spare time fixes it).
About the RT ticket I have to disagree, this is still not wideley used, even I 
don''t use it all the time (yeah shame on me :) And it still helps
people who
have no yet dived into the specific issue to get an idea of what to look at in 
case they want to do something.

Cheers
Nico
-- 
Nico Golde - http://www.ngolde.de - nion at jabber.ccc.de - GPG: 0xA0A0AAAA
For security reasons, all text in this mail is double-rot13 encrypted.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20091030/e86cd5d4/attachment.pgp>

Steffen Joeris wrote:
> I am with Moritz here, if an issue is in serious need of fixing, it
> usually gets an RT ticket in one of the security queues (and quite often
> the first person that spots it and has the needed spare time fixes it).
> 
I think the process regarding RT tickets should be made a bit more public,
or at least documented. While this may work for stable and oldstable, it is
leaving unstable behind as an issue in unstable may not be addressed as
soon as in the stable releases.

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net

On 11/1/09, Raphael Geissert wrote:
> Steffen Joeris wrote:
>> I am with Moritz here, if an issue is in serious need of fixing, it
>> usually gets an RT ticket in one of the security queues (and quite
often
>> the first person that spots it and has the needed spare time fixes it).
>>
>
> I think the process regarding RT tickets should be made a bit more public,
> or at least documented. While this may work for stable and oldstable, it is
> leaving unstable behind as an issue in unstable may not be addressed as
> soon as in the stable releases.
>
> Cheers,
hi all,

i was thinking about this problem some more recently and came up with
the following conjecture: perhaps the key to making urgencies more
useful is to define them as time-based; rather than
qualification-based.

for example, the definitions could be plain and simple:

- high: an issue for which a fix optimally needs to be issued within
the next 24 hours (with a 36 hour maximum).  this category is
primarily reserved for issues that are known to be currently exploited
in the wild, weaknesses in cryptographic systems, and recently
disclosed vendor-sec issues.
- medium: an issue for which a fix optimally should be issued within
the next 14 days (with a 1 month maximum)
- low: an issue for which a fix optimally should be issued withn the
next 6 months (with a 1 year maximum).

obviously this categorization still remains very subjective, which is
unavoidable for any system due to the sheer ammount of unkowns related
to security issues. the key benefit is that these categories define
very clear and useful objectives/goals for the security team to meet,
rather than the current lack of any objectives, which would be very
useful.  note that the timeframes specified above are open for debate;
those were just my initial thoughts.

note that in this scheme, leaving an issue uncategorized would be
undesirable because there may be high and medium priority issues among
those that are undefined, so no one is aware that they should be
prioritizing their work to get those issues fixed in the desired
timeframes.

it may also be useful to set up metrics in the tracker to see how well
those objectives are being met down the road.

as an aside, it may also be useful to do more claiming of issues so we
know who is already working an issue.  this would be especially useful
if there are to be goals for getting the work done (so others know
when to step in when something is falling through the cracks).  i''ve
been thinking about working on the new xpdf issues for a while, but
i''m not sure if someone else is already on it or not.  there is some
documentation in the intro about claiming a section of issues in the
CVE list, but i''ve never seen that used.  maybe it would be more
straightforward/useful if the claims were on a per-package basis,
rather than full sections.  for example, i could claim an issue by
inserting my name into the parenthesis for the packages that i am
working on, e.g.:

   - xpdf <unfixed> (low; gilbert-guest)

mike