Secure testing team - Feb 2011 - For discussion: security support strategy for the wheezy kernel

Hi,

Now that squeeze is released, I''ve started thinking about how to
improve the quality of security support for testing.  

The biggest problem I saw during the squeeze development cycle was that
kernel security update transitions were extremely slow due to unrelated
RC bugs.  This was bad since it left testing users vulnerable to issues
for much larger periods of time than stable/unstable users.  

Another issue was that a lot of vulnerabilities that were found in that
time frame were actually flaws in new kernel code, so testing/unstable
were vulnerable, but the stable kernel itself was unaffected, so it was
a bit safer to be running the stable kernel since the code was older
(i.e. there was more time to find and fix issues).  For example, the
vulnerability for one of the local privilege escalations that was
exploited in the wild was introduced in 2.6.30, so lenny wasn''t
affected, but testing/unstable were.

I''d like to propose a solution to these two problems: only upload known
rock solid longterm cadence upstream supported kernels into
testing/unstable. This will hopefully reduce the amount of transition
delay since the longterm kernels should have fewer RC issues (after
they''ve had a little time to cook of course).

There are, of course, some undesirable consequences to this plan.  One
is that a certain subset of testing users expect the latest shiny all
the time; but they can easily get their fix from the experimental
kernels instead, so that isn''t really a problem (I think).

The second issue is that testing d-i will not support the latest and
greatest hardware and features.  I think this can be solved by
providing two sets of d-i media for testing (one that uses the longterm
testing kernel and one that uses newer experimental kernel).  Of course
this means that some d-i development will need to move to experimental
to make use of the newer kernel feature, but I don''t think that would
really be a problem.

A benefit to this proposal is that there will be reduced work for those
currently supporting kernel security updates since the same package can
be uploaded to both stable-security and unstable.  Also, there should
be no RC issues that prevent transitions to testing since for example
the 2.6.32 kernel is so well-tested already.

Anyway, I think this would go a long way toward improving the quality
of security support in testing and thus reducing the common advice/meme
that users should avoid testing if they''re concerned about security.

So, my proposal in a nutshell is to only upload upstream 2.6.32 point
releases to wheezy/sid for the next 12-18 months.  At that time,
reevaluate and determine what the next longterm cadence kernel will be,
and then once that is reasonable stabilized in experimental, finally
upload it to unstable for the final stages of wheezy development
(perhaps a few months before the freeze).

Looking forward to thoughts and discussion on the matter.

Best wishes,
Mike

Disclaimer: note that I haven''t participated in kernel packaging or
applied kernel security patches directly myself (yet), but I have been
triaging and tracking kernel security issues for about a year and a
half now [0], so I have a pretty good understanding of the status quo.

[0] http://svn.debian.org/wsvn/kernel-sec/?op=log

Well, I have a few horses in this race in between testing-security,
d-i, and CUT, as well as having been a Release Assistant in the past,
and I think this proposal stinks from every perspective.

Michael Gilbert wrote:
> The biggest problem I saw during the squeeze development cycle was that
> kernel security update transitions were extremely slow due to unrelated
> RC bugs.  This was bad since it left testing users vulnerable to issues
> for much larger periods of time than stable/unstable users.  
Which proves nicely that not all RC bugs are equally RC, and that
britney''s
old_rc_count <= new_rc_count
is a flawed heuristic whose main virtue is that it''s probably the best
a machine can do with the available information. Which is why the release
team has override files..
> Another issue was that a lot of vulnerabilities that were found in that
> time frame were actually flaws in new kernel code, so testing/unstable
> were vulnerable, but the stable kernel itself was unaffected, so it was
> a bit safer to be running the stable kernel since the code was older
> (i.e. there was more time to find and fix issues).  For example, the
> vulnerability for one of the local privilege escalations that was
> exploited in the wild was introduced in 2.6.30, so lenny wasn''t
> affected, but testing/unstable were.
LWN''s analysis of age of introduction of kernel security holes in 2010
doesn''t seem to agree? http://lwn.net/Articles/410606/

| So, in a sense, the above-mentioned kernel hacker was correct - an
| awful lot of the vulnerabilities fixed over the last year predate the
| git era, and are thus over five years old.

At best, you seem to be grossly oversimplifying. In the immortal words
of FaceBook, "it''s complicated".
> I''d like to propose a solution to these two problems: only upload
known
> rock solid longterm cadence upstream supported kernels into
> testing/unstable. This will hopefully reduce the amount of transition
> delay since the longterm kernels should have fewer RC issues (after
> they''ve had a little time to cook of course).
> 
> There are, of course, some undesirable consequences to this plan.  One
> is that a certain subset of testing users expect the latest shiny all
> the time; but they can easily get their fix from the experimental
> kernels instead, so that isn''t really a problem (I think).
I feel that unstable is heading in much too stable a direction already.
I think that Bdale mentioned this (as a possible negative feature of CUT)
at DebConf. If testing is as stale as stable, why would anyone want to
bother with CUT?
> The second issue is that testing d-i will not support the latest and
> greatest hardware and features.  I think this can be solved by
> providing two sets of d-i media for testing (one that uses the longterm
> testing kernel and one that uses newer experimental kernel).
This seems likely to increase the workload of the d-i (and CD..) team
significantly. And wouldn''t it result in either an installed system
that
used unstable''s old kernel and so didn''t work, or used
experimental''s
kernel and so didn''t auto-upgrade to fix security holes?

Also, what are teams like the X team to do, when their packages depend
on new kernel features? How are we supposed to ever integrate support
for the new kernel distribution-wide if unstable constantly has an old
version? Should the whole distribution lag years behind the state of the
art?
> Anyway, I think this would go a long way toward improving the quality
> of security support in testing and thus reducing the common advice/meme
> that users should avoid testing if they''re concerned about
security.
A meme that is not founded in truth, as you can see if you compare
existing known security holes in stable and in testing at most points in
the release cycle. Perhaps the project should consider how it allows
these types of myths to stand up when their foundations are so easily
disproven?

-- 
see shy jo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 828 bytes
Desc: Digital signature
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20110206/c76e7710/attachment.pgp>

Michael Gilbert <michael.s.gilbert at gmail.com>
schrieb:
> Hi,
> So, my proposal in a nutshell is to only upload upstream 2.6.32 point
> releases to wheezy/sid for the next 12-18 months.  At that time,
> reevaluate and determine what the next longterm cadence kernel will be,
> and then once that is reasonable stabilized in experimental, finally
> upload it to unstable for the final stages of wheezy development
> (perhaps a few months before the freeze).
No way. The idea of unstable is to get the current upstream code in
shape and that won''t be achieved with staying with an old kernel.

Whatever the technical solution to testing-security kernel might be,
it needs to be based on following the upstream kernel development.

Cheers,
        Moritz

On Mon, Feb 07, 2011 at 07:12:48PM +0100, Moritz M?hlenhoff
wrote:
> Michael Gilbert <michael.s.gilbert at gmail.com> schrieb:
> > Hi,
> 
> > So, my proposal in a nutshell is to only upload upstream 2.6.32 point
> > releases to wheezy/sid for the next 12-18 months.  At that time,
> > reevaluate and determine what the next longterm cadence kernel will
be,
> > and then once that is reasonable stabilized in experimental, finally
> > upload it to unstable for the final stages of wheezy development
> > (perhaps a few months before the freeze).
> 
> No way. The idea of unstable is to get the current upstream code in
> shape and that won''t be achieved with staying with an old kernel.
> 
> Whatever the technical solution to testing-security kernel might be,
> it needs to be based on following the upstream kernel development.
 
Totally agreed.  We should be tracking current upstream releases,
and not just in experimental (which can now be used for upstream
release candidates).

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
                                                              - Albert Camus

On Mon, 7 Feb 2011 19:12:48 +0100, Moritz M?hlenhoff
wrote:
> Michael Gilbert <michael.s.gilbert at gmail.com> schrieb:
> > Hi,
> 
> > So, my proposal in a nutshell is to only upload upstream 2.6.32 point
> > releases to wheezy/sid for the next 12-18 months.  At that time,
> > reevaluate and determine what the next longterm cadence kernel will
be,
> > and then once that is reasonable stabilized in experimental, finally
> > upload it to unstable for the final stages of wheezy development
> > (perhaps a few months before the freeze).
> 
> No way. The idea of unstable is to get the current upstream code in
> shape and that won''t be achieved with staying with an old kernel.
I''m not sure if there''s a precise definition of unstable other
than
the statement at [0], but it seems to be whatever teams make of it.
It''s perfectly ok to upload only stable versions (if that''s
what the
team wants to do), and its perfectly ok to upload the most unstable
thing ever, but then the consequences of that have to be dealt with.  I
guess what I''m saying is that each team can decide specifically how
they want to use unstable, so the kernel team can deviate from the
status quo if they decide to; that is if I can make a sufficiently
convincing argument.

Also, my suggestion does involve eventually moving to a newer kernel in
the wheezy development cycle; just a while from now, rather than
rushing in to things.
> Whatever the technical solution to testing-security kernel might be,
> it needs to be based on following the upstream kernel development.
2.6.32.x is in fact an upstream kernel currently being developed ;)

Best wishes,
Mike

[0] http://www.debian.org/releases/unstable/

2011/2/7 Ben Hutchings wrote:
> On Mon, Feb 07, 2011 at 07:12:48PM +0100, Moritz M?hlenhoff wrote:
>> Michael Gilbert <michael.s.gilbert at gmail.com> schrieb:
>> > Hi,
>>
>> > So, my proposal in a nutshell is to only upload upstream 2.6.32
point
>> > releases to wheezy/sid for the next 12-18 months. ?At that time,
>> > reevaluate and determine what the next longterm cadence kernel
will be,
>> > and then once that is reasonable stabilized in experimental,
finally
>> > upload it to unstable for the final stages of wheezy development
>> > (perhaps a few months before the freeze).
>>
>> No way. The idea of unstable is to get the current upstream code in
>> shape and that won''t be achieved with staying with an old
kernel.
>>
>> Whatever the technical solution to testing-security kernel might be,
>> it needs to be based on following the upstream kernel development.
>
> Totally agreed. ?We should be tracking current upstream releases,
> and not just in experimental (which can now be used for upstream
> release candidates).
What about introducing a new linux-2.6-stable kernel package as a
compromise?  That way users that want stability and security support
in testing continue to have that as an option.  Also, I will assume
responsibility as the maintainer, so there will be hopefully very
little impact to any other part of Debian.  Also, I can look at
generating d-i media for this kernel.

Any thoughts on that?  The only downside I can think of right away is
introducing a huge code copy into the archive.

Best wishes,
Mike

On Mon, Feb  7, 2011 at 16:28:31 -0500, Michael Gilbert wrote:
> On Mon, 7 Feb 2011 19:12:48 +0100, Moritz M?hlenhoff wrote:
> > Michael Gilbert <michael.s.gilbert at gmail.com> schrieb:
> > > Hi,
> > 
> > > So, my proposal in a nutshell is to only upload upstream 2.6.32
point
> > > releases to wheezy/sid for the next 12-18 months.  At that time,
> > > reevaluate and determine what the next longterm cadence kernel
will be,
> > > and then once that is reasonable stabilized in experimental,
finally
> > > upload it to unstable for the final stages of wheezy development
> > > (perhaps a few months before the freeze).
> > 
> > No way. The idea of unstable is to get the current upstream code in
> > shape and that won''t be achieved with staying with an old
kernel.
> 
> I''m not sure if there''s a precise definition of unstable
other than
> the statement at [0], but it seems to be whatever teams make of it.
unstable is where debian development work happens.
> It''s perfectly ok to upload only stable versions (if
that''s what the
> team wants to do), and its perfectly ok to upload the most unstable
> thing ever, but then the consequences of that have to be dealt with.  I
> guess what I''m saying is that each team can decide specifically
how
> they want to use unstable, so the kernel team can deviate from the
> status quo if they decide to; that is if I can make a sufficiently
> convincing argument.
> 
> Also, my suggestion does involve eventually moving to a newer kernel in
> the wheezy development cycle; just a while from now, rather than
> rushing in to things.
> 
What does that buy us?  It means instead of dealing with bugs on an
ongoing basis, you get them all at the same time and get to bisect along
many kernel versions at once instead of just one.  It means problems
don''t get reported (and fixed) upstream until it''s too late. 
It means
any package that could use a newer kernel interface doesn''t get any
testing.  I''m sure there''s plenty of others.
> > Whatever the technical solution to testing-security kernel might be,
> > it needs to be based on following the upstream kernel development.
> 
> 2.6.32.x is in fact an upstream kernel currently being developed ;)
> 
No it''s not.  Go read the definition of development.

I''m sorry, but your proposal is insane.

Cheers,
Julien
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20110207/5148342c/attachment.pgp>

On Mon, Feb 7, 2011 at 5:08 PM, Michael Gilbert wrote:
> What about introducing a new linux-2.6-stable kernel package as a
> compromise? ?That way users that want stability and security support
> in testing continue to have that as an option. ?Also, I will assume
> responsibility as the maintainer, so there will be hopefully very
> little impact to any other part of Debian. ?Also, I can look at
> generating d-i media for this kernel.
>
> Any thoughts on that? ?The only downside I can think of right away is
> introducing a huge code copy into the archive.
Also the same effect can be achieved by apt-pinning the squeeze kernel.

Best wishes,
Mike

On Mon, Feb 7, 2011 at 5:09 PM, Julien Cristau wrote:
> What does that buy us? ?It means instead of dealing with bugs on an
> ongoing basis, you get them all at the same time and get to bisect along
> many kernel versions at once instead of just one. ?It means problems
> don''t get reported (and fixed) upstream until it''s too
late. ?It means
> any package that could use a newer kernel interface doesn''t get
any
> testing. ?I''m sure there''s plenty of others.
Bugs can be submitted and dealt with in experimental just as well as
in unstable.
>> > Whatever the technical solution to testing-security kernel might
be,
>> > it needs to be based on following the upstream kernel development.
>>
>> 2.6.32.x is in fact an upstream kernel currently being developed ;)
>>
> No it''s not. ?Go read the definition of development.
>
> I''m sorry, but your proposal is insane.
Is this kind of negativity really necessary?  I''m trying to guide a
discussion on a real problem, and I''m an engineer, so I never present
problems without at least an idea about a solution.  It may not be the
best, but you start at something and work toward bettering it until
you have something good.

Best wishes,
Mike

Hi Joey,

Thank you so much for your very thoughtful reply.

On Sun, 6 Feb 2011 21:58:08 -0400, Joey Hess wrote:
> Michael Gilbert wrote:
> > Another issue was that a lot of vulnerabilities that were found in
that
> > time frame were actually flaws in new kernel code, so testing/unstable
> > were vulnerable, but the stable kernel itself was unaffected, so it
was
> > a bit safer to be running the stable kernel since the code was older
> > (i.e. there was more time to find and fix issues).  For example, the
> > vulnerability for one of the local privilege escalations that was
> > exploited in the wild was introduced in 2.6.30, so lenny
wasn''t
> > affected, but testing/unstable were.
> 
> LWN''s analysis of age of introduction of kernel security holes in
2010
> doesn''t seem to agree? http://lwn.net/Articles/410606/
> 
> | So, in a sense, the above-mentioned kernel hacker was correct - an
> | awful lot of the vulnerabilities fixed over the last year predate the
> | git era, and are thus over five years old.
LWN''s analysis is far from comprehensive, and the conclusions are not
based in sufficiently rigorous analysis, but instead on the usual
numbers game.  Their reporting is however very useful as a basis for
further research.  

The first fact that they completely miss is that not all CVEs are
created equal.  A memory info disclosure gets a CVE just like a 
privilege escalation that has known exploits, but both are on the same
playing field in the numbers game. Annecdotally, CVE-2010-3301 and
CVE-2010-1146 had an exploit in the wild, and 2.6.26 was never
affected.  The only issue that had an in-the-wild exploit affecting
lenny in that list (that I''m aware of) was CVE-2010-3081 (and lenny
would have sidestepped that too if it had had been even more
conservative and gone with the older/stabler 2.6.25).

Even playing the numbers game with a bit more thoughtful analysis with
the LWN data, lenny looks a lot better.  It can be seen that lenny
(2.6.26) was vulnerable to 69% (36 out of 52) of the vulnerabilities
listed there, and squeeze (2.6.32) was vulnerable to 98% (51 out of
52).  In my opinion that''s a rather substantial difference, and thus a
problem worth pondering.
> At best, you seem to be grossly oversimplifying. In the immortal words
> of FaceBook, "it''s complicated".
Yes, I have taken a hard problem (that of improving overall testing
security), and divided it into a specifically manageable part that I
want to attempt to conquer.  That process can indeed be called
simplifying, but it is an inevitable consequence of problem solving,
and not something to be treated as a flaw.
> > I''d like to propose a solution to these two problems: only
upload known
> > rock solid longterm cadence upstream supported kernels into
> > testing/unstable. This will hopefully reduce the amount of transition
> > delay since the longterm kernels should have fewer RC issues (after
> > they''ve had a little time to cook of course).
> > 
> > There are, of course, some undesirable consequences to this plan.  One
> > is that a certain subset of testing users expect the latest shiny all
> > the time; but they can easily get their fix from the experimental
> > kernels instead, so that isn''t really a problem (I think).
> 
> I feel that unstable is heading in much too stable a direction already.
> I think that Bdale mentioned this (as a possible negative feature of CUT)
> at DebConf. 
Why is that such a bad thing?  Perhaps the current state is simply an
inevitable consequence of progress, or in the terminology of the
fashion industry, "experimental is the new unstable".
> > The second issue is that testing d-i will not support the latest and
> > greatest hardware and features.  I think this can be solved by
> > providing two sets of d-i media for testing (one that uses the
longterm
> > testing kernel and one that uses newer experimental kernel).
> 
> This seems likely to increase the workload of the d-i (and CD..) team
> significantly. 
Yes, there will of course be a larger workload to maintain two sets of
testing d-i media, although only one (the one based off experimental)
would be likely to have any breakage, so the stable one won''t need too
much attention.
> And wouldn''t it result in either an installed system that
> used unstable''s old kernel and so didn''t work, or used
experimental''s
> kernel and so didn''t auto-upgrade to fix security holes?
The experimental media would be provided with zero claimed security
support, the other media would get full security support.  I don''t see
why you think that would be broken though; it would be using the
current set of packages so it should remain stable (as long as all d-i
development moves to experimental).
> Also, what are teams like the X team to do, when their packages depend
> on new kernel features? How are we supposed to ever integrate support
> for the new kernel distribution-wide if unstable constantly has an old
> version? Should the whole distribution lag years behind the state of the
> art?
That can certainly still happen; it will just be close to the freeze,
rather than right now.  In the meantime, they can stage their updates
in experimental (or provide a compatibility layer).
> > Anyway, I think this would go a long way toward improving the quality
> > of security support in testing and thus reducing the common
advice/meme
> > that users should avoid testing if they''re concerned about
security.
> 
> A meme that is not founded in truth, as you can see if you compare
> existing known security holes in stable and in testing at most points in
> the release cycle. Perhaps the project should consider how it allows
> these types of myths to stand up when their foundations are so easily
> disproven?
The meme actually continues to be true, but to a lesser degree than in
the past.  That''s thanks to the herculean and thankless efforts of
members of the testing security team such as Moritz Muehlenhoff, Nico
Golde, Raphael Geissert, and many others for the past couple years.
Let''s take the freedom made available by the new cycle to try to
address one of the biggest remaining offenders.

Best wishes,
Mike

On Mon, 7 Feb 2011 22:54:53 -0500 Michael Gilbert wrote:
> Even playing the numbers game with a bit more thoughtful analysis with
> the LWN data, lenny looks a lot better.  It can be seen that lenny
> (2.6.26) was vulnerable to 69% (36 out of 52) of the vulnerabilities
> listed there, and squeeze (2.6.32) was vulnerable to 98% (51 out of
> 52).  In my opinion that''s a rather substantial difference, and
thus a
> problem worth pondering.
Minor correction: lenny was vulnerable to 67% (35 out of 51) and
squeeze was vulnerable to 98% (50 out of 51).  I had missed the issue
that was fixed in 2.6.20 and didn''t affect either releases.

Best wishes,
Mike

On Mon, Feb 07, 2011 at 05:15:07PM -0500, Michael Gilbert
wrote:
> On Mon, Feb 7, 2011 at 5:09 PM, Julien Cristau wrote:
> > What does that buy us? ??It means instead of dealing with bugs on an
> > ongoing basis, you get them all at the same time and get to bisect
along
> > many kernel versions at once instead of just one. ??It means problems
> > don''t get reported (and fixed) upstream until it''s
too late. ??It means
> > any package that could use a newer kernel interface doesn''t
get any
> > testing. ??I''m sure there''s plenty of others.
> Bugs can be submitted and dealt with in experimental just as well as
> in unstable.
Realistically people don''t generally go randomly installing things from
experimental so the testing coverage we get from having unstable and
testing is substantially reduced - we get to deal with everything in the
freeze instead, plus all the effects of running an old kernel which
isn''t being actively developed.

On Mon, 7 Feb 2011 22:54:53 -0500 Michael Gilbert wrote:
> On Sun, 6 Feb 2011 21:58:08 -0400, Joey Hess wrote:
> > Michael Gilbert wrote:
> > > Another issue was that a lot of vulnerabilities that were found
in that
> > > time frame were actually flaws in new kernel code, so
testing/unstable
> > > were vulnerable, but the stable kernel itself was unaffected, so
it was
> > > a bit safer to be running the stable kernel since the code was
older
> > > (i.e. there was more time to find and fix issues).  For example,
the
> > > vulnerability for one of the local privilege escalations that was
> > > exploited in the wild was introduced in 2.6.30, so lenny
wasn''t
> > > affected, but testing/unstable were.
> > 
> > LWN''s analysis of age of introduction of kernel security
holes in 2010
> > doesn''t seem to agree? http://lwn.net/Articles/410606/
> > 
> > | So, in a sense, the above-mentioned kernel hacker was correct - an
> > | awful lot of the vulnerabilities fixed over the last year predate
the
> > | git era, and are thus over five years old.
> 
> LWN''s analysis is far from comprehensive, and the conclusions are
not
> based in sufficiently rigorous analysis, but instead on the usual
> numbers game.  Their reporting is however very useful as a basis for
> further research.  
> 
> The first fact that they completely miss is that not all CVEs are
> created equal.  A memory info disclosure gets a CVE just like a 
> privilege escalation that has known exploits, but both are on the same
> playing field in the numbers game. Annecdotally, CVE-2010-3301 and
> CVE-2010-1146 had an exploit in the wild, and 2.6.26 was never
> affected.  The only issue that had an in-the-wild exploit affecting
> lenny in that list (that I''m aware of) was CVE-2010-3081 (and
lenny
> would have sidestepped that too if it had had been even more
> conservative and gone with the older/stabler 2.6.25).
> 
> Even playing the numbers game with a bit more thoughtful analysis with
> the LWN data, lenny looks a lot better.  It can be seen that lenny
> (2.6.26) was vulnerable to 69% (36 out of 52) of the vulnerabilities
> listed there, and squeeze (2.6.32) was vulnerable to 98% (51 out of
> 52).  In my opinion that''s a rather substantial difference, and
thus a
> problem worth pondering.
So, now that I''ve had some time to contemplate the replies on this
issue, I have a much better appreciation of the need to keep unstable
closely in line with upstream development, and thus don''t want to push
the original solution anymore. Also 2.6.37 is in unstable now, so that
idea is impossible, which is OK.

However, as I justify above, there is still a problem, and I think it
can still be solved relatively easily and without too much impact.
This time I suggest blocking 2.6.37 so 2.6.32 remains in testing for a
while.  This will allow it to get updates in sync with stable kernel
security updates (without any additional effort by Dann, Moritz, and
other kernel team members other than the package upload to testing as
well).

This will also help to provide a bit more stability for CUT [0].  Over
a 1.5-year period (the non-freeze timeframe) roughly 6 new upstream
kernels will be released, and each new kernel comes along with a high
probability of introducing breakage.  I''m trying to provide CUT
releases once a month, so every 3 months I will be looking at a likely
broken CUT release (or 25% of the time).  However, if there is just one
kernel transition in testing development cycle, then there is only 1
likely broken period (or 4% of the time).

Anyway, I know that the fact that this may have a future effect on my
efforts isn''t a great argument, but this does impact whether CUT will
be
successful, which impacts the whole project.  I wonder if we can at
least try it for a few months, and if there really are problems with
keeping the old kernel in testing, then we can just end this experiment
and unblock the newer upstream version.

If there is some concurrence on this, I''ll submit an RC blocker bug on
the kernel.  Note there are only 8 days to discuss before the
transition is automatic (I think the current RC blocker [1] may
disappear by then).

Best wishes,
Mike

[0] http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000195.html
[1] http://qa.debian.org/excuses.php?package=linux-2.6

On 18/02/11 at 17:24 -0500, Michael Gilbert wrote:
> On Mon, 7 Feb 2011 22:54:53 -0500 Michael Gilbert wrote:
> > On Sun, 6 Feb 2011 21:58:08 -0400, Joey Hess wrote:
> > > Michael Gilbert wrote:
> > > > Another issue was that a lot of vulnerabilities that were
found in that
> > > > time frame were actually flaws in new kernel code, so
testing/unstable
> > > > were vulnerable, but the stable kernel itself was
unaffected, so it was
> > > > a bit safer to be running the stable kernel since the code
was older
> > > > (i.e. there was more time to find and fix issues).  For
example, the
> > > > vulnerability for one of the local privilege escalations
that was
> > > > exploited in the wild was introduced in 2.6.30, so lenny
wasn''t
> > > > affected, but testing/unstable were.
> > > 
> > > LWN''s analysis of age of introduction of kernel security
holes in 2010
> > > doesn''t seem to agree? http://lwn.net/Articles/410606/
> > > 
> > > | So, in a sense, the above-mentioned kernel hacker was correct -
an
> > > | awful lot of the vulnerabilities fixed over the last year
predate the
> > > | git era, and are thus over five years old.
> > 
> > LWN''s analysis is far from comprehensive, and the conclusions
are not
> > based in sufficiently rigorous analysis, but instead on the usual
> > numbers game.  Their reporting is however very useful as a basis for
> > further research.  
> > 
> > The first fact that they completely miss is that not all CVEs are
> > created equal.  A memory info disclosure gets a CVE just like a 
> > privilege escalation that has known exploits, but both are on the same
> > playing field in the numbers game. Annecdotally, CVE-2010-3301 and
> > CVE-2010-1146 had an exploit in the wild, and 2.6.26 was never
> > affected.  The only issue that had an in-the-wild exploit affecting
> > lenny in that list (that I''m aware of) was CVE-2010-3081 (and
lenny
> > would have sidestepped that too if it had had been even more
> > conservative and gone with the older/stabler 2.6.25).
> > 
> > Even playing the numbers game with a bit more thoughtful analysis with
> > the LWN data, lenny looks a lot better.  It can be seen that lenny
> > (2.6.26) was vulnerable to 69% (36 out of 52) of the vulnerabilities
> > listed there, and squeeze (2.6.32) was vulnerable to 98% (51 out of
> > 52).  In my opinion that''s a rather substantial difference,
and thus a
> > problem worth pondering.
> 
> So, now that I''ve had some time to contemplate the replies on this
> issue, I have a much better appreciation of the need to keep unstable
> closely in line with upstream development, and thus don''t want to
push
> the original solution anymore. Also 2.6.37 is in unstable now, so that
> idea is impossible, which is OK.
> 
> However, as I justify above, there is still a problem, and I think it
> can still be solved relatively easily and without too much impact.
> This time I suggest blocking 2.6.37 so 2.6.32 remains in testing for a
> while.  This will allow it to get updates in sync with stable kernel
> security updates (without any additional effort by Dann, Moritz, and
> other kernel team members other than the package upload to testing as
> well).
You base your reasoning on the assumption that CUT users prefer a more
stable kernel to a more recent kernel. I''m tempted to think the
opposite.

- lucas

On Fri, 18 Feb 2011, Michael Gilbert wrote:
> This will also help to provide a bit more stability for CUT [0].  Over
> a 1.5-year period (the non-freeze timeframe) roughly 6 new upstream
> kernels will be released, and each new kernel comes along with a high
> probability of introducing breakage.  I''m trying to provide CUT
> releases once a month, so every 3 months I will be looking at a likely
> broken CUT release (or 25% of the time).  However, if there is just one
> kernel transition in testing development cycle, then there is only 1
> likely broken period (or 4% of the time).
The kernel is not the sole component that can introduce breakage. So it
seems ridiculous to do such statistics based on the kernel only.

And like Lucas said, CUT users want recent software and that includes the
kernel (which is also important for new hardware support).

Cheers,
-- 
Rapha?l Hertzog ? Debian Developer

Follow my Debian News ? http://RaphaelHertzog.com (English)
                      ? http://RaphaelHertzog.fr (Fran?ais)

On Sat, 19 Feb 2011 14:09:50 +0100 Raphael Hertzog wrote:
> On Fri, 18 Feb 2011, Michael Gilbert wrote:
> > This will also help to provide a bit more stability for CUT [0].  Over
> > a 1.5-year period (the non-freeze timeframe) roughly 6 new upstream
> > kernels will be released, and each new kernel comes along with a high
> > probability of introducing breakage.  I''m trying to provide
CUT
> > releases once a month, so every 3 months I will be looking at a likely
> > broken CUT release (or 25% of the time).  However, if there is just
one
> > kernel transition in testing development cycle, then there is only 1
> > likely broken period (or 4% of the time).
> 
> The kernel is not the sole component that can introduce breakage. So it
> seems ridiculous to do such statistics based on the kernel only.
Hi Raphael, I completely concur that there are indeed other important
components like grub and xorg where breakage would be a showstopper.
However, I don''t think the consequences would be quite so substantial.
There is just so much intimately dependent on the kernel that breakage
there has a very wide area of effect; whereas grub or xorg breakage is
somewhat contained.  This is why I''m only significantly concerned about
the kernel.  Anyway, I agree that there will be breakage, but I don''t
want CUT to be mostly about constantly resolving the same repetitive
kernel breakage.

Also, this solution isn''t just about CUT stability.  As I''ve
been
describing, it is about killing about 2 birds with one stone:

1. Make testing always installable by retaining a stable/well-tested
kernel and associated d-i infrastructure
2. Improve testing security by reducing the amount of vulnerabilities
existent in older kernels (roughly 67% fewer in 2.6.32 vs 2.6.37 as
described previously)

In fact these are two of the three known problems in testing mentioned
in your LWN article [0].  And the only consequence is that testing users
don''t get the latest kernel bling.  However, that is offset by the
availability of 2.6.37 and associated infrastructure in unstable.  So
there really are no downsides that can''t be worked around with a little
education/effort on the part of the user.
> And like Lucas said, CUT users want recent software and that includes the
> kernel
I don''t think it is appropriate to project a singular perspective onto
all users.  Ultimately our goal is to perform a balancing act between
freshness and stability.  In the past the primary focus has been on
freshness in testing, but I think there is room to swing the pendulum a
bit more toward stability.  It''s really a requirement if we''re
ever
going to be able to achieve something a testing that''s "constantly
usable". Also, users needing bleeding-edge freshness can always take
the plunge into unstable.
> (which is also important for new hardware support).
This seems to be a meme that continues to persist without much in the
way of evidence.  It certainly may have been true in the past, but I
think things have changed for the better with the advent of stable
upstream support (i.e. support for new hardware is backported to the
stable kernels).

Also, I''ve read about 10 reviews of squeeze, and none of them have
indicated any problems with hardware support (except for missing
support for non-free firmware) even though that uses a kernel initially
released almost a year and a half ago.  In fact, I''ve only ever had one
piece of hardware (a wireless card) that was unsupported by Debian, and
that was when sarge was still in testing, and all that was needed was
to enter the device id into the discover list.  It wasn''t even a kernel
issue.

Ultimately I don''t see why the newest blingy kernel is a necessity in
testing.

Best wishes,
Mike

[0] http://lwn.net/Articles/406301/

On Sat, 2011-02-19 at 13:12 -0500, Michael Gilbert wrote:
[...]
> Also, this solution isn''t just about CUT stability.  As
I''ve been
> describing, it is about killing about 2 birds with one stone:
> 
> 1. Make testing always installable by retaining a stable/well-tested
> kernel and associated d-i infrastructure
We do backport new hardware support to stable but we do not have the
resources (time and equipment) to cover everything.  So this would mean
that neither stable nor testing would be installable on some newer
hardware.
> 2. Improve testing security by reducing the amount of vulnerabilities
> existent in older kernels (roughly 67% fewer in 2.6.32 vs 2.6.37 as
> described previously)
Huh?  I don''t see any source for this figure.

[...]
> > (which is also important for new hardware support).
> 
> This seems to be a meme that continues to persist without much in the
> way of evidence.  It certainly may have been true in the past, but I
> think things have changed for the better with the advent of stable
> upstream support (i.e. support for new hardware is backported to the
> stable kernels).
> 
> Also, I''ve read about 10 reviews of squeeze, and none of them have
> indicated any problems with hardware support (except for missing
> support for non-free firmware) even though that uses a kernel initially
> released almost a year and a half ago.
[...]

I can assure you there is already a substantial backlog of new hardware
that is currently unsupported in squeeze.  For example, any current ATI
graphics chip.  And this is at the start of squeeze''s lifetime, not the
end.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 828 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20110219/78fbfa69/attachment.pgp>

On Sat, 19 Feb 2011 18:48:40 +0000 Ben Hutchings wrote:
> On Sat, 2011-02-19 at 13:12 -0500, Michael Gilbert wrote:
> [...]
> > Also, this solution isn''t just about CUT stability.  As
I''ve been
> > describing, it is about killing about 2 birds with one stone:
> > 
> > 1. Make testing always installable by retaining a stable/well-tested
> > kernel and associated d-i infrastructure
> 
> We do backport new hardware support to stable but we do not have the
> resources (time and equipment) to cover everything.  So this would mean
> that neither stable nor testing would be installable on some newer
> hardware.
Right, and in those rare cases, the user will have to sufficiently
educate themselves to be able to use unstable.
> > 2. Improve testing security by reducing the amount of vulnerabilities
> > existent in older kernels (roughly 67% fewer in 2.6.32 vs 2.6.37 as
> > described previously)
> 
> Huh?  I don''t see any source for this figure.
http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000193.html
http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000194.html
> [...]
> > > (which is also important for new hardware support).
> > 
> > This seems to be a meme that continues to persist without much in the
> > way of evidence.  It certainly may have been true in the past, but I
> > think things have changed for the better with the advent of stable
> > upstream support (i.e. support for new hardware is backported to the
> > stable kernels).
> > 
> > Also, I''ve read about 10 reviews of squeeze, and none of them
have
> > indicated any problems with hardware support (except for missing
> > support for non-free firmware) even though that uses a kernel
initially
> > released almost a year and a half ago.
> [...]
> 
> I can assure you there is already a substantial backlog of new hardware
> that is currently unsupported in squeeze.  For example, any current ATI
> graphics chip.  And this is at the start of squeeze''s lifetime,
not the
> end.
I''ve been using ati cards exclusively for some time now; although
I''ve
also been willing to install the fglrx driver for full support ;)
Also, the xorg vesa driver does work.

Again, if the user is interested in such new developments, they will
need to be willing to learn how to run an unstable system.

Best wishes,
Mike

On Sat, 2011-02-19 at 14:04 -0500, Michael Gilbert
wrote:
> On Sat, 19 Feb 2011 18:48:40 +0000 Ben Hutchings wrote:
> 
> > On Sat, 2011-02-19 at 13:12 -0500, Michael Gilbert wrote:
[...]
> > > 2. Improve testing security by reducing the amount of
vulnerabilities
> > > existent in older kernels (roughly 67% fewer in 2.6.32 vs 2.6.37
as
> > > described previously)
> > 
> > Huh?  I don''t see any source for this figure.
> 
> http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000193.html
> http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000194.html
I read those and I can''t see any source for comparison between 2.6.32
and 2.6.37.  In fact you say that ''squeeze (2.6.32) was vulnerable to
98% (51 out of 52)'' which implies only 2% fewer vulnerabilities.
> > [...]
> > > > (which is also important for new hardware support).
> > > 
> > > This seems to be a meme that continues to persist without much in
the
> > > way of evidence.  It certainly may have been true in the past,
but I
> > > think things have changed for the better with the advent of
stable
> > > upstream support (i.e. support for new hardware is backported to
the
> > > stable kernels).
> > > 
> > > Also, I''ve read about 10 reviews of squeeze, and none of
them have
> > > indicated any problems with hardware support (except for missing
> > > support for non-free firmware) even though that uses a kernel
initially
> > > released almost a year and a half ago.
> > [...]
> > 
> > I can assure you there is already a substantial backlog of new
hardware
> > that is currently unsupported in squeeze.  For example, any current
ATI
> > graphics chip.  And this is at the start of squeeze''s
lifetime, not the
> > end.
> 
> I''ve been using ati cards exclusively for some time now; although
I''ve
> also been willing to install the fglrx driver for full support ;)
Then I really can''t take your concern for security seriously.  The
changelog for fglrx-source has no mention of security fixes, and I
don''t
for one moment believe there are no vulnerabilities in it.
> Also, the xorg vesa driver does work.
Seems like a waste of money to buy an ATI card and then use it as a dumb
framebuffer.
> Again, if the user is interested in such new developments, they will
> need to be willing to learn how to run an unstable system.
I thought that users interested in new stuff were supposed to run CUT.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 828 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20110219/dbf12abe/attachment.pgp>

On Sat, 19 Feb 2011 19:32:08 +0000 Ben Hutchings wrote:
> On Sat, 2011-02-19 at 14:04 -0500, Michael Gilbert wrote:
> > On Sat, 19 Feb 2011 18:48:40 +0000 Ben Hutchings wrote:
> > 
> > > On Sat, 2011-02-19 at 13:12 -0500, Michael Gilbert wrote:
> [...]
> > > > 2. Improve testing security by reducing the amount of
vulnerabilities
> > > > existent in older kernels (roughly 67% fewer in 2.6.32 vs
2.6.37 as
> > > > described previously)
> > > 
> > > Huh?  I don''t see any source for this figure.
> > 
> >
http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000193.html
> >
http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000194.html
> 
> I read those and I can''t see any source for comparison between
2.6.32
> and 2.6.37.  In fact you say that ''squeeze (2.6.32) was vulnerable
to
> 98% (51 out of 52)'' which implies only 2% fewer vulnerabilities.
I suppose the way I said that is confusing.  That research was from
past results, and my latest statement is a projection based on the
past.  In other words, if lenny was vulnerable to 67% of the issues that
squeeze was, I''m projecting that it will be similar for squeeze: it
will be vulnerable to about 67% of the issues that wheezy will;
although that could be +-10%, +-20%, who knows since events have yet
to happen.
> > I''ve been using ati cards exclusively for some time now;
although I''ve
> > also been willing to install the fglrx driver for full support ;)
> 
> Then I really can''t take your concern for security seriously.  The
> changelog for fglrx-source has no mention of security fixes, and I
don''t
> for one moment believe there are no vulnerabilities in it.
Well, that''s a risk I''m willing to accept for myself.  Others
may have a
differing perspective, and that''s fine. My risk mitigation strategy
should have nothing to do with the rest of the project''s.
> > Also, the xorg vesa driver does work.
> 
> Seems like a waste of money to buy an ATI card and then use it as a dumb
> framebuffer.
Not all ati cards are top of the line, and not all users need 3D anyway.
> > Again, if the user is interested in such new developments, they will
> > need to be willing to learn how to run an unstable system.
> 
> I thought that users interested in new stuff were supposed to run CUT.
Most packages will in fact be new, just the kernel and reverse
dependencies will be held back.  Hence CUT users will get 99% new
stuff (with respect to stable), and a tiny bit held back simply for
stability. Like I''ve said a couple times now, its a balancing act.

All I''m asking for is a few month long experiment.  And if the
experiment shows signs of flaws/weaknesses, then the blocker can
certainly be lifted.

Best wishes,
Mike

On Sat, 19 Feb 2011 14:59:27 -0500 Michael Gilbert wrote:
> On Sat, 19 Feb 2011 19:32:08 +0000 Ben Hutchings wrote:
> 
> > On Sat, 2011-02-19 at 14:04 -0500, Michael Gilbert wrote:
> > > On Sat, 19 Feb 2011 18:48:40 +0000 Ben Hutchings wrote:
> > > 
> > > > On Sat, 2011-02-19 at 13:12 -0500, Michael Gilbert wrote:
> > [...]
> > > > > 2. Improve testing security by reducing the amount of
vulnerabilities
> > > > > existent in older kernels (roughly 67% fewer in 2.6.32
vs 2.6.37 as
> > > > > described previously)
> > > > 
> > > > Huh?  I don''t see any source for this figure.
> > > 
> > >
http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000193.html
> > >
http://lists.alioth.debian.org/pipermail/cut-team/2011-February/000194.html
> > 
> > I read those and I can''t see any source for comparison
between 2.6.32
> > and 2.6.37.  In fact you say that ''squeeze (2.6.32) was
vulnerable to
> > 98% (51 out of 52)'' which implies only 2% fewer
vulnerabilities.
> 
> I suppose the way I said that is confusing.  That research was from
> past results, and my latest statement is a projection based on the
> past.  In other words, if lenny was vulnerable to 67% of the issues that
> squeeze was, I''m projecting that it will be similar for squeeze:
it
> will be vulnerable to about 67% of the issues that wheezy will;
> although that could be +-10%, +-20%, who knows since events have yet
> to happen.
I still think I''ve written this confusingly.  Let me try one more time.
Over squeeze''s testing period, the stable kernel (2.6.26) was
vulnerable to roughly 67% of the issues that the testing kernels
(2.6.28-32) were vulnerable to. Hence projecting for the future,
over wheezy''s testing period, the stable kernel (2.6.32) will be
vulnerable to roughly 67% (+- some uncertainty value) of the issues
that the testing kernels (2.6.37-4x) will be vulnerable to.

Best wishes,
Mike

On Sat, 2011-02-19 at 14:59 -0500, Michael Gilbert
wrote:
> On Sat, 19 Feb 2011 19:32:08 +0000 Ben Hutchings wrote:
[...]
> > > Again, if the user is interested in such new developments, they
will
> > > need to be willing to learn how to run an unstable system.
> > 
> > I thought that users interested in new stuff were supposed to run CUT.
> 
> Most packages will in fact be new, just the kernel and reverse
> dependencies will be held back.  Hence CUT users will get 99% new
> stuff (with respect to stable), and a tiny bit held back simply for
> stability. Like I''ve said a couple times now, its a balancing act.
> 
> All I''m asking for is a few month long experiment.  And if the
> experiment shows signs of flaws/weaknesses, then the blocker can
> certainly be lifted.
If an experiment is to have any validity, the hypothesis and the
criteria for assessing the outcome must be decided in advance.  If you
can do that, perhaps you will persuade some people that this is worth
doing.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 828 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20110219/718c6302/attachment.pgp>

On Sat, 19 Feb 2011 20:30:47 +0000 Ben Hutchings wrote:
> On Sat, 2011-02-19 at 14:59 -0500, Michael Gilbert wrote:
> > On Sat, 19 Feb 2011 19:32:08 +0000 Ben Hutchings wrote:
> [...]
> > > > Again, if the user is interested in such new developments,
they will
> > > > need to be willing to learn how to run an unstable system.
> > > 
> > > I thought that users interested in new stuff were supposed to run
CUT.
> > 
> > Most packages will in fact be new, just the kernel and reverse
> > dependencies will be held back.  Hence CUT users will get 99% new
> > stuff (with respect to stable), and a tiny bit held back simply for
> > stability. Like I''ve said a couple times now, its a balancing
act.
> > 
> > All I''m asking for is a few month long experiment.  And if
the
> > experiment shows signs of flaws/weaknesses, then the blocker can
> > certainly be lifted.
> 
> If an experiment is to have any validity, the hypothesis and the
> criteria for assessing the outcome must be decided in advance.  If you
> can do that, perhaps you will persuade some people that this is worth
> doing.
Hypothesis 1: using an older kernel in testing results in fewer vulnerabilities

  Criteria: fewer vulnerabilities in lenny than squeeze during squeeze testing
cycle
  Evidence: lenny''s kernel was vulnerable to 67% of the vulnerabilities
that squeeze
  Conclusion: hypothesis verified
  
  Criteria: fewer vulnerabilities in squeeze than wheezy during wheezy testing
cycle
  Evidence: to be collected # vulnerabilities in squeeze and wheezy
  Conclusion: to be determined

Hypothesis 2: using an older kernel version makes less work to provide CUT

  Criteria: how often CUT target release date is met
  Evidence: to be collected monthly release date by retaining 2.6.32 and monthly
            for standard unstable->testing transitions
            (note: requires a 2.6.32-only period for reference)
  Conclusion: to be determined

I can''t imagine anyone else being put through such a arduous process
to try an experiment for a couple months.  Why does it have to be so
difficult?

Best wishes,
Mike

On Sat, Feb 19, 2011 at 03:55:03PM -0500, Michael Gilbert
wrote:
> Hypothesis 1: using an older kernel in testing results in fewer
vulnerabilities
> 
>   Criteria: fewer vulnerabilities in lenny than squeeze during squeeze
testing cycle
>   Evidence: lenny''s kernel was vulnerable to 67% of the
vulnerabilities that squeeze
>   Conclusion: hypothesis verified
Actually you did not yet proof this. Please do it.
>   Criteria: fewer vulnerabilities in squeeze than wheezy during wheezy
testing cycle
>   Evidence: to be collected # vulnerabilities in squeeze and wheezy
>   Conclusion: to be determined
> 
> Hypothesis 2: using an older kernel version makes less work to provide CUT
> 
>   Criteria: how often CUT target release date is met
>   Evidence: to be collected monthly release date by retaining 2.6.32 and
monthly
>             for standard unstable->testing transitions
>             (note: requires a 2.6.32-only period for reference)
>   Conclusion: to be determined
Hypothesis 3: Testing users wants old software

  Criteria: to be determined
  Evidence: easy
  Conclusion: sorry, no chance
> I can''t imagine anyone else being put through such a arduous
process
> to try an experiment for a couple months.  Why does it have to be so
> difficult?
You can run you little experiment. For blocking packages please persuade
the release team as responsible entity within Debian.

Bastian

-- 
The joys of love made her human and the agonies of love destroyed her.
		-- Spock, "Requiem for Methuselah", stardate 5842.8

On Sat, 2011-02-19 at 15:55 -0500, Michael Gilbert
wrote:
> On Sat, 19 Feb 2011 20:30:47 +0000 Ben Hutchings wrote:
> 
> > On Sat, 2011-02-19 at 14:59 -0500, Michael Gilbert wrote:
> > > On Sat, 19 Feb 2011 19:32:08 +0000 Ben Hutchings wrote:
> > [...]
> > > > > Again, if the user is interested in such new
developments, they will
> > > > > need to be willing to learn how to run an unstable
system.
> > > > 
> > > > I thought that users interested in new stuff were supposed
to run CUT.
> > > 
> > > Most packages will in fact be new, just the kernel and reverse
> > > dependencies will be held back.  Hence CUT users will get 99% new
> > > stuff (with respect to stable), and a tiny bit held back simply
for
> > > stability. Like I''ve said a couple times now, its a
balancing act.
> > > 
> > > All I''m asking for is a few month long experiment.  And
if the
> > > experiment shows signs of flaws/weaknesses, then the blocker can
> > > certainly be lifted.
> > 
> > If an experiment is to have any validity, the hypothesis and the
> > criteria for assessing the outcome must be decided in advance.  If you
> > can do that, perhaps you will persuade some people that this is worth
> > doing.
> 
> Hypothesis 1: using an older kernel in testing results in fewer
vulnerabilities
> 
>   Criteria: fewer vulnerabilities in lenny than squeeze during squeeze
testing cycle
>   Evidence: lenny''s kernel was vulnerable to 67% of the
vulnerabilities that squeeze
>   Conclusion: hypothesis verified
>   
>   Criteria: fewer vulnerabilities in squeeze than wheezy during wheezy
testing cycle
>   Evidence: to be collected # vulnerabilities in squeeze and wheezy
>   Conclusion: to be determined
This experiment does not require that the propagation of kernel packages
into testing is changed.
> Hypothesis 2: using an older kernel version makes less work to provide CUT
> 
>   Criteria: how often CUT target release date is met
>   Evidence: to be collected monthly release date by retaining 2.6.32 and
monthly
>             for standard unstable->testing transitions
>             (note: requires a 2.6.32-only period for reference)
>   Conclusion: to be determined
OK, that''s a real experiment.  However I suspect there will be many
confounding factors that make it difficult to single out any one cause
for delays.
> I can''t imagine anyone else being put through such a arduous
process
> to try an experiment for a couple months.  Why does it have to be so
> difficult?
Because this experiment would involve many thousands of users, and you
have to convince other developers that the benefit to these users may be
worth the cost.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 828 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.alioth.debian.org/pipermail/secure-testing-team/attachments/20110219/422f63ce/attachment.pgp>

On Sat, 19 Feb 2011 22:28:17 +0100 Bastian Blank wrote:
> On Sat, Feb 19, 2011 at 03:55:03PM -0500, Michael Gilbert wrote:
> > Hypothesis 1: using an older kernel in testing results in fewer
vulnerabilities
> > 
> >   Criteria: fewer vulnerabilities in lenny than squeeze during squeeze
testing cycle
> >   Evidence: lenny''s kernel was vulnerable to 67% of the
vulnerabilities that squeeze
> >   Conclusion: hypothesis verified
> 
> Actually you did not yet proof this. Please do it.
I did verify it for the timeframe of the LWN study.  Actually, there is
no such thing as a proof ;  I suppose I could do a more thorough study
using the detailed kernel-sec data over the entire squeeze development
cycle...but that may take a while since I don''t have much free time
any more.
> >   Criteria: fewer vulnerabilities in squeeze than wheezy during wheezy
testing cycle
> >   Evidence: to be collected # vulnerabilities in squeeze and wheezy
> >   Conclusion: to be determined
> > 
> > Hypothesis 2: using an older kernel version makes less work to provide
CUT
> > 
> >   Criteria: how often CUT target release date is met
> >   Evidence: to be collected monthly release date by retaining 2.6.32
and monthly
> >             for standard unstable->testing transitions
> >             (note: requires a 2.6.32-only period for reference)
> >   Conclusion: to be determined
> 
> Hypothesis 3: Testing users wants old software
> 
>   Criteria: to be determined
>   Evidence: easy
>   Conclusion: sorry, no chance
Users have a variety of desires.  It''s the developers job to make the
best overall decision regardless of users'' immediate demands. The
release team fights with this all the time during the freeze (users
want new software, but the risk of breakage outweighs their immediate
wants).
> > I can''t imagine anyone else being put through such a arduous
process
> > to try an experiment for a couple months.  Why does it have to be so
> > difficult?
> 
> You can run you little experiment. For blocking packages please persuade
> the release team as responsible entity within Debian.
Isn''t it the kernel team that I need to convince? That''s what
this
discussion is all about.

Thanks,
Mike

On Sat, 19 Feb 2011 21:39:03 +0000 Ben Hutchings wrote:
> > Hypothesis 1: using an older kernel in testing results in fewer
vulnerabilities
> > 
> >   Criteria: fewer vulnerabilities in lenny than squeeze during squeeze
testing cycle
> >   Evidence: lenny''s kernel was vulnerable to 67% of the
vulnerabilities that squeeze
> >   Conclusion: hypothesis verified
> >   
> >   Criteria: fewer vulnerabilities in squeeze than wheezy during wheezy
testing cycle
> >   Evidence: to be collected # vulnerabilities in squeeze and wheezy
> >   Conclusion: to be determined
> 
> This experiment does not require that the propagation of kernel packages
> into testing is changed.
OK, revised hypothesis 1: using 2.6.32 in wheezy for the first year of its
development
                          will result in fewer vulnerabilities

  Criteria: fewer vulnerabilities in wheezy/2.6.32 vs unstable kernel over 1
year period
  Evidence: to be collected # vulnerabilities affecting 2.6.32 and kernel in
            unstable at the same time
  Conclusion: to be determined
> > I can''t imagine anyone else being put through such a arduous
process
> > to try an experiment for a couple months.  Why does it have to be so
> > difficult?
> 
> Because this experiment would involve many thousands of users, and you
> have to convince other developers that the benefit to these users may be
> worth the cost.
OK, are you sufficiently convinced to give me a chance at this
experiment, at least for a couple months???

Best wishes,
Mike

On Sat, Feb 19, 2011 at 04:58:50PM -0500, Michael Gilbert
wrote:
> On Sat, 19 Feb 2011 22:28:17 +0100 Bastian Blank wrote:
> > On Sat, Feb 19, 2011 at 03:55:03PM -0500, Michael Gilbert wrote:
> > > Hypothesis 1: using an older kernel in testing results in fewer
vulnerabilities
> > >   Evidence: lenny''s kernel was vulnerable to 67% of the
vulnerabilities that squeeze
> > Actually you did not yet proof this. Please do it.
> I did verify it for the timeframe of the LWN study.
The LWN study is for a wrong time frame. We speak about .26-.32 here,
not .33-.36. Also it does not take stable kernel releases into account.
> > Hypothesis 3: Testing users wants old software
> >   Criteria: to be determined
> >   Evidence: easy
> >   Conclusion: sorry, no chance
> Users have a variety of desires.
Yes. Stable users uses stable. So you have to show that a majority of
users uses testing not to get new hardware support/new software.
> > > I can''t imagine anyone else being put through such a
arduous process
> > > to try an experiment for a couple months.  Why does it have to be
so
> > > difficult?
> > You can run you little experiment. For blocking packages please
persuade
> > the release team as responsible entity within Debian.
> Isn''t it the kernel team that I need to convince? That''s
what this
> discussion is all about.
You were not able to convince one person of the kernel team. And I still
don''t see what this experiment would provide for the _users_ (I
explicitely exclude your effort, because our priority are the users and
not your experiment).

Bastian

-- 
Time is fluid ... like a river with currents, eddies, backwash.
		-- Spock, "The City on the Edge of Forever", stardate 3134.0

On 19/02/11 at 17:40 -0500, Michael Gilbert wrote:
> On Sat, 19 Feb 2011 21:39:03 +0000 Ben Hutchings wrote:
> > > Hypothesis 1: using an older kernel in testing results in fewer
vulnerabilities
> > > 
> > >   Criteria: fewer vulnerabilities in lenny than squeeze during
squeeze testing cycle
> > >   Evidence: lenny''s kernel was vulnerable to 67% of the
vulnerabilities that squeeze
> > >   Conclusion: hypothesis verified
> > >   
> > >   Criteria: fewer vulnerabilities in squeeze than wheezy during
wheezy testing cycle
> > >   Evidence: to be collected # vulnerabilities in squeeze and
wheezy
> > >   Conclusion: to be determined
> > 
> > This experiment does not require that the propagation of kernel
packages
> > into testing is changed.
> 
> OK, revised hypothesis 1: using 2.6.32 in wheezy for the first year of its
development
>                           will result in fewer vulnerabilities
> 
>   Criteria: fewer vulnerabilities in wheezy/2.6.32 vs unstable kernel over
1 year period
>   Evidence: to be collected # vulnerabilities affecting 2.6.32 and kernel
in
>             unstable at the same time
>   Conclusion: to be determined
> 
> > > I can''t imagine anyone else being put through such a
arduous process
> > > to try an experiment for a couple months.  Why does it have to be
so
> > > difficult?
> > 
> > Because this experiment would involve many thousands of users, and you
> > have to convince other developers that the benefit to these users may
be
> > worth the cost.
> 
> OK, are you sufficiently convinced to give me a chance at this
> experiment, at least for a couple months???
I don''t understand why you think that testing or CUT users want an
"old"
kernel, but want to run recent software for everything else on their
system.

Also, you need to see the downsides of this proposed experiment. By not
upgrading the kernel in testing, you will limit the amount of testing
that the new kernel will receive. That could, in turn, cause more bugs
to be found late in the wheezy release process, making it harder to
reach a newer stable kernel.
Or are you suggesting that we stay with 2.6.32 forever? ;)

- Lucas

On Sun, 20 Feb 2011 08:24:32 +0100 Lucas Nussbaum wrote:
> On 19/02/11 at 17:40 -0500, Michael Gilbert wrote:
> > On Sat, 19 Feb 2011 21:39:03 +0000 Ben Hutchings wrote:
> > > > Hypothesis 1: using an older kernel in testing results in
fewer vulnerabilities
> > > > 
> > > >   Criteria: fewer vulnerabilities in lenny than squeeze
during squeeze testing cycle
> > > >   Evidence: lenny''s kernel was vulnerable to 67% of
the vulnerabilities that squeeze
> > > >   Conclusion: hypothesis verified
> > > >   
> > > >   Criteria: fewer vulnerabilities in squeeze than wheezy
during wheezy testing cycle
> > > >   Evidence: to be collected # vulnerabilities in squeeze and
wheezy
> > > >   Conclusion: to be determined
> > > 
> > > This experiment does not require that the propagation of kernel
packages
> > > into testing is changed.
> > 
> > OK, revised hypothesis 1: using 2.6.32 in wheezy for the first year of
its development
> >                           will result in fewer vulnerabilities
> > 
> >   Criteria: fewer vulnerabilities in wheezy/2.6.32 vs unstable kernel
over 1 year period
> >   Evidence: to be collected # vulnerabilities affecting 2.6.32 and
kernel in
> >             unstable at the same time
> >   Conclusion: to be determined
> > 
> > > > I can''t imagine anyone else being put through such
a arduous process
> > > > to try an experiment for a couple months.  Why does it have
to be so
> > > > difficult?
> > > 
> > > Because this experiment would involve many thousands of users,
and you
> > > have to convince other developers that the benefit to these users
may be
> > > worth the cost.
> > 
> > OK, are you sufficiently convinced to give me a chance at this
> > experiment, at least for a couple months???
> 
> I don''t understand why you think that testing or CUT users want an
"old"
> kernel, but want to run recent software for everything else on their
> system.
I''ve already answered that in another mail.  But basically the answer
is a transition to unstable for users that *don''t want* the
older/stabler kernel. Of course that does require users to change their
ways, but that''s not so bad.  In fact it may be good to have more users
running unstable, finding bugs, and submitting reports.
> Also, you need to see the downsides of this proposed experiment. By not
> upgrading the kernel in testing, you will limit the amount of testing
> that the new kernel will receive. That could, in turn, cause more bugs
> to be found late in the wheezy release process, making it harder to
> reach a newer stable kernel.
> Or are you suggesting that we stay with 2.6.32 forever? ;)
Of course not ;)  I actually think there are a lot of people using
unstable, and those are really the best users to find bugs in the
kernel anyway (since presumably they actually know what they are
doing).

I suppose that would be an alternative hypothesis: keeping 2.6.32 in
testing for too long will lead to a lower quality final wheezy kernel.

That''s more of a subjective/qualitative issue, and I don''t
really know
how to define criteria to quantify it.  But like I said, in my opinion
unstable users are sufficient to work out the bugs.

Best wishes,
Mike