llvm dev - Jan 2013 - [LLVMdev] Obsolete PTX is NOT completely removed in 3.2 release

On Fri, Jan 11, 2013 at 02:47:01PM -0600, Pawel Wodnicki
wrote:
> On 1/11/2013 2:40 PM, Brooks Davis wrote:
> > On Fri, Jan 11, 2013 at 09:33:17PM +0100, Benjamin Kramer wrote:
> >> 
> >> On 11.01.2013, at 21:31, Justin Holewinski 
> >> <justin.holewinski at gmail.com> wrote:
> >> 
> >>> On Fri, Jan 11, 2013 at 3:26 PM, Benjamin Kramer 
> >>> <benny.kra at gmail.com> wrote:
> >>> 
> >>> On 11.01.2013, at 07:36, ????????? (Wei-Ren Chen) 
> >>> <chenwj at iis.sinica.edu.tw> wrote:
> >>> 
> >>>> Hi Pawel,
> >>>> 
> >>>> PTX already be replaced with NVPTX. However, PTX
subdirectory
> >>>> still sit in lib/Target in 3.2 release. Do you think
update
> >>>> the release tarball is a good idea? Also could you remove
it
> >>>> from the trunk?
> >>> 
> >>> Please do not, under no circumstances, change the 3.2 release 
> >>> tarballs at this point. They are mirrored around the world now
> >>> with cryptographic hashes and signatures. Changing them will 
> >>> break things for many people, especially for an extremely
> >>> minor thing like an empty directory.
> >>> 
> >>> I'm not sure if Pawel's tarball change should be
reverted now
> >>> as it already caused uproar, so changing it back might only 
> >>> make matters worse.
> >>> 
> >>> The tarballs were changed?
> >> 
> >> r172208
> > 
> > I finally updated the FreeBSD ports yesterday and today a user 
> > complained about distfile changes.  IMO, this revision should be 
> > reverted or all the other BSDs will have to chase checksums as 
> > well.
> > 
> > If you really want to remove the directory, ship a 3.2.1 tarball 
> > rather than screwing all the downstream consumers who's 
> > infrastructure exists to detect trojan'd tarballs.
> 
> Tarball is signed, it is not trjoan.
> Your infrastructure should be able to deal with it?
The FreeBSD ports collection maintains a set of sha256 hashes for
each distfile.  The system can deal with them changing, but it's an
inconvenience to port maintainers and users.  Even if we did have
infrastructure to verify the signatures we'd still have to check the
contents and would not just trust the signature since there's always a
risk of keys being compromised.

-- Brooks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130111/c3ecc424/attachment.sig>

On 1/11/2013 3:59 PM, Brooks Davis wrote:
> On Fri, Jan 11, 2013 at 02:47:01PM -0600, Pawel Wodnicki wrote:
>> On 1/11/2013 2:40 PM, Brooks Davis wrote:
>>> On Fri, Jan 11, 2013 at 09:33:17PM +0100, Benjamin Kramer
>>> wrote:
>>>> 
>>>> On 11.01.2013, at 21:31, Justin Holewinski 
>>>> <justin.holewinski at gmail.com> wrote:
>>>> 
>>>>> On Fri, Jan 11, 2013 at 3:26 PM, Benjamin Kramer 
>>>>> <benny.kra at gmail.com> wrote:
>>>>> 
>>>>> On 11.01.2013, at 07:36, ????????? (Wei-Ren Chen) 
>>>>> <chenwj at iis.sinica.edu.tw> wrote:
>>>>> 
>>>>>> Hi Pawel,
>>>>>> 
>>>>>> PTX already be replaced with NVPTX. However, PTX
>>>>>> subdirectory still sit in lib/Target in 3.2 release. Do
>>>>>> you think update the release tarball is a good idea?
Also
>>>>>> could you remove it from the trunk?
>>>>> 
>>>>> Please do not, under no circumstances, change the 3.2
>>>>> release tarballs at this point. They are mirrored around
>>>>> the world now with cryptographic hashes and signatures.
>>>>> Changing them will break things for many people, especially
>>>>> for an extremely minor thing like an empty directory.
>>>>> 
>>>>> I'm not sure if Pawel's tarball change should be
reverted
>>>>> now as it already caused uproar, so changing it back might
>>>>> only make matters worse.
>>>>> 
>>>>> The tarballs were changed?
>>>> 
>>>> r172208
>>> 
>>> I finally updated the FreeBSD ports yesterday and today a user
>>>  complained about distfile changes.  IMO, this revision should
>>> be reverted or all the other BSDs will have to chase checksums
>>> as well.
>>> 
>>> If you really want to remove the directory, ship a 3.2.1
>>> tarball rather than screwing all the downstream consumers who's
>>>  infrastructure exists to detect trojan'd tarballs.
>> 
>> Tarball is signed, it is not trjoan. Your infrastructure should
>> be able to deal with it?
> 
> The FreeBSD ports collection maintains a set of sha256 hashes for 
> each distfile.  The system can deal with them changing, but it's
> an inconvenience to port maintainers and users.  Even if we did
> have infrastructure to verify the signatures we'd still have to
> check the contents and would not just trust the signature since
> there's always a risk of keys being compromised.
 Since you do not use llvm.org security features (signatures) I think
understanding your process will be useful to whoever is doing the next
release. I would suggest communicating it early in the release cycle.

 As for the inconvenience with the 3.2, I really do not have a choice
under current release process. Changes have been made in the SVN
branches and tarball has to reflect that.
> 
> -- Brooks
> 
Paweł

On Fri, Jan 11, 2013 at 3:19 PM, Pawel Wodnicki <root at 32bitmicro.com>
wrote:
> On 1/11/2013 3:59 PM, Brooks Davis wrote:
>> On Fri, Jan 11, 2013 at 02:47:01PM -0600, Pawel Wodnicki wrote:
>>> On 1/11/2013 2:40 PM, Brooks Davis wrote:
>>>> On Fri, Jan 11, 2013 at 09:33:17PM +0100, Benjamin Kramer
>>>> wrote:
>>>>>
>>>>> On 11.01.2013, at 21:31, Justin Holewinski
>>>>> <justin.holewinski at gmail.com> wrote:
>>>>>
>>>>>> On Fri, Jan 11, 2013 at 3:26 PM, Benjamin Kramer
>>>>>> <benny.kra at gmail.com> wrote:
>>>>>>
>>>>>> On 11.01.2013, at 07:36, ????????? (Wei-Ren Chen)
>>>>>> <chenwj at iis.sinica.edu.tw> wrote:
>>>>>>
>>>>>>> Hi Pawel,
>>>>>>>
>>>>>>> PTX already be replaced with NVPTX. However, PTX
>>>>>>> subdirectory still sit in lib/Target in 3.2
release. Do
>>>>>>> you think update the release tarball is a good
idea? Also
>>>>>>> could you remove it from the trunk?
>>>>>>
>>>>>> Please do not, under no circumstances, change the 3.2
>>>>>> release tarballs at this point. They are mirrored
around
>>>>>> the world now with cryptographic hashes and signatures.
>>>>>> Changing them will break things for many people,
especially
>>>>>> for an extremely minor thing like an empty directory.
>>>>>>
>>>>>> I'm not sure if Pawel's tarball change should
be reverted
>>>>>> now as it already caused uproar, so changing it back
might
>>>>>> only make matters worse.
>>>>>>
>>>>>> The tarballs were changed?
>>>>>
>>>>> r172208
>>>>
>>>> I finally updated the FreeBSD ports yesterday and today a user
>>>>  complained about distfile changes.  IMO, this revision should
>>>> be reverted or all the other BSDs will have to chase checksums
>>>> as well.
>>>>
>>>> If you really want to remove the directory, ship a 3.2.1
>>>> tarball rather than screwing all the downstream consumers
who's
>>>>  infrastructure exists to detect trojan'd tarballs.
>>>
>>> Tarball is signed, it is not trjoan. Your infrastructure should
>>> be able to deal with it?
>>
>> The FreeBSD ports collection maintains a set of sha256 hashes for
>> each distfile.  The system can deal with them changing, but it's
>> an inconvenience to port maintainers and users.  Even if we did
>> have infrastructure to verify the signatures we'd still have to
>> check the contents and would not just trust the signature since
>> there's always a risk of keys being compromised.
>
>  Since you do not use llvm.org security features (signatures) I think
> understanding your process will be useful to whoever is doing the next
> release. I would suggest communicating it early in the release cycle.
>
>  As for the inconvenience with the 3.2, I really do not have a choice
> under current release process. Changes have been made in the SVN
> branches and tarball has to reflect that.
Which release process document are you referring to that indicates that?

Generally, so far as I know, this kind of post-release change is unprecedented.
>
>>
>> -- Brooks
>>
>
> Paweł
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Fri, Jan 11, 2013 at 05:19:55PM -0600, Pawel Wodnicki
wrote:
>  As for the inconvenience with the 3.2, I really do not have a choice
> under current release process. Changes have been made in the SVN
> branches and tarball has to reflect that.
Replacing an existing tarball is inexcusable. Period. Use a different
name or directory if you want to role a new tarball, but don't just
replace the existing one.

Joerg

Brooks,
> On Fri, Jan 11, 2013 at 02:47:01PM -0600, Pawel Wodnicki wrote:
>> On 1/11/2013 2:40 PM, Brooks Davis wrote:
>>> On Fri, Jan 11, 2013 at 09:33:17PM +0100, Benjamin Kramer
>>> wrote:
>>>> 
>>>> On 11.01.2013, at 21:31, Justin Holewinski 
>>>> <justin.holewinski at gmail.com> wrote:
>>>> 
>>>>> On Fri, Jan 11, 2013 at 3:26 PM, Benjamin Kramer 
>>>>> <benny.kra at gmail.com> wrote:
>>>>> 
>>>>> On 11.01.2013, at 07:36, ????????? (Wei-Ren Chen) 
>>>>> <chenwj at iis.sinica.edu.tw> wrote:
>>>>> 
>>>>>> Hi Pawel,
>>>>>> 
>>>>>> PTX already be replaced with NVPTX. However, PTX
>>>>>> subdirectory still sit in lib/Target in 3.2 release. Do
>>>>>> you think update the release tarball is a good idea?
Also
>>>>>> could you remove it from the trunk?
>>>>> 
>>>>> Please do not, under no circumstances, change the 3.2
>>>>> release tarballs at this point. They are mirrored around
>>>>> the world now with cryptographic hashes and signatures.
>>>>> Changing them will break things for many people, especially
>>>>> for an extremely minor thing like an empty directory.
>>>>> 
>>>>> I'm not sure if Pawel's tarball change should be
reverted
>>>>> now as it already caused uproar, so changing it back might
>>>>> only make matters worse.
>>>>> 
>>>>> The tarballs were changed?
>>>> 
>>>> r172208
>>> 
>>> I finally updated the FreeBSD ports yesterday and today a user
>>>  complained about distfile changes.  IMO, this revision should
>>> be reverted or all the other BSDs will have to chase checksums
>>> as well.
>>> 
>>> If you really want to remove the directory, ship a 3.2.1
>>> tarball rather than screwing all the downstream consumers who's
>>>  infrastructure exists to detect trojan'd tarballs.
>> 
>> Tarball is signed, it is not trjoan. Your infrastructure should
>> be able to deal with it?
> 
> The FreeBSD ports collection maintains a set of sha256 hashes for 
> each distfile.  The system can deal with them changing, but it's
> an inconvenience to port maintainers and users.  Even if we did
> have infrastructure to verify the signatures we'd still have to
> check the contents and would not just trust the signature since
> there's always a risk of keys being compromised.
> 
> -- Brooks
> 
 At first your reaction has puzzled me, but despite a few fire
breathing dragons getting loose and after carefully re-reading
these posts over the weekend I think I understand the underlying
issue.

 Simply put we both take this release seriously.

 The problem is the release process is not entirely
defined. Which is reflected in question from David Blaikie
(the other post I am quoting here).
> Which release process document are you referring to that indicates
> that?
> 
 Simple answer is the one on the llvm.org website.
But the real answer is, it all depends on how you read it.
Apparently, we read it differently which is not that
surprising as even our well defined compilers have
a bit of a problem with undefined behavior.

So what can we expect from a release process document?
My hope is that we have reached a sort of sequence
point and the ambiguity of the release process
can be resolved despite all the side effects.


David also stated this:
> Generally, so far as I know, this kind of post-release change is
> unprecedented.
 I agree with this conclusion but even more unprecedented
is to a expect timely and problem free release with zero and
I'll repeat again zero involvement from the wider community that
depends on the clang+llvm!
 This is how the 3.2 release has happened, just compare
the list of 3.2 to 3.1 binaries and BTW not a single
3.2 binary has been produced with the help of a respective
project, distribution or who ever.

 I hear the cries of your wounded infrastructures
and I understand that in the days of forged root certificates
my signature on the tarballs may not guarantee much of a
security.

 But why didn't you get involved in the 3.2 release?

 We could have resolved all this in the 2 months since
the 3.2 release got rolling. Everybody knew about the
changes for the 3.2 release, simple e-mail to the new
LLVMRM would go a long way to resolve any security or
other release related issues not to mention a bit
of a courtesy.

 I am doing this as service to the LLVM community
but frankly I'd rather be chasing my dragons of
probability now.

Paweł

On Sun, Jan 13, 2013 at 11:00 AM, Pawel Wodnicki <root at 32bitmicro.com>
wrote:
> Brooks,
>
>> On Fri, Jan 11, 2013 at 02:47:01PM -0600, Pawel Wodnicki wrote:
>>> On 1/11/2013 2:40 PM, Brooks Davis wrote:
>>>> On Fri, Jan 11, 2013 at 09:33:17PM +0100, Benjamin Kramer
>>>> wrote:
>>>>>
>>>>> On 11.01.2013, at 21:31, Justin Holewinski
>>>>> <justin.holewinski at gmail.com> wrote:
>>>>>
>>>>>> On Fri, Jan 11, 2013 at 3:26 PM, Benjamin Kramer
>>>>>> <benny.kra at gmail.com> wrote:
>>>>>>
>>>>>> On 11.01.2013, at 07:36, ????????? (Wei-Ren Chen)
>>>>>> <chenwj at iis.sinica.edu.tw> wrote:
>>>>>>
>>>>>>> Hi Pawel,
>>>>>>>
>>>>>>> PTX already be replaced with NVPTX. However, PTX
>>>>>>> subdirectory still sit in lib/Target in 3.2
release. Do
>>>>>>> you think update the release tarball is a good
idea? Also
>>>>>>> could you remove it from the trunk?
>>>>>>
>>>>>> Please do not, under no circumstances, change the 3.2
>>>>>> release tarballs at this point. They are mirrored
around
>>>>>> the world now with cryptographic hashes and signatures.
>>>>>> Changing them will break things for many people,
especially
>>>>>> for an extremely minor thing like an empty directory.
>>>>>>
>>>>>> I'm not sure if Pawel's tarball change should
be reverted
>>>>>> now as it already caused uproar, so changing it back
might
>>>>>> only make matters worse.
>>>>>>
>>>>>> The tarballs were changed?
>>>>>
>>>>> r172208
>>>>
>>>> I finally updated the FreeBSD ports yesterday and today a user
>>>>  complained about distfile changes.  IMO, this revision should
>>>> be reverted or all the other BSDs will have to chase checksums
>>>> as well.
>>>>
>>>> If you really want to remove the directory, ship a 3.2.1
>>>> tarball rather than screwing all the downstream consumers
who's
>>>>  infrastructure exists to detect trojan'd tarballs.
>>>
>>> Tarball is signed, it is not trjoan. Your infrastructure should
>>> be able to deal with it?
>>
>> The FreeBSD ports collection maintains a set of sha256 hashes for
>> each distfile.  The system can deal with them changing, but it's
>> an inconvenience to port maintainers and users.  Even if we did
>> have infrastructure to verify the signatures we'd still have to
>> check the contents and would not just trust the signature since
>> there's always a risk of keys being compromised.
>>
>> -- Brooks
>>
>
>  At first your reaction has puzzled me, but despite a few fire
> breathing dragons getting loose and after carefully re-reading
> these posts over the weekend I think I understand the underlying
> issue.
>
>  Simply put we both take this release seriously.
>
>  The problem is the release process is not entirely
> defined. Which is reflected in question from David Blaikie
> (the other post I am quoting here).
>
>> Which release process document are you referring to that indicates
>> that?
>>
>
>  Simple answer is the one on the llvm.org website.
I'm looking for something a bit more specific. Something that explains
(even ambiguously) the post-release process that might
discuss/imply/support the situation we've ended up in (with a
post-release change to the original tarballs well after the release
announcement, etc).

So far as I can tell the release documentation ends at the point where
the release is finalized/announced/etc.
> But the real answer is, it all depends on how you read it.
> Apparently, we read it differently which is not that
> surprising as even our well defined compilers have
> a bit of a problem with undefined behavior.
>
> So what can we expect from a release process document?
> My hope is that we have reached a sort of sequence
> point and the ambiguity of the release process
> can be resolved despite all the side effects.
While that does need to be addressed (& I did bring it up), there's
also the immediate issue at hand to deal with as well.
> David also stated this:
>
>> Generally, so far as I know, this kind of post-release change is
>> unprecedented.
>
>  I agree with this conclusion but even more unprecedented
> is to a expect timely and problem free release with zero and
> I'll repeat again zero involvement from the wider community that
> depends on the clang+llvm!
>  This is how the 3.2 release has happened, just compare
> the list of 3.2 to 3.1 binaries and BTW not a single
> 3.2 binary has been produced with the help of a respective
> project, distribution or who ever.
>
>  I hear the cries of your wounded infrastructures
> and I understand that in the days of forged root certificates
> my signature on the tarballs may not guarantee much of a
> security.
>
>  But why didn't you get involved in the 3.2 release?
Presumably: because the process has worked in the past. There would be
no reason to dictate how the release works when experience (&
documentation) seems to support the desired model.
>  We could have resolved all this in the 2 months since
> the 3.2 release got rolling.
Not exactly - it wasn't (& isn't) an unreasonable expectation that
releases are static. That's sort of the definition of a release,
really.
> Everybody knew about the changes for the 3.2 release,
What changes are you referring to? I don't know of any changes to the
/release process/ in 3.2, other than the change in RM (you rather than
Bill Wendling). Obviously with any such change there may be
miscommunications/misunderstandings, and that seems like all this is -
and couldn't really have been anticipated by 3rd party consumers of
LLVM.
> simple e-mail to the new
> LLVMRM would go a long way to resolve any security or
> other release related issues not to mention a bit
> of a courtesy.
>
>  I am doing this as service to the LLVM community
> but frankly I'd rather be chasing my dragons of
> probability now.

On Sun, Jan 13, 2013 at 01:00:55PM -0600, Pawel Wodnicki
wrote:
> Brooks,
> 
> > On Fri, Jan 11, 2013 at 02:47:01PM -0600, Pawel Wodnicki wrote:
> >> On 1/11/2013 2:40 PM, Brooks Davis wrote:
> >>> On Fri, Jan 11, 2013 at 09:33:17PM +0100, Benjamin Kramer
> >>> wrote:
> >>>> 
> >>>> On 11.01.2013, at 21:31, Justin Holewinski 
> >>>> <justin.holewinski at gmail.com> wrote:
> >>>> 
> >>>>> On Fri, Jan 11, 2013 at 3:26 PM, Benjamin Kramer 
> >>>>> <benny.kra at gmail.com> wrote:
> >>>>> 
> >>>>> On 11.01.2013, at 07:36, ????????? (Wei-Ren Chen) 
> >>>>> <chenwj at iis.sinica.edu.tw> wrote:
> >>>>> 
> >>>>>> Hi Pawel,
> >>>>>> 
> >>>>>> PTX already be replaced with NVPTX. However, PTX
> >>>>>> subdirectory still sit in lib/Target in 3.2
release. Do
> >>>>>> you think update the release tarball is a good
idea? Also
> >>>>>> could you remove it from the trunk?
> >>>>> 
> >>>>> Please do not, under no circumstances, change the 3.2
> >>>>> release tarballs at this point. They are mirrored
around
> >>>>> the world now with cryptographic hashes and
signatures.
> >>>>> Changing them will break things for many people,
especially
> >>>>> for an extremely minor thing like an empty directory.
> >>>>> 
> >>>>> I'm not sure if Pawel's tarball change should
be reverted
> >>>>> now as it already caused uproar, so changing it back
might
> >>>>> only make matters worse.
> >>>>> 
> >>>>> The tarballs were changed?
> >>>> 
> >>>> r172208
> >>> 
> >>> I finally updated the FreeBSD ports yesterday and today a user
> >>>  complained about distfile changes.  IMO, this revision should
> >>> be reverted or all the other BSDs will have to chase checksums
> >>> as well.
> >>> 
> >>> If you really want to remove the directory, ship a 3.2.1
> >>> tarball rather than screwing all the downstream consumers
who's
> >>>  infrastructure exists to detect trojan'd tarballs.
> >> 
> >> Tarball is signed, it is not trjoan. Your infrastructure should
> >> be able to deal with it?
> > 
> > The FreeBSD ports collection maintains a set of sha256 hashes for 
> > each distfile.  The system can deal with them changing, but it's
> > an inconvenience to port maintainers and users.  Even if we did
> > have infrastructure to verify the signatures we'd still have to
> > check the contents and would not just trust the signature since
> > there's always a risk of keys being compromised.
> > 
> > -- Brooks
> > 
> 
>  At first your reaction has puzzled me, but despite a few fire
> breathing dragons getting loose and after carefully re-reading
> these posts over the weekend I think I understand the underlying
> issue.
> 
>  Simply put we both take this release seriously.
> 
>  The problem is the release process is not entirely
> defined. Which is reflected in question from David Blaikie
> (the other post I am quoting here).
> 
> > Which release process document are you referring to that indicates
> > that?
> > 
> 
>  Simple answer is the one on the llvm.org website.
> But the real answer is, it all depends on how you read it.
> Apparently, we read it differently which is not that
> surprising as even our well defined compilers have
> a bit of a problem with undefined behavior.
> 
> So what can we expect from a release process document?
> My hope is that we have reached a sort of sequence
> point and the ambiguity of the release process
> can be resolved despite all the side effects.
> 
> 
> David also stated this:
> 
> > Generally, so far as I know, this kind of post-release change is
> > unprecedented.
> 
>  I agree with this conclusion but even more unprecedented
> is to a expect timely and problem free release with zero and
> I'll repeat again zero involvement from the wider community that
> depends on the clang+llvm!
>  This is how the 3.2 release has happened, just compare
> the list of 3.2 to 3.1 binaries and BTW not a single
> 3.2 binary has been produced with the help of a respective
> project, distribution or who ever.
> 
>  I hear the cries of your wounded infrastructures
> and I understand that in the days of forged root certificates
> my signature on the tarballs may not guarantee much of a
> security.
> 
>  But why didn't you get involved in the 3.2 release?
>
>  We could have resolved all this in the 2 months since
> the 3.2 release got rolling. Everybody knew about the
> changes for the 3.2 release, simple e-mail to the new
> LLVMRM would go a long way to resolve any security or
> other release related issues not to mention a bit
> of a courtesy.
I don't understand how my lack of involvement with this release has
anything to do with your decision to fix what appears to me to be
a non-problem in the release tarball well after the release and in
the process do something (replacing release tarballs) that is quite
simply NOT DONE to by projects who wish to treat their downstreams
with respect.  This is something many projects end up learning due to
the painful way so it's not something you should feel bad about as
long as the project learns the right lesson(s).

If anything I'd be more annoyed if I'd actually gotten the port updated
sooner since then it would have worked for more than a day.  As it
is, I could really use a final decision on which tarball will be at
http://www.llvm.org/releases/3.2/llvm-3.2.src.tar.gz in perpetuity so I
can change the port or not as required and avoid churn.

-- Brooks
> 
>  I am doing this as service to the LLVM community
> but frankly I'd rather be chasing my dragons of
> probability now.
> 
> Pawe??
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130114/71088329/attachment.sig>