llvm dev - Apr 2014 - [LLVMdev] RFC: Binary format for instrumentation based profiling data

On Wed, Apr 16, 2014 at 2:15 PM, Bob Wilson <bob.wilson at apple.com>
wrote:
> On Apr 16, 2014, at 2:01 PM, Chandler Carruth <chandlerc at
google.com>
> wrote:
>
>
> On Wed, Apr 16, 2014 at 10:48 AM, Bob Wilson <bob.wilson at
apple.com> wrote:
>
>> We need to settle of a file format ASAP for our internal work, but from
>> the perspective of the LLVM community, it makes sense to me that this
>> should remain wide open to change, at least until it goes out in an
>> open-source LLVM release.
>
>
> Ok, I don't really know how to reconcile this.
>
> Is it fine to make breaking, backwards incompatible changes to the file
> format in the coming months or not?
>
>
> I would really like it if we can avoid making those kinds of changes, but
> I don’t know how to justify that from an open-source LLVM project
> perspective.
>
> I can’t expect you to care about Apple’s release schedules but neither can
> I change them. It would be nice if the open-source version of LLVM can work
> with the PGO profile data used by Apple (especially because it seems that
> you guys are taking a totally different approach to PGO which won’t even
> use these same files).
>
See, I think this is backwards. I think it would be nice if Apple could use
the open source version of LLVM's profile data. But the priority for LLVM
is getting this *right*.

> If we can get a “good enough” solution into trunk now, and if we can
> continue to provide support for that format, then we can make that work. If
> you really think it’s important to continue iterating on the file format,
> without adding support for reading the earlier format, then we’ll lose
> that. It would be unfortunate for the community IMO, but you may not agree.
>
I think maintaining support for a legacy format that has never even been
part of an open source LLVM release is a really lame burden to place on the
community. It'll add complexity to Clang and the profile data tools. But
there are many things that make pragmatic sense even though they are lame.

However, I think even worrying about this is getting in the way of
developing an actual *good* file format. I think that the design
discussion, iteration, and evaluation should proceed regardless of what you
happen to do at Apple. Later on, when if you end up needing to add reading
support for some fixed "external" format we don't control, propose
that as
a separate patch that clearly factors all of the logic away into detection
and reading of an alternative file format. That way its clear which is the
supported and recommended format and design for the open source tools, and
which is just a compatibility layer for some existing files and tools that
can't be changed.

For example, this is how I plan to suggest teaching the llvm-profdata tool
to read and merge profile data produced by profiling tools outside of LLVM
like the Linux perf events, etc. It's going to come after we essentially
have the pipeline well designed and worked out, as a clearly separated
layer that just adds compatibility with specific fixed file formats we
can't easily control.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140416/26346e2c/attachment.html>

On Apr 16, 2014, at 2:35 PM, Chandler Carruth <chandlerc at google.com>
wrote:
> 
> On Wed, Apr 16, 2014 at 2:15 PM, Bob Wilson <bob.wilson at apple.com>
wrote:
> On Apr 16, 2014, at 2:01 PM, Chandler Carruth <chandlerc at
google.com> wrote:
> 
>> 
>> On Wed, Apr 16, 2014 at 10:48 AM, Bob Wilson <bob.wilson at
apple.com> wrote:
>> We need to settle of a file format ASAP for our internal work, but from
the perspective of the LLVM community, it makes sense to me that this should
remain wide open to change, at least until it goes out in an open-source LLVM
release.
>> 
>> Ok, I don't really know how to reconcile this.
>> 
>> Is it fine to make breaking, backwards incompatible changes to the file
format in the coming months or not?
> 
> I would really like it if we can avoid making those kinds of changes, but I
don’t know how to justify that from an open-source LLVM project perspective.
> 
> I can’t expect you to care about Apple’s release schedules but neither can
I change them. It would be nice if the open-source version of LLVM can work with
the PGO profile data used by Apple (especially because it seems that you guys
are taking a totally different approach to PGO which won’t even use these same
files).
> 
> See, I think this is backwards. I think it would be nice if Apple could use
the open source version of LLVM's profile data. But the priority for LLVM is
getting this *right*.
>  
> If we can get a “good enough” solution into trunk now, and if we can
continue to provide support for that format, then we can make that work. If you
really think it’s important to continue iterating on the file format, without
adding support for reading the earlier format, then we’ll lose that. It would be
unfortunate for the community IMO, but you may not agree.
> 
> I think maintaining support for a legacy format that has never even been
part of an open source LLVM release is a really lame burden to place on the
community. It'll add complexity to Clang and the profile data tools. But
there are many things that make pragmatic sense even though they are lame.
> 
> However, I think even worrying about this is getting in the way of
developing an actual *good* file format. I think that the design discussion,
iteration, and evaluation should proceed regardless of what you happen to do at
Apple.
I agree the top priority should always be "getting it right”. But I can’t
agree with this thinking completely. This has to be balanced with pragmatism. If
we completely disregard the practical concerns of commercial use, it makes LLVM
hostile towards an important group of users.

Evan
> Later on, when if you end up needing to add reading support for some fixed
"external" format we don't control, propose that as a separate
patch that clearly factors all of the logic away into detection and reading of
an alternative file format. That way its clear which is the supported and
recommended format and design for the open source tools, and which is just a
compatibility layer for some existing files and tools that can't be changed.
> 
> For example, this is how I plan to suggest teaching the llvm-profdata tool
to read and merge profile data produced by profiling tools outside of LLVM like
the Linux perf events, etc. It's going to come after we essentially have the
pipeline well designed and worked out, as a clearly separated layer that just
adds compatibility with specific fixed file formats we can't easily control.
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140418/07a0db0f/attachment.html>

On Fri, Apr 18, 2014 at 12:22 PM, Evan Cheng <evan.cheng at apple.com>
wrote:
> I agree the top priority should always be "getting it right”. But I
can’t
> agree with this thinking completely. This has to be balanced with
> pragmatism. If we completely disregard the practical concerns of commercial
> use, it makes LLVM hostile towards an important group of users.

Clearly, I can't argue with that. =] We benefit from it as well. And I
don't think I'm arguing against a pragmatic approach in this case,
although
I'm sorry if it comes off that way.

Just so we're on the same page, the only real question in my mind is: can
we make breaking changes as we iterate on the design.

What I would like to do is figure out the right design first,
incrementally, trying one format, and seeing how it does, but potentially
with backwards incompatible changes. Once we have the experience and know
what the right format is, *then* we should consider pragmatic concerns such
as how to structure support for reading legacy formats, or formats outside
of the control of the project. I think if we start off with a format that
we can't change because of external reasons (whatever they may be), it will
be much harder to incrementally get to the right design. Does that seem
reasonable (and hopefully help explain the concrete suggestion I'm making)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140418/59277e03/attachment.html>