llvm dev - Oct 2014 - [LLVMdev] Recent changes in -gmlt break sample profiling

On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at gmail.com>
wrote:
> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo at
google.com>
> wrote:
>
>> I'm not sure if this was intended, but it's going to be a
problem for
>> sample profiles.
>>
>> When we compile with -gmlt, the profiler expects to find the line
number
>> for all the function headers so that it can compute relative line
locations
>> for the profile.
>>
>> The tool that reads the ELF binary is not finding them, so it writes
out
>> absolute line numbers, which are impossible to match during the
profile-use
>> phase.
>>
>> The problem seems to be that we are missing DW_TAG_subprogram for all
the
>> functions in the file.
>>
>> Attached are the dwarf dumps of the same program. One compiled with my
>> system's clang 3.4 and the other with today's trunk. In both
compiles, I
>> used -gline-tables-only.
>>
>> The trunk version is missing all the subprogram tags for the functions
in
>> the file. This breaks the sample profiler.
>>
>> Should I file a bug, or is -gmlt going to be like this from now on? The
>> latter would be a problem for us.
>>
>
> Open to negotiation, but this change is intentional ( for details, see the
> commit: http://llvm.org/viewvc/llvm-project?rev=218129&view=rev ).
>
Well, this breaks us. Very hard. It absolutely blocks us from using
SamplePGO with LLVM.

The alternative would be to make the compiler use absolute line numbers,
but in the experience we've collected with GCC, this makes the profiles
very brittle wrt source changes.

I don't have a better idea atm. Would there be any other way to generate
relative line numbers?  Perhaps we could use the first line number we find
with samples?  The problem here is that this line will shift, depending on
how the profile was obtained.

Would it be possible to add these line numbers in some other way?


Thanks.  Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141024/d1b857c0/attachment.html>

On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at google.com>
wrote:
>
>
> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com> wrote:
>
>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo at
google.com>
>> wrote:
>>
>>> I'm not sure if this was intended, but it's going to be a
problem for
>>> sample profiles.
>>>
>>> When we compile with -gmlt, the profiler expects to find the line
number
>>> for all the function headers so that it can compute relative line
locations
>>> for the profile.
>>>
>>> The tool that reads the ELF binary is not finding them, so it
writes out
>>> absolute line numbers, which are impossible to match during the
profile-use
>>> phase.
>>>
>>> The problem seems to be that we are missing DW_TAG_subprogram for
all
>>> the functions in the file.
>>>
>>> Attached are the dwarf dumps of the same program. One compiled with
my
>>> system's clang 3.4 and the other with today's trunk. In
both compiles, I
>>> used -gline-tables-only.
>>>
>>> The trunk version is missing all the subprogram tags for the
functions
>>> in the file. This breaks the sample profiler.
>>>
>>> Should I file a bug, or is -gmlt going to be like this from now on?
The
>>> latter would be a problem for us.
>>>
>>
>> Open to negotiation, but this change is intentional ( for details, see
>> the commit: http://llvm.org/viewvc/llvm-project?rev=218129&view=rev
).
>>
>
> Well, this breaks us. Very hard. It absolutely blocks us from using
> SamplePGO with LLVM.
>
Sorry about that - I knew ASan needed gmlt data of a certain form and
worked carefully to ensure llvm-symbolizer could still symbolize with these
changes, but wasn't aware of this particular dependence from PGO (just
assumed it used the line table directly).

> The alternative would be to make the compiler use absolute line numbers,
> but in the experience we've collected with GCC, this makes the profiles
> very brittle wrt source changes.
>
It'd be interesting to see the data - I guess you throw out profiles that
don't match on a per-function basis? So adding a few lines to one function
will invalidate the profile for that function, but not all the subsequent
ones in the file?

> I don't have a better idea atm. Would there be any other way to
generate
> relative line numbers?
>
Possibly.

>   Perhaps we could use the first line number we find with samples?
>
Probably - I guess you use the ELF symbol table to compute the address
range of each function? Given that, you could find the smallest line number
in that address range in the line table, and make everything else relative
to that.

> The problem here is that this line will shift, depending on how the
> profile was obtained.
>
Not sure what you're referring to here ^

>
> Would it be possible to add these line numbers in some other way?
>
>
> Thanks.  Diego.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141024/fc121fb8/attachment.html>

On Fri Oct 24 2014 at 6:21:14 PM David Blaikie <dblaikie at gmail.com>
wrote:
> On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at
google.com>
> wrote:
>
>>
>>
>> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com>
>> wrote:
>>
>>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo at
google.com>
>>> wrote:
>>>
>>>> I'm not sure if this was intended, but it's going to be
a problem for
>>>> sample profiles.
>>>>
>>>> When we compile with -gmlt, the profiler expects to find the
line
>>>> number for all the function headers so that it can compute
relative line
>>>> locations for the profile.
>>>>
>>>> The tool that reads the ELF binary is not finding them, so it
writes
>>>> out absolute line numbers, which are impossible to match during
the
>>>> profile-use phase.
>>>>
>>>> The problem seems to be that we are missing DW_TAG_subprogram
for all
>>>> the functions in the file.
>>>>
>>>> Attached are the dwarf dumps of the same program. One compiled
with my
>>>> system's clang 3.4 and the other with today's trunk. In
both compiles, I
>>>> used -gline-tables-only.
>>>>
>>>> The trunk version is missing all the subprogram tags for the
functions
>>>> in the file. This breaks the sample profiler.
>>>>
>>>> Should I file a bug, or is -gmlt going to be like this from now
on? The
>>>> latter would be a problem for us.
>>>>
>>>
>>> Open to negotiation, but this change is intentional ( for details,
see
>>> the commit:
http://llvm.org/viewvc/llvm-project?rev=218129&view=rev ).
>>>
>>
>> Well, this breaks us. Very hard. It absolutely blocks us from using
>> SamplePGO with LLVM.
>>
>
> Sorry about that - I knew ASan needed gmlt data of a certain form and
> worked carefully to ensure llvm-symbolizer could still symbolize with these
> changes, but wasn't aware of this particular dependence from PGO (just
> assumed it used the line table directly).
>
It does, but it uses relative line numbers. That's why it needs the source
locs for the function headers.

>
>
>> The alternative would be to make the compiler use absolute line
numbers,
>> but in the experience we've collected with GCC, this makes the
profiles
>> very brittle wrt source changes.
>>
>
> It'd be interesting to see the data - I guess you throw out profiles
that
> don't match on a per-function basis? So adding a few lines to one
function
> will invalidate the profile for that function, but not all the subsequent
> ones in the file?
>
Right. Dehao started using absolute numbers, but then moved to relative
numbers when he saw that the performance drops were fairly pronounced as
the profile aged. I don't recall the exact drops he noticed.

>
>
>> I don't have a better idea atm. Would there be any other way to
generate
>> relative line numbers?
>>
>
> Possibly.
>
>
>>   Perhaps we could use the first line number we find with samples?
>>
>
> Probably - I guess you use the ELF symbol table to compute the address
> range of each function? Given that, you could find the smallest line number
> in that address range in the line table, and make everything else relative
> to that.
>
That could probably work, but I'm not sure how the ELF reading code in the
conversion tool does this calculation. I'll check it out.

>
>
>> The problem here is that this line will shift, depending on how the
>> profile was obtained.
>>
>
> Not sure what you're referring to here ^
>
Say the function starts at line 20. If in one profile run, we get samples
at line 20 and line 23, then we'll compute the relative locations as 0 and
3. But if the first sample you get is at line 21, then you'll compute the
relative locations as  0 and 2.

Using the address ranges in the line table may be the way to go. I'll look
at this next week.


Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141024/21525a74/attachment.html>