llvm dev - Oct 2014 - [LLVMdev] Recent changes in -gmlt break sample profiling

On Fri Oct 24 2014 at 6:21:14 PM David Blaikie <dblaikie at gmail.com>
wrote:
> On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at
google.com>
> wrote:
>
>>
>>
>> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com>
>> wrote:
>>
>>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo at
google.com>
>>> wrote:
>>>
>>>> I'm not sure if this was intended, but it's going to be
a problem for
>>>> sample profiles.
>>>>
>>>> When we compile with -gmlt, the profiler expects to find the
line
>>>> number for all the function headers so that it can compute
relative line
>>>> locations for the profile.
>>>>
>>>> The tool that reads the ELF binary is not finding them, so it
writes
>>>> out absolute line numbers, which are impossible to match during
the
>>>> profile-use phase.
>>>>
>>>> The problem seems to be that we are missing DW_TAG_subprogram
for all
>>>> the functions in the file.
>>>>
>>>> Attached are the dwarf dumps of the same program. One compiled
with my
>>>> system's clang 3.4 and the other with today's trunk. In
both compiles, I
>>>> used -gline-tables-only.
>>>>
>>>> The trunk version is missing all the subprogram tags for the
functions
>>>> in the file. This breaks the sample profiler.
>>>>
>>>> Should I file a bug, or is -gmlt going to be like this from now
on? The
>>>> latter would be a problem for us.
>>>>
>>>
>>> Open to negotiation, but this change is intentional ( for details,
see
>>> the commit:
http://llvm.org/viewvc/llvm-project?rev=218129&view=rev ).
>>>
>>
>> Well, this breaks us. Very hard. It absolutely blocks us from using
>> SamplePGO with LLVM.
>>
>
> Sorry about that - I knew ASan needed gmlt data of a certain form and
> worked carefully to ensure llvm-symbolizer could still symbolize with these
> changes, but wasn't aware of this particular dependence from PGO (just
> assumed it used the line table directly).
>
It does, but it uses relative line numbers. That's why it needs the source
locs for the function headers.

>
>
>> The alternative would be to make the compiler use absolute line
numbers,
>> but in the experience we've collected with GCC, this makes the
profiles
>> very brittle wrt source changes.
>>
>
> It'd be interesting to see the data - I guess you throw out profiles
that
> don't match on a per-function basis? So adding a few lines to one
function
> will invalidate the profile for that function, but not all the subsequent
> ones in the file?
>
Right. Dehao started using absolute numbers, but then moved to relative
numbers when he saw that the performance drops were fairly pronounced as
the profile aged. I don't recall the exact drops he noticed.

>
>
>> I don't have a better idea atm. Would there be any other way to
generate
>> relative line numbers?
>>
>
> Possibly.
>
>
>>   Perhaps we could use the first line number we find with samples?
>>
>
> Probably - I guess you use the ELF symbol table to compute the address
> range of each function? Given that, you could find the smallest line number
> in that address range in the line table, and make everything else relative
> to that.
>
That could probably work, but I'm not sure how the ELF reading code in the
conversion tool does this calculation. I'll check it out.

>
>
>> The problem here is that this line will shift, depending on how the
>> profile was obtained.
>>
>
> Not sure what you're referring to here ^
>
Say the function starts at line 20. If in one profile run, we get samples
at line 20 and line 23, then we'll compute the relative locations as 0 and
3. But if the first sample you get is at line 21, then you'll compute the
relative locations as  0 and 2.

Using the address ranges in the line table may be the way to go. I'll look
at this next week.


Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141024/21525a74/attachment.html>

Diego,

I think sampleFDO needs to be designed in a way which can protect itself
from future breakage like this. The roots in the unnecessary dependency of
sample FDO on gmlt setting. It is totally reasonable to tune debug binary
size by changes like this.

The right way is to fix this is to introduce an internal -g<...> flag for
use by sampleFDO -- it will have a fixed definition of what needs to be
emitted.

David

On Fri Oct 24 2014 at 3:34:36 PM Diego Novillo <dnovillo at google.com>
wrote:
> On Fri Oct 24 2014 at 6:21:14 PM David Blaikie <dblaikie at
gmail.com> wrote:
>
>> On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at
google.com>
>> wrote:
>>
>>>
>>>
>>> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com>
>>> wrote:
>>>
>>>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo at
google.com>
>>>> wrote:
>>>>
>>>>> I'm not sure if this was intended, but it's going
to be a problem for
>>>>> sample profiles.
>>>>>
>>>>> When we compile with -gmlt, the profiler expects to find
the line
>>>>> number for all the function headers so that it can compute
relative line
>>>>> locations for the profile.
>>>>>
>>>>> The tool that reads the ELF binary is not finding them, so
it writes
>>>>> out absolute line numbers, which are impossible to match
during the
>>>>> profile-use phase.
>>>>>
>>>>> The problem seems to be that we are missing
DW_TAG_subprogram for all
>>>>> the functions in the file.
>>>>>
>>>>> Attached are the dwarf dumps of the same program. One
compiled with my
>>>>> system's clang 3.4 and the other with today's
trunk. In both compiles, I
>>>>> used -gline-tables-only.
>>>>>
>>>>> The trunk version is missing all the subprogram tags for
the functions
>>>>> in the file. This breaks the sample profiler.
>>>>>
>>>>> Should I file a bug, or is -gmlt going to be like this from
now on?
>>>>> The latter would be a problem for us.
>>>>>
>>>>
>>>> Open to negotiation, but this change is intentional ( for
details, see
>>>> the commit:
http://llvm.org/viewvc/llvm-project?rev=218129&view=rev ).
>>>>
>>>
>>> Well, this breaks us. Very hard. It absolutely blocks us from using
>>> SamplePGO with LLVM.
>>>
>>
>> Sorry about that - I knew ASan needed gmlt data of a certain form and
>> worked carefully to ensure llvm-symbolizer could still symbolize with
these
>> changes, but wasn't aware of this particular dependence from PGO
(just
>> assumed it used the line table directly).
>>
>
> It does, but it uses relative line numbers. That's why it needs the
source
> locs for the function headers.
>
>
>>
>>
>>> The alternative would be to make the compiler use absolute line
numbers,
>>> but in the experience we've collected with GCC, this makes the
profiles
>>> very brittle wrt source changes.
>>>
>>
>> It'd be interesting to see the data - I guess you throw out
profiles that
>> don't match on a per-function basis? So adding a few lines to one
function
>> will invalidate the profile for that function, but not all the
subsequent
>> ones in the file?
>>
>
> Right. Dehao started using absolute numbers, but then moved to relative
> numbers when he saw that the performance drops were fairly pronounced as
> the profile aged. I don't recall the exact drops he noticed.
>
>
>>
>>
>>> I don't have a better idea atm. Would there be any other way to
generate
>>> relative line numbers?
>>>
>>
>> Possibly.
>>
>>
>>>   Perhaps we could use the first line number we find with samples?
>>>
>>
>> Probably - I guess you use the ELF symbol table to compute the address
>> range of each function? Given that, you could find the smallest line
number
>> in that address range in the line table, and make everything else
relative
>> to that.
>>
>
> That could probably work, but I'm not sure how the ELF reading code in
the
> conversion tool does this calculation. I'll check it out.
>
>
>>
>>
>>> The problem here is that this line will shift, depending on how the
>>> profile was obtained.
>>>
>>
>> Not sure what you're referring to here ^
>>
>
> Say the function starts at line 20. If in one profile run, we get samples
> at line 20 and line 23, then we'll compute the relative locations as 0
and
> 3. But if the first sample you get is at line 21, then you'll compute
the
> relative locations as  0 and 2.
>
> Using the address ranges in the line table may be the way to go. I'll
look
> at this next week.
>
>
> Diego.
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141024/53131505/attachment.html>

This sounds like a problem best solved by tracking source code movement via
your source control system.
If you know the commit of the code that produced the sample, you should be
able to use source control history / diffs to translate absolute line
numbers to the location where the source has moved.
This would have the added advantage of highlighting where those samples are
likely to be useless or completely misleading.

On Sat, Oct 25, 2014 at 8:57 AM, Diego Novillo <dnovillo at google.com>
wrote:
>
>
> On Fri Oct 24 2014 at 6:21:14 PM David Blaikie <dblaikie at
gmail.com> wrote:
>
>> On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at
google.com>
>> wrote:
>>
>>>
>>>
>>> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com>
>>> wrote:
>>>
>>>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo at
google.com>
>>>> wrote:
>>>>
>>>>> I'm not sure if this was intended, but it's going
to be a problem for
>>>>> sample profiles.
>>>>>
>>>>> When we compile with -gmlt, the profiler expects to find
the line
>>>>> number for all the function headers so that it can compute
relative line
>>>>> locations for the profile.
>>>>>
>>>>> The tool that reads the ELF binary is not finding them, so
it writes
>>>>> out absolute line numbers, which are impossible to match
during the
>>>>> profile-use phase.
>>>>>
>>>>> The problem seems to be that we are missing
DW_TAG_subprogram for all
>>>>> the functions in the file.
>>>>>
>>>>> Attached are the dwarf dumps of the same program. One
compiled with my
>>>>> system's clang 3.4 and the other with today's
trunk. In both compiles, I
>>>>> used -gline-tables-only.
>>>>>
>>>>> The trunk version is missing all the subprogram tags for
the functions
>>>>> in the file. This breaks the sample profiler.
>>>>>
>>>>> Should I file a bug, or is -gmlt going to be like this from
now on?
>>>>> The latter would be a problem for us.
>>>>>
>>>>
>>>> Open to negotiation, but this change is intentional ( for
details, see
>>>> the commit:
http://llvm.org/viewvc/llvm-project?rev=218129&view=rev ).
>>>>
>>>
>>> Well, this breaks us. Very hard. It absolutely blocks us from using
>>> SamplePGO with LLVM.
>>>
>>
>> Sorry about that - I knew ASan needed gmlt data of a certain form and
>> worked carefully to ensure llvm-symbolizer could still symbolize with
these
>> changes, but wasn't aware of this particular dependence from PGO
(just
>> assumed it used the line table directly).
>>
>
> It does, but it uses relative line numbers. That's why it needs the
source
> locs for the function headers.
>
>
>>
>>
>>> The alternative would be to make the compiler use absolute line
numbers,
>>> but in the experience we've collected with GCC, this makes the
profiles
>>> very brittle wrt source changes.
>>>
>>
>> It'd be interesting to see the data - I guess you throw out
profiles that
>> don't match on a per-function basis? So adding a few lines to one
function
>> will invalidate the profile for that function, but not all the
subsequent
>> ones in the file?
>>
>
> Right. Dehao started using absolute numbers, but then moved to relative
> numbers when he saw that the performance drops were fairly pronounced as
> the profile aged. I don't recall the exact drops he noticed.
>
>
>>
>>
>>> I don't have a better idea atm. Would there be any other way to
generate
>>> relative line numbers?
>>>
>>
>> Possibly.
>>
>>
>>>   Perhaps we could use the first line number we find with samples?
>>>
>>
>> Probably - I guess you use the ELF symbol table to compute the address
>> range of each function? Given that, you could find the smallest line
number
>> in that address range in the line table, and make everything else
relative
>> to that.
>>
>
> That could probably work, but I'm not sure how the ELF reading code in
the
> conversion tool does this calculation. I'll check it out.
>
>
>>
>>
>>> The problem here is that this line will shift, depending on how the
>>> profile was obtained.
>>>
>>
>> Not sure what you're referring to here ^
>>
>
> Say the function starts at line 20. If in one profile run, we get samples
> at line 20 and line 23, then we'll compute the relative locations as 0
and
> 3. But if the first sample you get is at line 21, then you'll compute
the
> relative locations as  0 and 2.
>
> Using the address ranges in the line table may be the way to go. I'll
look
> at this next week.
>
>
> Diego.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141027/898edbb2/attachment.html>

On Fri, Oct 24, 2014 at 4:06 PM, Xinliang David Li <xinliangli at
gmail.com>
wrote:
> Diego,
>
> I think sampleFDO needs to be designed in a way which can protect itself
> from future breakage like this. The roots in the unnecessary dependency of
> sample FDO on gmlt setting. It is totally reasonable to tune debug binary
> size by changes like this.

FWIW, it's not 100% clear to me that this is the right change. Might be
worth chatting about on Monday.

> The right way is to fix this is to introduce an internal -g<...> flag
for
> use by sampleFDO -- it will have a fixed definition of what needs to be
> emitted.
>
This doesn't seem worth having an internal flag for. Google is probably the
primary consumer of Clang's -gline-tables-only data. If it doesn't meet
our
needs, we can lobby to make it behave the way we need it to.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141026/698c9c4a/attachment.html>

On Sun, Oct 26, 2014 at 3:47 PM, Jeremy Lakeman
<Jeremy.Lakeman at gmail.com> wrote:
> This sounds like a problem best solved by tracking source code movement via
> your source control system.
> If you know the commit of the code that produced the sample, you should be
> able to use source control history / diffs to translate absolute line
> numbers to the location where the source has moved.
> This would have the added advantage of highlighting where those samples are
> likely to be useless or completely misleading.
This particular scheme might not work, but I was thinking of something
similar last week.

Basically you'd use the same algorithms that the source control system
uses when merging or doing whitespace only diffs etc to construct a
better correspondence between source lines. The problem then comes
down to "we added some hot code that doesn't have any samples" at
which point you'd want the optimizer to pay attention to propagating
weights along edges without them.

Anyhow, some food for thought.

-eric
>
> On Sat, Oct 25, 2014 at 8:57 AM, Diego Novillo <dnovillo at
google.com> wrote:
>>
>>
>>
>> On Fri Oct 24 2014 at 6:21:14 PM David Blaikie <dblaikie at
gmail.com> wrote:
>>>
>>> On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at
google.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com>
>>>> wrote:
>>>>>
>>>>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo
at google.com>
>>>>> wrote:
>>>>>>
>>>>>> I'm not sure if this was intended, but it's
going to be a problem for
>>>>>> sample profiles.
>>>>>>
>>>>>> When we compile with -gmlt, the profiler expects to
find the line
>>>>>> number for all the function headers so that it can
compute relative line
>>>>>> locations for the profile.
>>>>>>
>>>>>> The tool that reads the ELF binary is not finding them,
so it writes
>>>>>> out absolute line numbers, which are impossible to
match during the
>>>>>> profile-use phase.
>>>>>>
>>>>>> The problem seems to be that we are missing
DW_TAG_subprogram for all
>>>>>> the functions in the file.
>>>>>>
>>>>>> Attached are the dwarf dumps of the same program. One
compiled with my
>>>>>> system's clang 3.4 and the other with today's
trunk. In both compiles, I
>>>>>> used -gline-tables-only.
>>>>>>
>>>>>> The trunk version is missing all the subprogram tags
for the functions
>>>>>> in the file. This breaks the sample profiler.
>>>>>>
>>>>>> Should I file a bug, or is -gmlt going to be like this
from now on?
>>>>>> The latter would be a problem for us.
>>>>>
>>>>>
>>>>> Open to negotiation, but this change is intentional ( for
details, see
>>>>> the commit:
http://llvm.org/viewvc/llvm-project?rev=218129&view=rev ).
>>>>
>>>>
>>>> Well, this breaks us. Very hard. It absolutely blocks us from
using
>>>> SamplePGO with LLVM.
>>>
>>>
>>> Sorry about that - I knew ASan needed gmlt data of a certain form
and
>>> worked carefully to ensure llvm-symbolizer could still symbolize
with these
>>> changes, but wasn't aware of this particular dependence from
PGO (just
>>> assumed it used the line table directly).
>>
>>
>> It does, but it uses relative line numbers. That's why it needs the
source
>> locs for the function headers.
>>
>>>
>>>
>>>>
>>>> The alternative would be to make the compiler use absolute line
numbers,
>>>> but in the experience we've collected with GCC, this makes
the profiles very
>>>> brittle wrt source changes.
>>>
>>>
>>> It'd be interesting to see the data - I guess you throw out
profiles that
>>> don't match on a per-function basis? So adding a few lines to
one function
>>> will invalidate the profile for that function, but not all the
subsequent
>>> ones in the file?
>>
>>
>> Right. Dehao started using absolute numbers, but then moved to relative
>> numbers when he saw that the performance drops were fairly pronounced
as the
>> profile aged. I don't recall the exact drops he noticed.
>>
>>>
>>>
>>>>
>>>> I don't have a better idea atm. Would there be any other
way to generate
>>>> relative line numbers?
>>>
>>>
>>> Possibly.
>>>
>>>>
>>>>   Perhaps we could use the first line number we find with
samples?
>>>
>>>
>>> Probably - I guess you use the ELF symbol table to compute the
address
>>> range of each function? Given that, you could find the smallest
line number
>>> in that address range in the line table, and make everything else
relative
>>> to that.
>>
>>
>> That could probably work, but I'm not sure how the ELF reading code
in the
>> conversion tool does this calculation. I'll check it out.
>>
>>>
>>>
>>>>
>>>> The problem here is that this line will shift, depending on how
the
>>>> profile was obtained.
>>>
>>>
>>> Not sure what you're referring to here ^
>>
>>
>> Say the function starts at line 20. If in one profile run, we get
samples
>> at line 20 and line 23, then we'll compute the relative locations
as 0 and
>> 3. But if the first sample you get is at line 21, then you'll
compute the
>> relative locations as  0 and 2.
>>
>> Using the address ranges in the line table may be the way to go.
I'll look
>> at this next week.
>>
>>
>> Diego.
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

On Fri, Oct 24, 2014 at 3:27 PM, Diego Novillo <dnovillo at google.com>
wrote:
>
>
> On Fri Oct 24 2014 at 6:21:14 PM David Blaikie <dblaikie at
gmail.com> wrote:
>>
>> On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at
google.com>
>> wrote:
>>>
>>>
>>>
>>> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com>
>>> wrote:
>>>>
>>>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo <dnovillo at
google.com>
>>>> wrote:
>>>>>
>>>>> I'm not sure if this was intended, but it's going
to be a problem for
>>>>> sample profiles.
>>>>>
>>>>> When we compile with -gmlt, the profiler expects to find
the line
>>>>> number for all the function headers so that it can compute
relative line
>>>>> locations for the profile.
>>>>>
>>>>> The tool that reads the ELF binary is not finding them, so
it writes
>>>>> out absolute line numbers, which are impossible to match
during the
>>>>> profile-use phase.
>>>>>
>>>>> The problem seems to be that we are missing
DW_TAG_subprogram for all
>>>>> the functions in the file.
>>>>>
>>>>> Attached are the dwarf dumps of the same program. One
compiled with my
>>>>> system's clang 3.4 and the other with today's
trunk. In both compiles, I
>>>>> used -gline-tables-only.
>>>>>
>>>>> The trunk version is missing all the subprogram tags for
the functions
>>>>> in the file. This breaks the sample profiler.
>>>>>
>>>>> Should I file a bug, or is -gmlt going to be like this from
now on? The
>>>>> latter would be a problem for us.
>>>>
>>>>
>>>> Open to negotiation, but this change is intentional ( for
details, see
>>>> the commit:
http://llvm.org/viewvc/llvm-project?rev=218129&view=rev ).
>>>
>>>
>>> Well, this breaks us. Very hard. It absolutely blocks us from using
>>> SamplePGO with LLVM.
>>
>>
>> Sorry about that - I knew ASan needed gmlt data of a certain form and
>> worked carefully to ensure llvm-symbolizer could still symbolize with
these
>> changes, but wasn't aware of this particular dependence from PGO
(just
>> assumed it used the line table directly).
>
>
> It does, but it uses relative line numbers. That's why it needs the
source
> locs for the function headers.
>
>>
>>
>>>
>>> The alternative would be to make the compiler use absolute line
numbers,
>>> but in the experience we've collected with GCC, this makes the
profiles very
>>> brittle wrt source changes.
>>
>>
>> It'd be interesting to see the data - I guess you throw out
profiles that
>> don't match on a per-function basis? So adding a few lines to one
function
>> will invalidate the profile for that function, but not all the
subsequent
>> ones in the file?
>
>
> Right. Dehao started using absolute numbers, but then moved to relative
> numbers when he saw that the performance drops were fairly pronounced as
the
> profile aged. I don't recall the exact drops he noticed.
>
>>
>>
>>>
>>> I don't have a better idea atm. Would there be any other way to
generate
>>> relative line numbers?
>>
>>
>> Possibly.
>>
>>>
>>>   Perhaps we could use the first line number we find with samples?
>>
>>
>> Probably - I guess you use the ELF symbol table to compute the address
>> range of each function? Given that, you could find the smallest line
number
>> in that address range in the line table, and make everything else
relative
>> to that.
>
>
> That could probably work, but I'm not sure how the ELF reading code in
the
> conversion tool does this calculation. I'll check it out.
>
>>
>>
>>>
>>> The problem here is that this line will shift, depending on how the
>>> profile was obtained.
>>
>>
>> Not sure what you're referring to here ^
>
>
> Say the function starts at line 20. If in one profile run, we get samples
at
> line 20 and line 23, then we'll compute the relative locations as 0 and
3.
> But if the first sample you get is at line 21, then you'll compute the
> relative locations as  0 and 2.
>
> Using the address ranges in the line table may be the way to go. I'll
look
> at this next week.
>
I'm nearly certain this is the way to go here.

-eric

On Sun, Oct 26, 2014 at 4:59 PM, Eric Christopher <echristo at gmail.com>
wrote:
> On Fri, Oct 24, 2014 at 3:27 PM, Diego Novillo <dnovillo at
google.com>
> wrote:
> >
> >
> > On Fri Oct 24 2014 at 6:21:14 PM David Blaikie <dblaikie at
gmail.com>
> wrote:
> >>
> >> On Fri, Oct 24, 2014 at 3:16 PM, Diego Novillo <dnovillo at
google.com>
> >> wrote:
> >>>
> >>>
> >>>
> >>> On Fri Oct 24 2014 at 6:11:21 PM David Blaikie <dblaikie at
gmail.com>
> >>> wrote:
> >>>>
> >>>> On Fri, Oct 24, 2014 at 2:48 PM, Diego Novillo
<dnovillo at google.com>
> >>>> wrote:
> >>>>>
> >>>>> I'm not sure if this was intended, but it's
going to be a problem for
> >>>>> sample profiles.
> >>>>>
> >>>>> When we compile with -gmlt, the profiler expects to
find the line
> >>>>> number for all the function headers so that it can
compute relative
> line
> >>>>> locations for the profile.
> >>>>>
> >>>>> The tool that reads the ELF binary is not finding
them, so it writes
> >>>>> out absolute line numbers, which are impossible to
match during the
> >>>>> profile-use phase.
> >>>>>
> >>>>> The problem seems to be that we are missing
DW_TAG_subprogram for all
> >>>>> the functions in the file.
> >>>>>
> >>>>> Attached are the dwarf dumps of the same program. One
compiled with
> my
> >>>>> system's clang 3.4 and the other with today's
trunk. In both
> compiles, I
> >>>>> used -gline-tables-only.
> >>>>>
> >>>>> The trunk version is missing all the subprogram tags
for the
> functions
> >>>>> in the file. This breaks the sample profiler.
> >>>>>
> >>>>> Should I file a bug, or is -gmlt going to be like this
from now on?
> The
> >>>>> latter would be a problem for us.
> >>>>
> >>>>
> >>>> Open to negotiation, but this change is intentional ( for
details, see
> >>>> the commit:
http://llvm.org/viewvc/llvm-project?rev=218129&view=rev
> ).
> >>>
> >>>
> >>> Well, this breaks us. Very hard. It absolutely blocks us from
using
> >>> SamplePGO with LLVM.
> >>
> >>
> >> Sorry about that - I knew ASan needed gmlt data of a certain form
and
> >> worked carefully to ensure llvm-symbolizer could still symbolize
with
> these
> >> changes, but wasn't aware of this particular dependence from
PGO (just
> >> assumed it used the line table directly).
> >
> >
> > It does, but it uses relative line numbers. That's why it needs
the
> source
> > locs for the function headers.
> >
> >>
> >>
> >>>
> >>> The alternative would be to make the compiler use absolute
line
> numbers,
> >>> but in the experience we've collected with GCC, this makes
the
> profiles very
> >>> brittle wrt source changes.
> >>
> >>
> >> It'd be interesting to see the data - I guess you throw out
profiles
> that
> >> don't match on a per-function basis? So adding a few lines to
one
> function
> >> will invalidate the profile for that function, but not all the
> subsequent
> >> ones in the file?
> >
> >
> > Right. Dehao started using absolute numbers, but then moved to
relative
> > numbers when he saw that the performance drops were fairly pronounced
as
> the
> > profile aged. I don't recall the exact drops he noticed.
> >
> >>
> >>
> >>>
> >>> I don't have a better idea atm. Would there be any other
way to
> generate
> >>> relative line numbers?
> >>
> >>
> >> Possibly.
> >>
> >>>
> >>>   Perhaps we could use the first line number we find with
samples?
> >>
> >>
> >> Probably - I guess you use the ELF symbol table to compute the
address
> >> range of each function? Given that, you could find the smallest
line
> number
> >> in that address range in the line table, and make everything else
> relative
> >> to that.
> >
> >
> > That could probably work, but I'm not sure how the ELF reading
code in
> the
> > conversion tool does this calculation. I'll check it out.
> >
> >>
> >>
> >>>
> >>> The problem here is that this line will shift, depending on
how the
> >>> profile was obtained.
> >>
> >>
> >> Not sure what you're referring to here ^
> >
> >
> > Say the function starts at line 20. If in one profile run, we get
> samples at
> > line 20 and line 23, then we'll compute the relative locations as
0 and
> 3.
> > But if the first sample you get is at line 21, then you'll compute
the
> > relative locations as  0 and 2.
> >
> > Using the address ranges in the line table may be the way to go.
I'll
> look
> > at this next week.
> >
>
> I'm nearly certain this is the way to go here.
>
How does it solve the problem?

David

>
> -eric
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141026/322c8482/attachment.html>

On Fri, Oct 24, 2014 at 4:06 PM, Xinliang David Li <xinliangli at
gmail.com>
wrote:
> Diego,
>
> I think sampleFDO needs to be designed in a way which can protect itself
> from future breakage like this. The roots in the unnecessary dependency of
> sample FDO on gmlt setting. It is totally reasonable to tune debug binary
> size by changes like this.
>
> The right way is to fix this is to introduce an internal -g<...> flag
for
> use by sampleFDO -- it will have a fixed definition of what needs to be
> emitted.
>
FWIW, I strongly disagree.

The more modes we have, the harder everything will be to support and keep
track of. The wide variety of modes used for debug information is already
really challenging to support and maintain. We shouldn't make it harder,
and less-used flags seem like the wrong direction.


At a higher level, I don't think "sample profiling" or
"asan" are good ways
to design a set of debug information that we want emitted in a particular
mode. Instead, we should look at what fundament information a collection of
tools need access to. Both sample profiling, the sanitizers, and crash
backtraces need access to a very minimal amount of information consisting
of line tables, and that is how we designed '-gline-tables-only' (the
LLVM
flag name).

And I think that both this design and Dave's change are totally fine. There
was only one thing we missed: a very particular use case for line tables
that had a particular usage pattern. The problem here is that we don't have
any profile *collection* tests in LLVM. There are some reasons for that
(its hard, etc) but we could probably work on improving it. But the correct
path is the one Dave and Diego identified -- the information *is* still
there, we just need a more clever way of extracting it. And (in addition)
we should probably add some testing strategy for this. =]
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141026/7d4946e7/attachment.html>

On Fri Oct 24 2014 at 4:06:05 PM Xinliang David Li <xinliangli at
gmail.com>
wrote:
> Diego,
>
> I think sampleFDO needs to be designed in a way which can protect itself
> from future breakage like this. The roots in the unnecessary dependency of
> sample FDO on gmlt setting. It is totally reasonable to tune debug binary
> size by changes like this.

Sure.

> The right way is to fix this is to introduce an internal -g<...> flag
for
> use by sampleFDO -- it will have a fixed definition of what needs to be
> emitted.
>
Not so sure. Semantically, -gmlt is exactly what samplefdo needs. Relying
on the subprogram tag for the function itself was an implementation detail.

Adding yet another debug mode that's very much like -gmlt but not quite is
going to be confusing for users.

I *believe* we can reproduce the same behaviour by using the source
location ranges. I'm still not quite sure how, but I've got Eric and
David
within smacking distance, so I'll make myself a nuisance until they give me
enough hints.


Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141027/ae0889c6/attachment.html>

On Sun Oct 26 2014 at 3:47:51 PM Jeremy Lakeman <Jeremy.Lakeman at
gmail.com>
wrote:
> This sounds like a problem best solved by tracking source code movement
> via your source control system.
> If you know the commit of the code that produced the sample, you should be
> able to use source control history / diffs to translate absolute line
> numbers to the location where the source has moved.
> This would have the added advantage of highlighting where those samples
> are likely to be useless or completely misleading.
>
Doable in principle, but challenging in reality. I don't think this idea is
straightforward at all to implement.


Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141027/2b488262/attachment.html>