thr3ads.net - llvm dev - [llvm-dev] Question about : lprofValueProfNodes [Dec 2017]

If this information is useful, please help other people find it:
Share via:

Xinliang David Li via llvm-dev

2017-Dec-20 01:37 UTC

[llvm-dev] Question about : lprofValueProfNodes

What Vedant said -- the profiler runtime provides buffer API for profile
dumping.  Note that value profiling dumping is not yet supported for buffer
API, but since you are using Front-end based instrumentation/profile-use,
value profiler is not turned on by default anyway.

David

On Tue, Dec 19, 2017 at 5:29 PM, Vedant Kumar <vsk at apple.com> wrote:
>
> On Dec 19, 2017, at 5:16 PM, Moshtaghi, Alireza <
> Alireza.Moshtaghi at netapp.com> wrote:
>
> Thank you
> So it does not seem to be relevant for what I’m trying to do.
> I’m doing something unconventional.
> The objective is to implement PGO and code coverage on a system that does
> not exit and does not have any file io, or any of  stdc libraries that
> libclang-profile is using. (more like a kernel)
> So what I’m trying to do is instead of calling __llvm_profile_write_file
> () from the application, read the sections __llvm_prf_data,
> __llvm_prf_names, __llvm_prf_cnts and __llvm_prf_vnds after the critical
> tasks are done and transfer them to outside of the system.
>
>
> You may already have worked this out, but for completeness I'll mention
> that this is typically done by:
> 1. Allocating a buffer big enough to contain all the data
> (see __llvm_profile_get_size_for_buffer_internal),
> 2. Filling out the buffer (see __llvm_profile_write_buffer_internal), and
> 3. Transferring the buffer to some host machine.
>
>
> Then dump these sections in a char * array in a c file and attribute them
> to be in the associated section.
>
>
> I'm not sure I follow -- why is a C file needed?
>
> Once you have the buffer from step 3 on the host machine, you've got a
> well-formed raw profile. You should be able to fwrite() it to e.g
> 'default.profraw' and test that it's a valid profile with
llvm-profdata.
>
>
>  Then compile that file with –u__llvm_profile_runtime to create an
> executable that calls __llvm_profile_write_file to dump my profraw data.
>
> Sofar, I’m able to do the mechanics but seems like something is going
> wrong because because fprofile-instr-use does not find profile data for the
> source files.
>
>
> What error are you getting exactly? Have you confirmed that your profile
> is valid?
>
> vedant
>
>
> Any suggestion?
>
> Thanks
> A
>
> From: <vsk at apple.com> on behalf of Vedant Kumar <vsk at
apple.com>
> Date: Tuesday, December 19, 2017 at 11:32 AM
> To: "Moshtaghi, Alireza" <Alireza.Moshtaghi at netapp.com>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>, Xinliang David Li <
> davidxl at google.com>
> Subject: Re: [llvm-dev] Question about : lprofValueProfNodes
>
> Hi,
>
>
> On Dec 19, 2017, at 10:26 AM, Moshtaghi, Alireza via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi
> This array is defined in compiler-rt: InstrProfilingValue.c but I can’t
> find where it is used?
>
> It's used in allocateOneNode(). Incrementing the current vnode pointer
> gives a fresh node (possibly backed by the shared pool).
>
>
>
> And the comment on it does not say much about why we need it either.
>
> It's used to avoid calling malloc(), which David (CC'd) found to be
a
> performance improvement.
>
> best,
> vedant
>
>
> Can someone explain why we need this and where it is used?
>
> /* A shared static pool in addition to the vnodes statically
> * allocated by the compiler.  */
> COMPILER_RT_VISIBILITY ValueProfNode
>     lprofValueProfNodes[INSTR_PROF_VNODE_POOL_SIZE]
> COMPILER_RT_SECTION(COMPILER_RT_SEG INSTR_PROF_VNODES_SECT_NAME_STR);
>
> Thanks
> A
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171219/16d50f44/attachment.html>

Moshtaghi, Alireza via llvm-dev

2017-Dec-20 20:38 UTC

head link

[llvm-dev] Question about : lprofValueProfNodes

For __llvm_profile_get_size_for_buffer_internal and
__llvm_profile_write_buffer_internal to work in our system, we have to modify
the compiler-rt to use our kernel’s memory allocation. To avoid that, I was
hoping to be able to do all the processing offline by copying only the sections
to that offline process.

We currently have a workaround, but please let me know if you think just copying
the __llvm_prf_xxx sections and doing the processing offline is not possible
with the current API of compiler-rt profiling.

Thanks
A

From: Xinliang David Li <davidxl at google.com>
Date: Tuesday, December 19, 2017 at 5:38 PM
To: Vedant Kumar <vsk at apple.com>
Cc: "Moshtaghi, Alireza" <Alireza.Moshtaghi at netapp.com>,
llvm-dev <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] Question about : lprofValueProfNodes

What Vedant said -- the profiler runtime provides buffer API for profile
dumping.  Note that value profiling dumping is not yet supported for buffer API,
but since you are using Front-end based instrumentation/profile-use, value
profiler is not turned on by default anyway.

David

On Tue, Dec 19, 2017 at 5:29 PM, Vedant Kumar <vsk at apple.com<mailto:vsk
at apple.com>> wrote:

On Dec 19, 2017, at 5:16 PM, Moshtaghi, Alireza <Alireza.Moshtaghi at
netapp.com<mailto:Alireza.Moshtaghi at netapp.com>> wrote:

Thank you
So it does not seem to be relevant for what I’m trying to do.
I’m doing something unconventional.
The objective is to implement PGO and code coverage on a system that does not
exit and does not have any file io, or any of  stdc libraries that
libclang-profile is using. (more like a kernel)
So what I’m trying to do is instead of calling __llvm_profile_write_file () from
the application, read the sections __llvm_prf_data, __llvm_prf_names,
__llvm_prf_cnts and __llvm_prf_vnds after the critical tasks are done and
transfer them to outside of the system.

You may already have worked this out, but for completeness I'll mention that
this is typically done by:
1. Allocating a buffer big enough to contain all the data (see
__llvm_profile_get_size_for_buffer_internal),
2. Filling out the buffer (see __llvm_profile_write_buffer_internal), and
3. Transferring the buffer to some host machine.



Then dump these sections in a char * array in a c file and attribute them to be
in the associated section.

I'm not sure I follow -- why is a C file needed?

Once you have the buffer from step 3 on the host machine, you've got a
well-formed raw profile. You should be able to fwrite() it to e.g
'default.profraw' and test that it's a valid profile with
llvm-profdata.



 Then compile that file with –u__llvm_profile_runtime to create an executable
that calls __llvm_profile_write_file to dump my profraw data.

Sofar, I’m able to do the mechanics but seems like something is going wrong
because because fprofile-instr-use does not find profile data for the source
files.

What error are you getting exactly? Have you confirmed that your profile is
valid?

vedant



Any suggestion?

Thanks
A

From: <vsk at apple.com<mailto:vsk at apple.com>> on behalf of
Vedant Kumar <vsk at apple.com<mailto:vsk at apple.com>>
Date: Tuesday, December 19, 2017 at 11:32 AM
To: "Moshtaghi, Alireza" <Alireza.Moshtaghi at
netapp.com<mailto:Alireza.Moshtaghi at netapp.com>>
Cc: llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>, Xinliang David Li <davidxl at
google.com<mailto:davidxl at google.com>>
Subject: Re: [llvm-dev] Question about : lprofValueProfNodes

Hi,


On Dec 19, 2017, at 10:26 AM, Moshtaghi, Alireza via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

Hi
This array is defined in compiler-rt: InstrProfilingValue.c but I can’t find
where it is used?

It's used in allocateOneNode(). Incrementing the current vnode pointer gives
a fresh node (possibly backed by the shared pool).



And the comment on it does not say much about why we need it either.

It's used to avoid calling malloc(), which David (CC'd) found to be a
performance improvement.

best,
vedant


Can someone explain why we need this and where it is used?

/* A shared static pool in addition to the vnodes statically
* allocated by the compiler.  */
COMPILER_RT_VISIBILITY ValueProfNode
    lprofValueProfNodes[INSTR_PROF_VNODE_POOL_SIZE]
COMPILER_RT_SECTION(COMPILER_RT_SEG INSTR_PROF_VNODES_SECT_NAME_STR);

Thanks
A

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171220/1cc4ec8a/attachment.html>

Xinliang David Li via llvm-dev

2017-Dec-20 21:07 UTC

head link

[llvm-dev] Question about : lprofValueProfNodes

On Wed, Dec 20, 2017 at 12:38 PM, Moshtaghi, Alireza <
Alireza.Moshtaghi at netapp.com> wrote:
> For __llvm_profile_get_size_for_buffer_internal and
> __llvm_profile_write_buffer_internal to work in our system, we have to
> modify the compiler-rt to use our kernel’s memory allocation. To avoid
> that, I was hoping to be able to do all the processing offline by copying
> only the sections to that offline process.
>
>
>
> We currently have a workaround, but please let me know if you think just
> copying the __llvm_prf_xxx sections and doing the processing offline is not
> possible with the current API of compiler-rt profiling.
>
It should work if the raw profile header is initialized properly with sizes
of data, counter, and name sections as well as counter base address.

David

>
>
> Thanks
>
> A
>
>
>
> *From: *Xinliang David Li <davidxl at google.com>
> *Date: *Tuesday, December 19, 2017 at 5:38 PM
> *To: *Vedant Kumar <vsk at apple.com>
> *Cc: *"Moshtaghi, Alireza" <Alireza.Moshtaghi at
netapp.com>, llvm-dev <
> llvm-dev at lists.llvm.org>
>
> *Subject: *Re: [llvm-dev] Question about : lprofValueProfNodes
>
>
>
> What Vedant said -- the profiler runtime provides buffer API for profile
> dumping.  Note that value profiling dumping is not yet supported for buffer
> API, but since you are using Front-end based instrumentation/profile-use,
> value profiler is not turned on by default anyway.
>
>
>
> David
>
>
>
> On Tue, Dec 19, 2017 at 5:29 PM, Vedant Kumar <vsk at apple.com>
wrote:
>
>
>
> On Dec 19, 2017, at 5:16 PM, Moshtaghi, Alireza <
> Alireza.Moshtaghi at netapp.com> wrote:
>
>
>
> Thank you
>
> So it does not seem to be relevant for what I’m trying to do.
>
> I’m doing something unconventional.
>
> The objective is to implement PGO and code coverage on a system that does
> not exit and does not have any file io, or any of  stdc libraries that
> libclang-profile is using. (more like a kernel)
>
> So what I’m trying to do is instead of calling __llvm_profile_write_file
> () from the application, read the sections __llvm_prf_data,
> __llvm_prf_names, __llvm_prf_cnts and __llvm_prf_vnds after the critical
> tasks are done and transfer them to outside of the system.
>
>
>
> You may already have worked this out, but for completeness I'll mention
> that this is typically done by:
>
> 1. Allocating a buffer big enough to contain all the data
> (see __llvm_profile_get_size_for_buffer_internal),
> 2. Filling out the buffer (see __llvm_profile_write_buffer_internal), and
>
> 3. Transferring the buffer to some host machine.
>
>
>
>
> Then dump these sections in a char * array in a c file and attribute them
> to be in the associated section.
>
>
>
> I'm not sure I follow -- why is a C file needed?
>
>
>
> Once you have the buffer from step 3 on the host machine, you've got a
> well-formed raw profile. You should be able to fwrite() it to e.g
> 'default.profraw' and test that it's a valid profile with
llvm-profdata.
>
>
>
>
>
>  Then compile that file with –u__llvm_profile_runtime to create an
> executable that calls __llvm_profile_write_file to dump my profraw data.
>
> Sofar, I’m able to do the mechanics but seems like something is going
> wrong because because fprofile-instr-use does not find profile data for the
> source files.
>
>
>
> What error are you getting exactly? Have you confirmed that your profile
> is valid?
>
>
>
> vedant
>
>
>
>
> Any suggestion?
>
> Thanks
> A
>
> From: <vsk at apple.com> on behalf of Vedant Kumar <vsk at
apple.com>
> Date: Tuesday, December 19, 2017 at 11:32 AM
> To: "Moshtaghi, Alireza" <Alireza.Moshtaghi at netapp.com>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>, Xinliang David Li <
> davidxl at google.com>
> Subject: Re: [llvm-dev] Question about : lprofValueProfNodes
>
> Hi,
>
>
> On Dec 19, 2017, at 10:26 AM, Moshtaghi, Alireza via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi
> This array is defined in compiler-rt: InstrProfilingValue.c but I can’t
> find where it is used?
>
> It's used in allocateOneNode(). Incrementing the current vnode pointer
> gives a fresh node (possibly backed by the shared pool).
>
>
>
> And the comment on it does not say much about why we need it either.
>
> It's used to avoid calling malloc(), which David (CC'd) found to be
a
> performance improvement.
>
> best,
> vedant
>
>
> Can someone explain why we need this and where it is used?
>
> /* A shared static pool in addition to the vnodes statically
> * allocated by the compiler.  */
> COMPILER_RT_VISIBILITY ValueProfNode
>     lprofValueProfNodes[INSTR_PROF_VNODE_POOL_SIZE]
> COMPILER_RT_SECTION(COMPILER_RT_SEG INSTR_PROF_VNODES_SECT_NAME_STR);
>
> Thanks
> A
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171220/3b242802/attachment.html>

llvm dev - Dec 2017 - Question about : lprofValueProfNodes

[llvm-dev] Question about : lprofValueProfNodes

[llvm-dev] Question about : lprofValueProfNodes

[llvm-dev] Question about : lprofValueProfNodes