llvm dev - May 2018 - Slow IR compilation/JIT, profiling points to LLVM?

I'm having issues of my compiler, and JIT execution, of LLVM IR being
rather slow. It's accounting for the vast majority of my full
compilation time.  I'm trying to figure out why this is happening, since
it's becoming an impediment.  (Note: by slow I mean about 3s of time for
only about 2K of my front-end code, 65K lines of LLVM-IR)


Using valgrind I see some functions which seem out of place and
accounting for the vast majority of the time.

    5.72%; 635,008 Calls; llvm::PMTopLevelManager::findAnalysisPass(void
const*) <cycle 4>

    4.54%; 3,722,489 Calls; llvm::PMDataManager::findAnalysisPass(void
const*, bool)'2

    4.11%; 4,604,499 Calls; bool
llvm::DenseMapBase<>::LookupBucketFor<>(void const* const&,
llvm::detail::DenseMapPair<> const*&) const

Also of interest, given the high call count:

    1.32%; 6,915,882 Calls; llvm::FoldingSetNodeID::AddInteger(unsigned
int) <cycle 4>

The call counts seem quite high given the size of the code. Also, the
`findAnalysisPass` function just seems off, but I don't understand
LLVM's architecture to be sure.


Could this be pointing to something I'm doing wrong in my LLVM setup, or
is it just slow?


I'm reasonably certain I'm compiling LLVM in optimized mode, but for
reference, this is my build line:

    cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_EH=ON
-DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DLLVM_ENABLE_CXX1Y=ON
-DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_ENABLE_FFI=ON
-DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_OPTIMIZED_TABLEGEN=ON
-DCMAKE_INSTALL_PREFIX="/opt/llvm/install"


The overall time split, from valgrind, between shared libraries in my
code is:

    80.48%, libLLVM-6.0.so
    8.83% libc-2.23.so
    2.34% libleaf_lang.so (my front-end)

-- 
edA-qa mort-ora-y  
	http://mortoray.com/

Creator of the Leaf language
	http://leaflang.org/

Streaming algorithms, AI, and design on Twitch
	https://www.twitch.tv/mortoray

Twitter
	edaqa

Hi,

Could you share how you compile IR and which version of JIT you use (Orc,
MCJIT)?
Could it be that you are using interpreter instead of actual JIT?

Cheers,
Alex.
> On 5. May 2018, at 08:04, edA-qa mort-ora-y via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I'm having issues of my compiler, and JIT execution, of LLVM IR being
> rather slow. It's accounting for the vast majority of my full
> compilation time.  I'm trying to figure out why this is happening,
since
> it's becoming an impediment.  (Note: by slow I mean about 3s of time
for
> only about 2K of my front-end code, 65K lines of LLVM-IR)
> 
> 
> Using valgrind I see some functions which seem out of place and
> accounting for the vast majority of the time.
> 
>     5.72%; 635,008 Calls; llvm::PMTopLevelManager::findAnalysisPass(void
> const*) <cycle 4>
> 
>     4.54%; 3,722,489 Calls; llvm::PMDataManager::findAnalysisPass(void
> const*, bool)'2
> 
>     4.11%; 4,604,499 Calls; bool
> llvm::DenseMapBase<>::LookupBucketFor<>(void const* const&,
> llvm::detail::DenseMapPair<> const*&) const
> 
> Also of interest, given the high call count:
> 
>     1.32%; 6,915,882 Calls; llvm::FoldingSetNodeID::AddInteger(unsigned
> int) <cycle 4>
> 
> The call counts seem quite high given the size of the code. Also, the
> `findAnalysisPass` function just seems off, but I don't understand
> LLVM's architecture to be sure.
> 
> 
> Could this be pointing to something I'm doing wrong in my LLVM setup,
or
> is it just slow?
> 
> 
> I'm reasonably certain I'm compiling LLVM in optimized mode, but
for
> reference, this is my build line:
> 
>     cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_EH=ON
> -DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DLLVM_ENABLE_CXX1Y=ON
> -DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_ENABLE_FFI=ON
> -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_OPTIMIZED_TABLEGEN=ON
> -DCMAKE_INSTALL_PREFIX="/opt/llvm/install"
> 
> 
> The overall time split, from valgrind, between shared libraries in my
> code is:
> 
>     80.48%, libLLVM-6.0.so
>     8.83% libc-2.23.so
>     2.34% libleaf_lang.so (my front-end)
> 
> --
> edA-qa mort-ora-y
> 	http://mortoray.com/
> 
> Creator of the Leaf language
> 	http://leaflang.org/
> 
> Streaming algorithms, AI, and design on Twitch
> 	https://www.twitch.tv/mortoray
> 
> Twitter
> 	edaqa
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: Message signed with OpenPGP
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180505/b4c08aeb/attachment.sig>

I'll post the fragments of code below related to setup/exec.  I
double-checked the object file generation and that appears to be a lot
better now, around only 2% total time (time in ld is very high, but I
guess that's not LLVM's responsibility).  This has changed recently I
think, it used to be higher. Nonetheless, let's look at the JIT (or not
JIT) for now).

I use the execution engine:

    std::string errStr;
     llvm::EngineBuilder builder{ unique_ptr<llvm::Module>(module) };
    llvm::ExecutionEngine * ee = builder.
        setErrorStr( &errStr ).
        setEngineKind( llvm::EngineKind::JIT ).
        setTargetOptions( topts ).
        create();

    ee->finalizeObject();
    auto main = ee->FindFunctionNamed( "main" );
    STATE_CHECK( main, "missing-main" );
   
    std::vector<llvm::GenericValue> args(2);
    args[0].IntVal = llvm::APInt( platform::target->abi_int_size, 0 );
    args[1].PointerVal = nullptr;
    llvm::GenericValue gv = ee->runFunction( main, args );
    auto ret = int(gv.IntVal.getSExtValue());
    delete ee;
    return ret;



In case relevant, this is how I setup the module/context:

    llvm::InitializeNativeTarget();
    llvm_context.reset( new llvm::LLVMContext );
    module = new llvm::Module( "test", *llvm_context );
    module->setTargetTriple( platform::target->triple );
    del_module = true;
   
    std::string err_str;
    auto triple_str = llvm::Triple::normalize(platform::target->triple);
    llvm::Target const *  target = llvm::TargetRegistry::lookupTarget(
triple_str, err_str );
    STATE_CHECK( target, err_str );
    auto targetMachine = target->createTargetMachine(triple_str,
"generic", "", llvm::TargetOptions{},
        llvm::Optional<llvm::Reloc::Model>());
   
    module->setDataLayout( targetMachine->createDataLayout() );



Versions:
LLVM 6.0.0
GCC 5.4.0
(K)Ubuntu 16.04

On 05/05/18 09:51, Alex Denisov wrote:
> Hi,
>
> Could you share how you compile IR and which version of JIT you use (Orc,
MCJIT)?
> Could it be that you are using interpreter instead of actual JIT?
>
> Cheers,
> Alex.
>
>> On 5. May 2018, at 08:04, edA-qa mort-ora-y via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>>
>> I'm having issues of my compiler, and JIT execution, of LLVM IR
being
>> rather slow. It's accounting for the vast majority of my full
>> compilation time.  I'm trying to figure out why this is happening,
since
>> it's becoming an impediment.  (Note: by slow I mean about 3s of
time for
>> only about 2K of my front-end code, 65K lines of LLVM-IR)
>>
>>
>> Using valgrind I see some functions which seem out of place and
>> accounting for the vast majority of the time.
>>
>>     5.72%; 635,008 Calls;
llvm::PMTopLevelManager::findAnalysisPass(void
>> const*) <cycle 4>
>>
>>     4.54%; 3,722,489 Calls; llvm::PMDataManager::findAnalysisPass(void
>> const*, bool)'2
>>
>>     4.11%; 4,604,499 Calls; bool
>> llvm::DenseMapBase<>::LookupBucketFor<>(void const*
const&,
>> llvm::detail::DenseMapPair<> const*&) const
>>
>> Also of interest, given the high call count:
>>
>>     1.32%; 6,915,882 Calls; llvm::FoldingSetNodeID::AddInteger(unsigned
>> int) <cycle 4>
>>
>> The call counts seem quite high given the size of the code. Also, the
>> `findAnalysisPass` function just seems off, but I don't understand
>> LLVM's architecture to be sure.
>>
>>
>> Could this be pointing to something I'm doing wrong in my LLVM
setup, or
>> is it just slow?
>>
>>
>> I'm reasonably certain I'm compiling LLVM in optimized mode,
but for
>> reference, this is my build line:
>>
>>     cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_EH=ON
>> -DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DLLVM_ENABLE_CXX1Y=ON
>> -DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_ENABLE_FFI=ON
>> -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_OPTIMIZED_TABLEGEN=ON
>> -DCMAKE_INSTALL_PREFIX="/opt/llvm/install"
>>
>>
>> The overall time split, from valgrind, between shared libraries in my
>> code is:
>>
>>     80.48%, libLLVM-6.0.so
>>     8.83% libc-2.23.so
>>     2.34% libleaf_lang.so (my front-end)
>>
>> --
>> edA-qa mort-ora-y
>> 	http://mortoray.com/
>>
>> Creator of the Leaf language
>> 	http://leaflang.org/
>>
>> Streaming algorithms, AI, and design on Twitch
>> 	https://www.twitch.tv/mortoray
>>
>> Twitter
>> 	edaqa
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
edA-qa mort-ora-y  
	http://mortoray.com/

Creator of the Leaf language
	http://leaflang.org/

Streaming algorithms, AI, and design on Twitch
	https://www.twitch.tv/mortoray

Twitter
	edaqa
	


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180505/7815d9ee/attachment.sig>

Hi,

On 2018-05-05 08:04:54 +0200, edA-qa mort-ora-y via llvm-dev
wrote:
> I'm having issues of my compiler, and JIT execution, of LLVM IR being
> rather slow. It's accounting for the vast majority of my full
> compilation time.  I'm trying to figure out why this is happening,
since
> it's becoming an impediment.  (Note: by slow I mean about 3s of time
for
> only about 2K of my front-end code, 65K lines of LLVM-IR)
> 
> 
> Using valgrind I see some functions which seem out of place and
> accounting for the vast majority of the time.
> 
>     5.72%; 635,008 Calls; llvm::PMTopLevelManager::findAnalysisPass(void
> const*) <cycle 4>
> 
>     4.54%; 3,722,489 Calls; llvm::PMDataManager::findAnalysisPass(void
> const*, bool)'2
> I'm reasonably certain I'm compiling LLVM in optimized mode, but
for
> reference, this is my build line:
> 
>     cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_EH=ON
> -DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DLLVM_ENABLE_CXX1Y=ON
> -DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_ENABLE_FFI=ON
> -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_OPTIMIZED_TABLEGEN=ON
> -DCMAKE_INSTALL_PREFIX="/opt/llvm/install"
You're building LLVM with assertions enabled (-DLLVM_ENABLE_ASSERTIONS=ON).
Some of those are fairly expensive...

- Andres

On 05/05/18 17:58, Andres Freund wrote:
> You're building LLVM with assertions enabled
> (-DLLVM_ENABLE_ASSERTIONS=ON).
> Some of those are fairly expensive...
>
Is there another way to get LLVM to check the correctness of my IR
without the assertions? That's what I'm assuming I need the flag for
(it's been a long time since I experimented with it)

If there is no way I guess I'll have to produce two versions of LLVM.  I
still commonly get type errors in my LLVM IR.

-- 
edA-qa mort-ora-y  
	http://mortoray.com/

Creator of the Leaf language
	http://leaflang.org/

Streaming algorithms, AI, and design on Twitch
	https://www.twitch.tv/mortoray

Twitter
	edaqa