llvm dev - Feb 2014 - [LLVMdev] multithreaded use of llvm::sys::RemoveFileOnSignal

Hi,
I am using clang::ToolInvocation class to compile some code in-memory:

   clang::tooling::ToolInvocation ti
      (
       compilerArgs,
       new clang::EmitBCAction(),
       new clang::FileManager(clang::FileSystemOptions())
      );
   //filename is the name of the source file, e.g. "Somefile.cpp"
   //sourcecode contains the source of the file
   ti.mapVirtualFile( filename, sourcecode );
   bool ret = ti.run();

In order to speed up compilation of several sources, I call the above code
concurrently in separate threads, with each thread compiling its own source
code. However, this does not work because clang::CompilerInstance calls
llvm::Sys::RemoveFileOnSignal which in turn calls RegisterHandlers() which
is not re-entrant.

Is there a way to make RegisterHandlers() re-entrant? I do call
llvm::start_multi_threaded() before any of this, hoping that it will make
llvm routines thread-safe, but to no avail.

Also, is there any other way of doing parallel compilation?

thx
Vikas.
========-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140226/61077bd4/attachment.html>

On Thu, Feb 27, 2014 at 12:50 AM, Vikas Bhargava
<vikasbhargava at gmail.com> wrote:
> Hi,
> I am using clang::ToolInvocation class to compile some code in-memory:
>
>    clang::tooling::ToolInvocation ti
>       (
>        compilerArgs,
>        new clang::EmitBCAction(),
>        new clang::FileManager(clang::FileSystemOptions())
>       );
>    //filename is the name of the source file, e.g. "Somefile.cpp"
>    //sourcecode contains the source of the file
>    ti.mapVirtualFile( filename, sourcecode );
>    bool ret = ti.run();
>
> In order to speed up compilation of several sources, I call the above code
> concurrently in separate threads, with each thread compiling its own source
> code. However, this does not work because clang::CompilerInstance calls
> llvm::Sys::RemoveFileOnSignal which in turn calls RegisterHandlers() which
> is not re-entrant.
>
> Is there a way to make RegisterHandlers() re-entrant? I do call
> llvm::start_multi_threaded() before any of this, hoping that it will make
> llvm routines thread-safe, but to no avail.
Seems like it is a bug in either RemoveFileOnSignal().  For some
reason, the first one releases the mutex before calling the
RegisterHandlers().

bool llvm::sys::RemoveFileOnSignal(StringRef Filename,
                                   std::string* ErrMsg) {
  SignalsMutex.acquire();
...
  SignalsMutex.release();

  RegisterHandlers();
  return false;
}

CC'ing Argyrios because this is important for libclang.

Argyrios: what do you think -- should we call RegisterHandlers() under a mutex?

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/

On Feb 27, 2014, at 3:23 AM, Dmitri Gribenko <gribozavr at gmail.com>
wrote:
> On Thu, Feb 27, 2014 at 12:50 AM, Vikas Bhargava
> <vikasbhargava at gmail.com> wrote:
>> Hi,
>> I am using clang::ToolInvocation class to compile some code in-memory:
>> 
>>   clang::tooling::ToolInvocation ti
>>      (
>>       compilerArgs,
>>       new clang::EmitBCAction(),
>>       new clang::FileManager(clang::FileSystemOptions())
>>      );
>>   //filename is the name of the source file, e.g.
"Somefile.cpp"
>>   //sourcecode contains the source of the file
>>   ti.mapVirtualFile( filename, sourcecode );
>>   bool ret = ti.run();
>> 
>> In order to speed up compilation of several sources, I call the above
code
>> concurrently in separate threads, with each thread compiling its own
source
>> code. However, this does not work because clang::CompilerInstance calls
>> llvm::Sys::RemoveFileOnSignal which in turn calls RegisterHandlers()
which
>> is not re-entrant.
>> 
>> Is there a way to make RegisterHandlers() re-entrant? I do call
>> llvm::start_multi_threaded() before any of this, hoping that it will
make
>> llvm routines thread-safe, but to no avail.
> 
> Seems like it is a bug in either RemoveFileOnSignal().  For some
> reason, the first one releases the mutex before calling the
> RegisterHandlers().
> 
> bool llvm::sys::RemoveFileOnSignal(StringRef Filename,
>                                   std::string* ErrMsg) {
>  SignalsMutex.acquire();
> ...
>  SignalsMutex.release();
> 
>  RegisterHandlers();
>  return false;
> }
> 
> CC'ing Argyrios because this is important for libclang.
> 
> Argyrios: what do you think -- should we call RegisterHandlers() under a
mutex?
That, or get RegisterHandlers to use the mutex.
Also access to CallBacksToRun is not thread-safe (the whole file probably needs
reviewing..)

But it's not so clear to me what is the exact symptom, is there a crash
occurring ? Vika, could you explain ?
> 
> Dmitri
> 
> -- 
> main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
> (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140227/8d87f041/attachment.html>