llvm dev - Jul 2021 - Do I have to lock ThreadSafeContext and ThreadSafeModule in single threaded env?

Hi all,

I’m new to JIT and just started experiencing on some example code with ORCv2,
but could not get my head around with the thread safety.

Scenario 1, Application has multiple threads and each thread spawns its own JIT
instance, within each JIT instance, there are multiple ThreadSafeModules share
one ThreadSafeContext

Do I have to lock TSM/ TSC?

Scenario 2,  If JIT itself has multiple compilation threads, like via setting
NumCompileThreads in LLJIT classes, and each ThreadSafeModule has its own
ThreadSafeContext, so context is not shared

In this case, do I have to lock? If so, what’s the granularity of each lock
should be, anything before IR generation in one locking?


I think in scenario 1, locking is not needed because of no concurrent access.
And I believe locking is a mandatory in scenario 2, but I don’t know why...

Hi Mo,

I’m new to JIT and just started experiencing on some example code
with
> ORCv2, but could not get my head around with the thread safety.

The short version is that ThreadSafeContext is a pair of an LLVMContext and
a mutex, and that mutex is locked whenever you call
ThreadSafeModule::withModuleDo on a module using that ThreadSafeContext.

That means that two Modules that share a Context can't be accessed
simultaneously (when wrapped in and accessed through ThreadSafeModule and
ThreadSafeContext).

In this system you rarely have to do anything explicitly. The idea is that
you access the module via withModuleDo and that takes care of the locking
for you.

Scenario 1, Application has multiple threads and each thread spawns its
own
> JIT instance, within each JIT instance, there are multiple
> ThreadSafeModules share one ThreadSafeContext
> Do I have to lock TSM/ TSC?

You should always use withModuleDo to access the module, regardless of your
setup. This prevents you from writing non-thread-safe code that might be
difficult to debug if you add threading in later. What's important here is
knowing the locking implications of the access pattern, since they'll
affect how much parallelism you can get out of your JIT.

In your Scenario 1 each thread accesses a single LLVMContext. There are no
LLVMContexts shared between threads, so no chance for lock contention.
That's good.

 Scenario 2,  If JIT itself has multiple compilation threads, like
via
> setting NumCompileThreads in LLJIT classes, and each ThreadSafeModule has
> its own ThreadSafeContext, so context is not shared

In this scenario each JIT has multiple threads, but each thread is
operating on a single module with its own LLVMContext. Again there is no
chance for lock contention, so no barrier to parallelism.

The only scenario where locking becomes relevant is when you have multiple
threads and a single ThreadSafeContext shared between two or more
ThreadSafeModules. Consider a JIT with 10 threads and 10 llvm::Modules all
sharing a single ThreadSafeContext. If you issue a lookup that triggers
materialization of all 10 modules simultaneously then they'll be sent to
each of your 10 worker threads to be compiled, but as soon as the 1st
worker thread takes the context lock the other 9 threads will be blocked
from progressing. Eventually the 1st worker thread will relinquish the lock
and the next worker thread will take it, preventing the other 8 threads
from progressing, and so on. You have 10 threads, but only one thread at a
time can actually do any work -- the single LLVMContext has serialized
compilation.

My recommendation is to always start with one context per module, and only
share contexts if you see memory consumption from the contexts becoming an
issue.

-- Lang.

On Thu, Jul 8, 2021 at 12:13 AM mo xiaoming via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi all,
>
> I’m new to JIT and just started experiencing on some example code with
> ORCv2, but could not get my head around with the thread safety.
>
> Scenario 1, Application has multiple threads and each thread spawns its
> own JIT instance, within each JIT instance, there are multiple
> ThreadSafeModules share one ThreadSafeContext
>
> Do I have to lock TSM/ TSC?
>
> Scenario 2,  If JIT itself has multiple compilation threads, like via
> setting NumCompileThreads in LLJIT classes, and each ThreadSafeModule has
> its own ThreadSafeContext, so context is not shared
>
> In this case, do I have to lock? If so, what’s the granularity of each
> lock should be, anything before IR generation in one locking?
>
>
> I think in scenario 1, locking is not needed because of no concurrent
> access. And I believe locking is a mandatory in scenario 2, but I don’t
> know why...
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210730/11fe81a2/attachment.html>

Thanks Lang, that’s the clearest answer I have heard about this question :)
> On Jul 30, 2021, at 14:22, Lang Hames <lhames at gmail.com> wrote:
> 
> Hi Mo,
> 
> I’m new to JIT and just started experiencing on some example code with
ORCv2, but could not get my head around with the thread safety.
> 
> The short version is that ThreadSafeContext is a pair of an LLVMContext and
a mutex, and that mutex is locked whenever you call
ThreadSafeModule::withModuleDo on a module using that ThreadSafeContext.
> 
> That means that two Modules that share a Context can't be accessed
simultaneously (when wrapped in and accessed through ThreadSafeModule and
ThreadSafeContext).
> 
> In this system you rarely have to do anything explicitly. The idea is that
you access the module via withModuleDo and that takes care of the locking for
you.
> 
> Scenario 1, Application has multiple threads and each thread spawns its own
JIT instance, within each JIT instance, there are multiple ThreadSafeModules
share one ThreadSafeContext
> Do I have to lock TSM/ TSC?
> 
> You should always use withModuleDo to access the module, regardless of your
setup. This prevents you from writing non-thread-safe code that might be
difficult to debug if you add threading in later. What's important here is
knowing the locking implications of the access pattern, since they'll affect
how much parallelism you can get out of your JIT.
> 
> In your Scenario 1 each thread accesses a single LLVMContext. There are no
LLVMContexts shared between threads, so no chance for lock contention.
That's good.
> 
>  Scenario 2,  If JIT itself has multiple compilation threads, like via
setting NumCompileThreads in LLJIT classes, and each ThreadSafeModule has its
own ThreadSafeContext, so context is not shared
> 
> In this scenario each JIT has multiple threads, but each thread is
operating on a single module with its own LLVMContext. Again there is no chance
for lock contention, so no barrier to parallelism.
> 
> The only scenario where locking becomes relevant is when you have multiple
threads and a single ThreadSafeContext shared between two or more
ThreadSafeModules. Consider a JIT with 10 threads and 10 llvm::Modules all
sharing a single ThreadSafeContext. If you issue a lookup that triggers
materialization of all 10 modules simultaneously then they'll be sent to
each of your 10 worker threads to be compiled, but as soon as the 1st worker
thread takes the context lock the other 9 threads will be blocked from
progressing. Eventually the 1st worker thread will relinquish the lock and the
next worker thread will take it, preventing the other 8 threads from
progressing, and so on. You have 10 threads, but only one thread at a time can
actually do any work -- the single LLVMContext has serialized compilation.
> 
> My recommendation is to always start with one context per module, and only
share contexts if you see memory consumption from the contexts becoming an
issue.
> 
> -- Lang.
> 
> On Thu, Jul 8, 2021 at 12:13 AM mo xiaoming via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi all,
> 
> I’m new to JIT and just started experiencing on some example code with
ORCv2, but could not get my head around with the thread safety.
> 
> Scenario 1, Application has multiple threads and each thread spawns its own
JIT instance, within each JIT instance, there are multiple ThreadSafeModules
share one ThreadSafeContext
> 
> Do I have to lock TSM/ TSC?
> 
> Scenario 2,  If JIT itself has multiple compilation threads, like via
setting NumCompileThreads in LLJIT classes, and each ThreadSafeModule has its
own ThreadSafeContext, so context is not shared
> 
> In this case, do I have to lock? If so, what’s the granularity of each lock
should be, anything before IR generation in one locking?
> 
> 
> I think in scenario 1, locking is not needed because of no concurrent
access. And I believe locking is a mandatory in scenario 2, but I don’t know
why...
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210803/990caacb/attachment.html>