llvm dev - Jan 2020 - ThinLTO caching & compression

Hi,

I was wondering if there were plans to add compression to the ThinLTO cache?

If no, what would be a good solution? Use llvm/Support/Compression.h? Integrate
any other compression algorithm? (it'd be good if it could work
out-of-the-box)
Support HTTP to take advantage of the compression?
Allow for plugins to handle the cache streams? (that would be great, we
wouldn't need any post-processing steps to store the files in our object
store)

Also side-question: I was wondering if cache files can be shared across a team?
(given relative paths in the compilation cmd-lines)

Thanks!
Alex.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200104/fd5c6e31/attachment.html>

Should be shareable - don't think anyone's working on compression, could
probably be implemented separately in your distributed storage layer?

On Sat, Jan 4, 2020 at 9:44 AM Alexandre Ganea via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
>
>
> I was wondering if there were plans to add compression to the ThinLTO
> cache?
>
>
>
> If no, what would be a good solution? Use llvm/Support/Compression.h?
> Integrate any other compression algorithm? (it’d be good if it could work
> out-of-the-box)
>
> Support HTTP to take advantage of the compression?
>
> Allow for plugins to handle the cache streams? (that would be great, we
> wouldn’t need any post-processing steps to store the files in our object
> store)
>
>
>
> Also side-question: I was wondering if cache files can be shared across a
> team? (given relative paths in the compilation cmd-lines)
>
>
>
> Thanks!
>
> Alex.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200104/cc2fda0a/attachment.html>

+pcc who implemented the caching for questions on compression. I focus more
on ThinLTO in distributed build mode, which doesn't utilize the built in
caching.
But yes, the cache files should be shareable.

On Sat, Jan 4, 2020 at 10:25 AM David Blaikie <dblaikie at gmail.com>
wrote:
> Should be shareable - don't think anyone's working on compression,
could
> probably be implemented separately in your distributed storage layer?
>
> On Sat, Jan 4, 2020 at 9:44 AM Alexandre Ganea via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>>
>>
>> I was wondering if there were plans to add compression to the ThinLTO
>> cache?
>>
>>
>>
>> If no, what would be a good solution? Use llvm/Support/Compression.h?
>> Integrate any other compression algorithm? (it’d be good if it could
work
>> out-of-the-box)
>>
>> Support HTTP to take advantage of the compression?
>>
>> Allow for plugins to handle the cache streams? (that would be great, we
>> wouldn’t need any post-processing steps to store the files in our
object
>> store)
>>
>>
>>
>> Also side-question: I was wondering if cache files can be shared across
a
>> team? (given relative paths in the compilation cmd-lines)
>>
>>
>>
>> Thanks!
>>
>> Alex.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200106/72885a47/attachment.html>