Tanoy Sinha via llvm-dev
2019-Jun-17 21:46 UTC
[llvm-dev] Running distributed thinLTO without thin archives.
I'm trying to run distributed ThinLTO without thin archives. When I do, I get an error in the optimizer when clang tries to open a nonexistent file: clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o main.o clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o lib/lib.o clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o src/lib.o llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o clang++ -flto=thin -o index -O3 -Wl,-plugin-opt,thinlto-index-only=thinlto.objects -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o -fthinlto-index=main.o.thinlto.bc Error loading imported file 'lib.a.llvm.2596.lib.cpp': No such file or directory In this case, gold has registered the modules within my archive with ThinLTO. The string "lib.a.llvm.2596.lib.cpp" is generated with the archive in question, plus an offset indicating where in the archive the particular object file is. Unfortunately, when the optimizer tries to include the proper modules, it's naively looking for a bitcode file with the name of the string provided, but there's obviously no "lib.a.llvm.2596.lib.cpp" for it to open. Has anyone else tried to get clang to understand distributed ThinLTO when using non thin archives? Is there some way to get clang to understand these out of the box? I'm actually a little confused about the ".cpp" in "lib.a.llvm.2596.lib.cpp". Seems like it should be a ".o"? It didn't seem like there was anything out of the box that supported this. I was looking at having clang actually read in the archive file and register the correct bitcode module. I wanted to run it by the list to get some second opinions before I started that. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190617/57badb70/attachment.html>
Teresa Johnson via llvm-dev
2019-Jun-18 14:37 UTC
[llvm-dev] Running distributed thinLTO without thin archives.
Hi Tanoy, You can't use distributed ThinLTO with archives (thin or not), at least not today. The reason is that we need to be able to identify specific bitcode object files to import from in the backends, and that logic does not know how to deal with objects within archives. We do distributed ThinLTO in our builds but don't use .a files, rather, we use --start-lib/--end-lib around the files that would be in the same archive when performing the thin link. I.e. if you change your thin link to be: clang++ -flto=thin -o index -O3 -Wl,-plugin-opt,thinlto-index-only=thinlto.objects -Wl,-plugin-opt,thinlto-emit-imports-files main.o --start-lib lib/lib.o src/lib.o --end-lib things should work. Note you also need to do the ThinLTO backend compile for each of the archive constituents anyway, e.g. something like: clang++ -c -x ir lib/lib.o -O3 -flto=thin -o lib/lib-native.o -fthinlto-index=lib/lib.o.thinlto.bc etc HTH, Teresa On Mon, Jun 17, 2019 at 2:46 PM Tanoy Sinha via llvm-dev < llvm-dev at lists.llvm.org> wrote:> I'm trying to run distributed ThinLTO without thin archives. > When I do, I get an error in the optimizer when clang tries to open a > nonexistent file: > > clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o main.o > clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o lib/lib.o > clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o src/lib.o > llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o > clang++ -flto=thin -o index -O3 > -Wl,-plugin-opt,thinlto-index-only=thinlto.objects > -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a > clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o > -fthinlto-index=main.o.thinlto.bc > Error loading imported file 'lib.a.llvm.2596.lib.cpp': No such file or > directory > > In this case, gold has registered the modules within my archive with > ThinLTO. > The string "lib.a.llvm.2596.lib.cpp" is generated with the archive in > question, plus an offset indicating where in the archive the particular > object file is. > Unfortunately, when the optimizer tries to include the proper modules, > it's naively looking for a bitcode file with the name of the string > provided, but there's obviously no "lib.a.llvm.2596.lib.cpp" for it to open. > > Has anyone else tried to get clang to understand distributed ThinLTO when > using non thin archives? > Is there some way to get clang to understand these out of the box? > > I'm actually a little confused about the ".cpp" in > "lib.a.llvm.2596.lib.cpp". > Seems like it should be a ".o"? > > It didn't seem like there was anything out of the box that supported this. > I was looking at having clang actually read in the archive file and > register the correct bitcode module. > I wanted to run it by the list to get some second opinions before I > started that. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-- Teresa Johnson | Software Engineer | tejohnson at google.com | -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190618/bd15988a/attachment.html>
Tanoy Sinha via llvm-dev
2019-Jun-18 20:40 UTC
[llvm-dev] Running distributed thinLTO without thin archives.
Thanks! Question about the final link step: Do I provide all the object files to the link step, i.e. something like: clang++ -o thinlto main-native.o lib/lib-native.o src/lib-native.o Do I need to provide --start-lib markers on that final link step as well? Tanoy On Tue, Jun 18, 2019 at 10:37 AM Teresa Johnson <tejohnson at google.com> wrote:> Hi Tanoy, > > You can't use distributed ThinLTO with archives (thin or not), at least > not today. The reason is that we need to be able to identify specific > bitcode object files to import from in the backends, and that logic does > not know how to deal with objects within archives. We do distributed > ThinLTO in our builds but don't use .a files, rather, we use > --start-lib/--end-lib around the files that would be in the same archive > when performing the thin link. I.e. if you change your thin link to be: > > clang++ -flto=thin -o index -O3 > -Wl,-plugin-opt,thinlto-index-only=thinlto.objects > -Wl,-plugin-opt,thinlto-emit-imports-files main.o --start-lib lib/lib.o > src/lib.o --end-lib > > things should work. > > Note you also need to do the ThinLTO backend compile for each of the > archive constituents anyway, e.g. something like: > clang++ -c -x ir lib/lib.o -O3 -flto=thin -o lib/lib-native.o > -fthinlto-index=lib/lib.o.thinlto.bc > etc > > HTH, > Teresa > > On Mon, Jun 17, 2019 at 2:46 PM Tanoy Sinha via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> I'm trying to run distributed ThinLTO without thin archives. >> When I do, I get an error in the optimizer when clang tries to open a >> nonexistent file: >> >> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o main.o >> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o lib/lib.o >> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o src/lib.o >> llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o >> clang++ -flto=thin -o index -O3 >> -Wl,-plugin-opt,thinlto-index-only=thinlto.objects >> -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a >> clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o >> -fthinlto-index=main.o.thinlto.bc >> Error loading imported file 'lib.a.llvm.2596.lib.cpp': No such file or >> directory >> >> In this case, gold has registered the modules within my archive with >> ThinLTO. >> The string "lib.a.llvm.2596.lib.cpp" is generated with the archive in >> question, plus an offset indicating where in the archive the particular >> object file is. >> Unfortunately, when the optimizer tries to include the proper modules, >> it's naively looking for a bitcode file with the name of the string >> provided, but there's obviously no "lib.a.llvm.2596.lib.cpp" for it to open. >> >> Has anyone else tried to get clang to understand distributed ThinLTO when >> using non thin archives? >> Is there some way to get clang to understand these out of the box? >> >> I'm actually a little confused about the ".cpp" in >> "lib.a.llvm.2596.lib.cpp". >> Seems like it should be a ".o"? >> >> It didn't seem like there was anything out of the box that supported >> this. >> I was looking at having clang actually read in the archive file and >> register the correct bitcode module. >> I wanted to run it by the list to get some second opinions before I >> started that. >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > > > -- > Teresa Johnson | Software Engineer | tejohnson at google.com | >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190618/82113c6c/attachment.html>