Alexey via llvm-dev
2020-Sep-14 11:17 UTC
[llvm-dev] [Proposal][Debuginfo] dsymutil-like tool for ELF.
Debuginfo folks, What is your opinion on this proposal? Do we need to work on better DWARFLinker library for now? or Can we start to integrate llvm-dwarfutil as a series of small patches? If it is OK to start integrating of llvm-dwarfutil, Is it OK to move llvm-objcopy implementation into separate library ObjCopy ? Thank you, Alexey. On 01.09.2020 20:18, James Y Knight wrote:> > > On Mon, Aug 31, 2020 at 11:06 AM Alexey <avl.lapshin at gmail.com > <mailto:avl.lapshin at gmail.com>> wrote: > > Hi James, > > Thank you for the comments. > > >I think we're not terribly far from that ideal, now, for ELF. > Maybe only these three things need to be done? -- > > 1. Teach lld how to emit a separated debuginfo output file > directly, without requiring an objcopy step. > > 2. Integrate DWARFLinker into lld. > > 3. Create a new tool which takes the separated debuginfo and > DWO/DWP files and uses DWARFLinker library > > to create a new (dwarf-linked) separated-debug file, that > doesn't depend on DWO/DWP files. > > The three goals which you`ve described are our far goals. > Indeed, the best solution would be to create valid optimized debug > info without additional > stages and additional modifications of resulting binaries. > > There was an attempt to use DWARFLinker from the lld - > https://reviews.llvm.org/D74169 > It did not receive enough support to be integrated yet. There are > fair reasons for that: > > 1. Execution time. The time required by DWARFLinker for processing > clang binary is 8x bigger > than the usual linking time. Linking clang binary with DWARFLinker > takes 72 sec, > linking with the only lld takes 9 sec. > > 2. "Removing obsolete debug info" could not be switched off. Thus, > lld could not use DWARFLinker for > other tasks(like generation of index tables - .gdb_index, > .debug_names) without significant performance > degradation. > > 3. DWARFLinker does not support split dwarf at the moment. > > All these reasons are not blockers. And I believe implementation > from D74169 might be integrated and > incrementally improved if there would be agreement on that. > > > Those do sound like absolutely critical issues for deploying this for > real -- whether as a separate tool or integrated with lld. But > possibly not critical enough to prevent adding this behind an > experimental flag, and working on the code incrementally in-tree. > However (without having looked at the code in question), I wonder if > the reported 8x regression in link-time is even going to be > salvageable just by incremental optimizations, or if it might require > a complete re-architecting of the DwarfLinker code. > > Using DWARFLinker from llvm-dwarfutil is another possibility to > use and improve it. > > When finally implemented - llvm-dwarfutil should solve the above > three issues and there > would probably be more reasons to include DWARFLinker into lld. > > > Is it the case that if the code is built to support the "read an > executable, output a new better executable" use-case, it will actually > be what's needed for the "output an optimized executable while linking > object files" use-case? I worry that those could have enough different > requirements that you really need to be developing the > linker-integrated version from the very beginning in order to get a > good result, rather than trying to shoehorn it in as an afterthought. > > Even if we would have the best solution - it is still useful to > have a tool like llvm-dwarfutil > > for cases when it is necessary to process already created binaries. > > > Sure -- I just think that should be considered as a secondary > use-case, and not the primary goal. >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200914/e1ebb3b2/attachment.html>
James Henderson via llvm-dev
2020-Sep-14 12:02 UTC
[llvm-dev] [Proposal][Debuginfo] dsymutil-like tool for ELF.
I'd be happy with the body of llvm-objcopy being moved into a library regardless of the outcome of llvm-dwarfutil. The big advantage from that would be that we can write gtest unit tests for the library components rather than having to coerce an input to trigger the exact behaviour we want with lit. One of last year's GSOC projects, worked on by Alex Brachet-Mialot (CC'ed), was to create a more general-purpose object editing library that objcopy would use. See also https://bugs.llvm.org/show_bug.cgi?id=41044. James On Mon, 14 Sep 2020 at 12:17, Alexey via llvm-dev <llvm-dev at lists.llvm.org> wrote:> Debuginfo folks, > > What is your opinion on this proposal? > > Do we need to work on better DWARFLinker library for now? or Can we start > to integrate llvm-dwarfutil as a series of small patches? > > If it is OK to start integrating of llvm-dwarfutil, Is it OK to move > llvm-objcopy implementation into separate library ObjCopy ? > > Thank you, Alexey. > On 01.09.2020 20:18, James Y Knight wrote: > > > > On Mon, Aug 31, 2020 at 11:06 AM Alexey <avl.lapshin at gmail.com> wrote: > >> Hi James, >> >> Thank you for the comments. >> >> >I think we're not terribly far from that ideal, now, for ELF. Maybe only >> these three things need to be done? -- >> > 1. Teach lld how to emit a separated debuginfo output file directly, >> without requiring an objcopy step. >> > 2. Integrate DWARFLinker into lld. >> > 3. Create a new tool which takes the separated debuginfo and DWO/DWP >> files and uses DWARFLinker library >> > to create a new (dwarf-linked) separated-debug file, that doesn't >> depend on DWO/DWP files. >> >> The three goals which you`ve described are our far goals. >> Indeed, the best solution would be to create valid optimized debug info >> without additional >> stages and additional modifications of resulting binaries. >> >> There was an attempt to use DWARFLinker from the lld - >> https://reviews.llvm.org/D74169 >> It did not receive enough support to be integrated yet. There are fair >> reasons for that: >> >> 1. Execution time. The time required by DWARFLinker for processing clang >> binary is 8x bigger >> than the usual linking time. Linking clang binary with DWARFLinker takes >> 72 sec, >> linking with the only lld takes 9 sec. >> >> 2. "Removing obsolete debug info" could not be switched off. Thus, lld >> could not use DWARFLinker for >> other tasks(like generation of index tables - .gdb_index, .debug_names) >> without significant performance >> degradation. >> >> 3. DWARFLinker does not support split dwarf at the moment. >> >> All these reasons are not blockers. And I believe implementation from >> D74169 might be integrated and >> incrementally improved if there would be agreement on that. >> > > Those do sound like absolutely critical issues for deploying this for real > -- whether as a separate tool or integrated with lld. But possibly not > critical enough to prevent adding this behind an experimental flag, and > working on the code incrementally in-tree. However (without having looked > at the code in question), I wonder if the reported 8x regression in > link-time is even going to be salvageable just by incremental > optimizations, or if it might require a complete re-architecting of the > DwarfLinker code. > > Using DWARFLinker from llvm-dwarfutil is another possibility to use and >> improve it. >> > When finally implemented - llvm-dwarfutil should solve the above three >> issues and there >> would probably be more reasons to include DWARFLinker into lld. >> > > Is it the case that if the code is built to support the "read an > executable, output a new better executable" use-case, it will actually be > what's needed for the "output an optimized executable while linking object > files" use-case? I worry that those could have enough different > requirements that you really need to be developing the linker-integrated > version from the very beginning in order to get a good result, rather than > trying to shoehorn it in as an afterthought. > > Even if we would have the best solution - it is still useful to have a >> tool like llvm-dwarfutil >> > for cases when it is necessary to process already created binaries. >> > > Sure -- I just think that should be considered as a secondary use-case, > and not the primary goal. > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200914/1da93c7a/attachment.html>
Alexey via llvm-dev
2020-Sep-15 10:24 UTC
[llvm-dev] [Proposal][Debuginfo] dsymutil-like tool for ELF.
On 14.09.2020 15:02, James Henderson wrote:> I'd be happy with the body of llvm-objcopy being moved into a library > regardless of the outcome of llvm-dwarfutil. The big advantage from > that would be that we can write gtest unit tests for the library > components rather than having to coerce an input to trigger the exact > behaviour we want with lit. One of last year's GSOC projects, worked > on by Alex Brachet-Mialot (CC'ed), was to create a more > general-purpose object editing library that objcopy would use. See > also https://bugs.llvm.org/show_bug.cgi?id=41044.Ok. I would prepare the patch for it then(so that it could be discussed whether it fits the project). Alexey.> > James > > On Mon, 14 Sep 2020 at 12:17, Alexey via llvm-dev > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > > Debuginfo folks, > > What is your opinion on this proposal? > > Do we need to work on better DWARFLinker library for now? or Can > we start to integrate llvm-dwarfutil as a series of small patches? > > If it is OK to start integrating of llvm-dwarfutil, Is it OK to > move llvm-objcopy implementation into separate library ObjCopy ? > > Thank you, Alexey. > > On 01.09.2020 20:18, James Y Knight wrote: >> >> >> On Mon, Aug 31, 2020 at 11:06 AM Alexey <avl.lapshin at gmail.com >> <mailto:avl.lapshin at gmail.com>> wrote: >> >> Hi James, >> >> Thank you for the comments. >> >> >I think we're not terribly far from that ideal, now, for >> ELF. Maybe only these three things need to be done? -- >> > 1. Teach lld how to emit a separated debuginfo output file >> directly, without requiring an objcopy step. >> > 2. Integrate DWARFLinker into lld. >> > 3. Create a new tool which takes the separated debuginfo >> and DWO/DWP files and uses DWARFLinker library >> > to create a new (dwarf-linked) separated-debug file, that >> doesn't depend on DWO/DWP files. >> >> The three goals which you`ve described are our far goals. >> Indeed, the best solution would be to create valid optimized >> debug info without additional >> stages and additional modifications of resulting binaries. >> >> There was an attempt to use DWARFLinker from the lld - >> https://reviews.llvm.org/D74169 >> It did not receive enough support to be integrated yet. There >> are fair reasons for that: >> >> 1. Execution time. The time required by DWARFLinker for >> processing clang binary is 8x bigger >> than the usual linking time. Linking clang binary with >> DWARFLinker takes 72 sec, >> linking with the only lld takes 9 sec. >> >> 2. "Removing obsolete debug info" could not be switched off. >> Thus, lld could not use DWARFLinker for >> other tasks(like generation of index tables - .gdb_index, >> .debug_names) without significant performance >> degradation. >> >> 3. DWARFLinker does not support split dwarf at the moment. >> >> All these reasons are not blockers. And I believe >> implementation from D74169 might be integrated and >> incrementally improved if there would be agreement on that. >> >> >> Those do sound like absolutely critical issues for deploying this >> for real -- whether as a separate tool or integrated with lld. >> But possibly not critical enough to prevent adding this behind an >> experimental flag, and working on the code incrementally in-tree. >> However (without having looked at the code in question), I wonder >> if the reported 8x regression in link-time is even going to be >> salvageable just by incremental optimizations, or if it might >> require a complete re-architecting of the DwarfLinker code. >> >> Using DWARFLinker from llvm-dwarfutil is another possibility >> to use and improve it. >> >> When finally implemented - llvm-dwarfutil should solve the >> above three issues and there >> would probably be more reasons to include DWARFLinker into lld. >> >> >> Is it the case that if the code is built to support the "read an >> executable, output a new better executable" use-case, it will >> actually be what's needed for the "output an optimized executable >> while linking object files" use-case? I worry that those could >> have enough different requirements that you really need to be >> developing the linker-integrated version from the very beginning >> in order to get a good result, rather than trying to shoehorn it >> in as an afterthought. >> >> Even if we would have the best solution - it is still useful >> to have a tool like llvm-dwarfutil >> >> for cases when it is necessary to process already created >> binaries. >> >> >> Sure -- I just think that should be considered as a secondary >> use-case, and not the primary goal. >> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200915/d4c1ede5/attachment.html>