Zachary Turner via llvm-dev
2018-Jan-25 17:52 UTC
[llvm-dev] [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
Actually I already have a theory that even though you are adding the section to the section table, you might not be adding a *symbol* for the section to the symbol table. So the existing symbols (which reference sections by index) will all be wrong because you've inserted a new section. Still though, obj2yaml would expose that. On Thu, Jan 25, 2018 at 9:50 AM Zachary Turner <zturner at google.com> wrote:> Yea as long as you compare clang-cl object file with automatically > generated .debug$H section against clang-cl object file without .debug$H > but added after the fact with llvm-objcopy, that should expose the problem > I think when you run obj2yaml on them. > > On Thu, Jan 25, 2018 at 9:49 AM Leonardo Santagada <santagada at gmail.com> > wrote: > >> I did reorder my sections, so that .debug$H is in the correct place, but >> now I get some errors on dubplicate symbols, I created a folder with >> examples: >> >> https://www.dropbox.com/sh/nmvzi44pi0boe76/AAA0f47O5PCJ9JiUc6wVuwBra?dl=0 >> >> t.obj is generated by vs 2015 and it links fine with lld-link.exe, but >> tout.obj gives this errors: >> >> lld-link.exe /DEBUG:GHASH tout.obj >> LLD-LINK.EXE: error: duplicate symbol: __local_stdio_printf_options in >> tout.obj and in LIBCMT.lib(default_local_stdio_options.obj) >> LLD-LINK.EXE: error: duplicate symbol: __local_stdio_printf_options in >> tout.obj and in libvcruntime.lib(undname.obj) >> >> I'm using PEView from http://wjradburn.com/software/ to look at the >> files and can't see anything wrong, except some valid differences in the >> offsets being used for the data (so pointer to data is different between >> them). >> >> I will look into yaml2obj now to see if I see anything else weird going >> on. >> >> >> On Thu, Jan 25, 2018 at 6:41 PM, Zachary Turner <zturner at google.com> >> wrote: >> >>> I'm pretty confident that cl is not putting anything strange in the >>> .debug$T sections. We've done a lot of testing and never seen anything >>> except CodeView type records in a .debug$T. My hunch is that your objcopy >>> patch is probably not doing the right thing in one or more of the section >>> headers, and this is confusing the linker. >>> >>> One idea might be to build a simple object file with clang-cl but >>> without the magic -mllvm -emit-codeview-ghash-section, then run your >>> llvm-objcopy on it. Then build the same object file passing -mllvm >>> -emit-codeview-ghash-section. Then run obj2yaml on both and diff the >>> results. They should be byte-for-byte identical. That should give you a >>> clue about if objcopy is doing something wrong. >>> >>> On Thu, Jan 25, 2018 at 2:21 AM Leonardo Santagada <santagada at gmail.com> >>> wrote: >>> >>>> Don't worry, I definetly want to perfect this to generate legal obj >>>> files, this is just to speed up testing. >>>> >>>> Now after patching all the obj files I get this errors when linking a >>>> small part of our code base (msvc 2017 15.5.3, lld and llvm-objcopy 7.0.0): >>>> lld-link.exe : error : relocation against symbol in discarded section: >>>> $LN8 >>>> lld-link.exe : error : relocation against symbol in discarded section: >>>> $LN43 >>>> lld-link.exe : error : relocation against symbol in discarded section: >>>> $LN37 >>>> >>>> I'm starting to guess that cl.exe might be putting some random comdat >>>> or other discardable symbols in the .debug$T and clang doesn't? I will try >>>> to debug this and see what more I can uncover. >>>> >>>> Linking works perfectly without my llvm-objcopy pass to add .debug$H? >>>> >>>> >>>> On Thu, Jan 25, 2018 at 1:53 AM, Zachary Turner <zturner at google.com> >>>> wrote: >>>> >>>>> It might not influence LLD, but at the same time we don't want to >>>>> upstream something that is producing technically illegal COFF files. Also >>>>> good to hear about the planned changes to your header files. Looking >>>>> forward to hearing about your experiences with clang-cl. >>>>> >>>>> On Wed, Jan 24, 2018 at 10:41 AM Leonardo Santagada < >>>>> santagada at gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I finally got my first .obj file patched with .debug$H to look >>>>>> somewhat right. I added the new section at the end of the file so I don't >>>>>> have to recalculate all sections (although now I probably could position it >>>>>> in the middle, knowing that each section is: SizeOfRawData + >>>>>> (last.Header.NumberOfRelocations * (4+4+2)) and the $H needs to come right >>>>>> after $T in the file). That although illegal based on the coff specs >>>>>> doesn't seem its going to influence lld. >>>>>> >>>>>> Also we talked and we are probably going to do something similar to a >>>>>> bunch of windows defines and a check for our own define (to guarantee that >>>>>> no one imported windows.h before win32.h) and drop the namespace and the >>>>>> conflicting names. >>>>>> >>>>>> >>>>>> On Tue, Jan 23, 2018 at 12:46 AM, Zachary Turner <zturner at google.com> >>>>>> wrote: >>>>>> >>>>>>> That's very possible that a 3rd party indirect header include is >>>>>>> involved. One idea might be like I suggested where you #define _WINDOWS_ >>>>>>> in win32.h and guarantee that it's always included first. Then those other >>>>>>> headers won't be able to #include <windows.h>. but it will probably >>>>>>> greatly expand the amount of stuff you have to add to win32.h, as you will >>>>>>> probably find some callers of functions that aren't yet in your win32.h >>>>>>> that you'd have to add. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Jan 22, 2018 at 3:28 PM Leonardo Santagada < >>>>>>> santagada at gmail.com> wrote: >>>>>>> >>>>>>>> Ok some information was lost on getting this example to you, I'm >>>>>>>> sorry for not being clear. >>>>>>>> >>>>>>>> We have a huge code base, let's say 90% of it doesn't include >>>>>>>> either header, 9% include win32.h and 1% includes both, I will try to >>>>>>>> discover why, but my guess is they include both a third party that includes >>>>>>>> windows.h and some of our libs that use win32.h. >>>>>>>> >>>>>>>> I will try to fully understand this tomorrow. >>>>>>>> >>>>>>>> I guess clang will not implement this ever so finishing the object >>>>>>>> copier is the best solution until all code is ported to clang. >>>>>>>> >>>>>>>> On 23 Jan 2018 00:02, "Zachary Turner" <zturner at google.com> wrote: >>>>>>>> >>>>>>>>> You said win32.h doesn't include windows.h, but main.cpp does. So >>>>>>>>> what's the disadvantage of just including it in win32.h anyway, since it's >>>>>>>>> already going to be in every translation unit? (Unless you didn't mean to >>>>>>>>> #include it in main.cpp) >>>>>>>>> >>>>>>>>> >>>>>>>>> I guess all I can do is warn you how bad of an idea this is. For >>>>>>>>> starters, I already found a bug in your code ;-) >>>>>>>>> >>>>>>>>> // stdint.h >>>>>>>>> typedef int int32_t; >>>>>>>>> >>>>>>>>> // winnt.h >>>>>>>>> typedef long LONG; >>>>>>>>> >>>>>>>>> // windef.h >>>>>>>>> typedef struct tagPOINT >>>>>>>>> { >>>>>>>>> LONG x; // long x >>>>>>>>> LONG y; // long y >>>>>>>>> } POINT, *PPOINT, NEAR *NPPOINT, FAR *LPPOINT; >>>>>>>>> >>>>>>>>> // win32.h >>>>>>>>> typedef int32_t LONG; >>>>>>>>> >>>>>>>>> struct POINT >>>>>>>>> { >>>>>>>>> LONG x; // int x >>>>>>>>> LONG y; // int y >>>>>>>>> }; >>>>>>>>> >>>>>>>>> So POINT is defined two different ways. In your minimal >>>>>>>>> interface, it's declared as 2 int32's, which are int. In the actual >>>>>>>>> Windows header files, it's declared as 2 longs. >>>>>>>>> >>>>>>>>> This might seem like a unimportant bug since int and long are the >>>>>>>>> same size, but int and long also mangle differently and affect overload >>>>>>>>> resolution, so you could have weird linker errors or call the wrong >>>>>>>>> function overload. >>>>>>>>> >>>>>>>>> Plus, it illustrates the fact that this struct *actually is* a >>>>>>>>> different type from the one in the windows header. >>>>>>>>> >>>>>>>>> You said at the end that you never intentionally import win32.h >>>>>>>>> and windows.h from the same translation unit. But then in this example you >>>>>>>>> did. I wonder if you could enforce that by doing this: >>>>>>>>> >>>>>>>>> // win32.h >>>>>>>>> #pragma once >>>>>>>>> >>>>>>>>> // Error if windows.h was included before us. >>>>>>>>> #if defined(_WINDOWS_) >>>>>>>>> #error "You're including win32.h after having already included >>>>>>>>> windows.h. Don't do this!" >>>>>>>>> #endif >>>>>>>>> >>>>>>>>> // And also make sure windows.h can't get included after us >>>>>>>>> #define _WINDOWS_ >>>>>>>>> >>>>>>>>> For the record, I tried the test case you linked when windows.h is >>>>>>>>> not included in main.cpp and it works (but still has the bug about int and >>>>>>>>> long). >>>>>>>>> >>>>>>>>> On Mon, Jan 22, 2018 at 2:23 PM Leonardo Santagada < >>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> It is super gross, but we copy parts of windows.h because having >>>>>>>>>> all of it if both gigantic and very very messy. So our win32.h has a couple >>>>>>>>>> thousands of lines and not 30k+ for windows.h and we try to have zero >>>>>>>>>> macros. Win32.h doesn't include windows.h so using ::BOOL wouldn't work. We >>>>>>>>>> don't want to create a namespace, we just want a cleaner interface to >>>>>>>>>> windows api. The namespace with c linkage is the way to trick cl into >>>>>>>>>> allowing us to in some files have both windows.h and Win32.h. I really >>>>>>>>>> don't see any way for us to have this Win32.h without this cl support, so >>>>>>>>>> maybe we should either put windows.h in a compiled header somewhere and not >>>>>>>>>> care that it is infecting everything or just have one place we can call to >>>>>>>>>> clean up after including windows.h (a massive set of undefs). >>>>>>>>>> >>>>>>>>>> So using can't work, because we never intentionally import >>>>>>>>>> windows.h and win32.h on the same translation unit. >>>>>>>>>> >>>>>>>>>> On Mon, Jan 22, 2018 at 7:08 PM, Zachary Turner < >>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>> >>>>>>>>>>> This is pretty gross, honestly :) >>>>>>>>>>> >>>>>>>>>>> Can't you just use using declarations? >>>>>>>>>>> >>>>>>>>>>> namespace Win32 { >>>>>>>>>>> extern "C" { >>>>>>>>>>> >>>>>>>>>>> using ::BOOL; >>>>>>>>>>> using ::LONG; >>>>>>>>>>> using ::POINT; >>>>>>>>>>> using ::LPPOINT; >>>>>>>>>>> >>>>>>>>>>> using ::GetCursorPos; >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> This works with clang-cl. >>>>>>>>>>> >>>>>>>>>>> On Mon, Jan 22, 2018 at 5:39 AM Leonardo Santagada < >>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Here it is a minimal example, we do this so we don't have to >>>>>>>>>>>> import the whole windows api everywhere. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://gist.github.com/santagada/7977e929d31c629c4bf18ebb987f6be3 >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Jan 21, 2018 at 2:31 AM, Zachary Turner < >>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Clang-cl maintains compatibility with msvc even in cases where >>>>>>>>>>>>> it’s non standards compliant (eg 2 phase name lookup), but we try to keep >>>>>>>>>>>>> these cases few and far between. >>>>>>>>>>>>> >>>>>>>>>>>>> To help me understand your case, do you mean you copy >>>>>>>>>>>>> windows.h and modify it? How does this lead to the same struct being >>>>>>>>>>>>> defined twice? If i were to write this: >>>>>>>>>>>>> >>>>>>>>>>>>> struct Foo {}; >>>>>>>>>>>>> struct Foo {}; >>>>>>>>>>>>> >>>>>>>>>>>>> Is this a small repro of the issue you’re talking about? >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Jan 20, 2018 at 3:44 PM Leonardo Santagada < >>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I can totally see something like incremental linking with a >>>>>>>>>>>>>> simple padding between obj and a mapping file (which can also help with >>>>>>>>>>>>>> edit and continue, something we also would love to have). >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have another developer doing the port to support clang-cl, >>>>>>>>>>>>>> but although most of our code also goes trough a version of clang, >>>>>>>>>>>>>> migrating the rest to clang-cl has been a fight. From what I heard the main >>>>>>>>>>>>>> problem is that we have a copy of parts of windows.h (so not to bring the >>>>>>>>>>>>>> awful parts of it like lower case macros) and that totally works on cl, but >>>>>>>>>>>>>> clang (at least 6.0) complains about two struct/vars with the same name, >>>>>>>>>>>>>> even though they are exactly the same. Making clang-cl as broken as cl.exe >>>>>>>>>>>>>> is not an option I suppose? I would love to turn on a flag >>>>>>>>>>>>>> --accept-that-cl-made-bad-decisions-and-live-with-it and have this at least >>>>>>>>>>>>>> until this is completely fixed in our code base. >>>>>>>>>>>>>> >>>>>>>>>>>>>> the biggest win with moving to cl would be a better more >>>>>>>>>>>>>> standards compliant compiler, no 1 minute compiles on heavily templated >>>>>>>>>>>>>> files and maybe the holy grail of ThinLTO. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner < >>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 10-15s will be hard without true incremental linking. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> At some point that's going to be the only way to get any >>>>>>>>>>>>>>> faster, but incremental linking is hard (putting it lightly), and since our >>>>>>>>>>>>>>> full links are already really fast we think we can get reasonably close to >>>>>>>>>>>>>>> link.exe incremental speeds with full links. But it's never enough and I >>>>>>>>>>>>>>> will always want it to be faster, so you may see incremental linking in the >>>>>>>>>>>>>>> future after we hit a performance wall with full link speed :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In any case, I'm definitely interested in seeing what kind >>>>>>>>>>>>>>> of numbers you get with /debug:ghash after you get this llvm-objcopy >>>>>>>>>>>>>>> feature implemented. So keep me updated :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As an aside, have you tried building with clang instead of >>>>>>>>>>>>>>> cl? If you build with clang you wouldn't even have to do this llvm-objcopy >>>>>>>>>>>>>>> work, because it would "just work". If you've tried but ran into issues >>>>>>>>>>>>>>> I'm interested in hearing about those too. On the other hand, it's also >>>>>>>>>>>>>>> reasonable to only switch one thing at a time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada < >>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> if we get to < 30s I think most users would prefer it to >>>>>>>>>>>>>>>> link.exe, just hopping there is still some more optimizations to get closer >>>>>>>>>>>>>>>> to ELF linking times (around 10-15s here). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner < >>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Generally speaking a good rule of thumb is that >>>>>>>>>>>>>>>>> /debug:ghash will be close to or faster than /debug:fastlink, but with none >>>>>>>>>>>>>>>>> of the penalties like slow debug time >>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner < >>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Chrome is actually one of my exact benchmark cases. When >>>>>>>>>>>>>>>>>> building blink_core.dll and browser_tests.exe, i get anywhere from a 20-40% >>>>>>>>>>>>>>>>>> reduction in link time. We have some other optimizations in the pipeline >>>>>>>>>>>>>>>>>> but not upstream yet. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> My best time so far (including other optimizations not >>>>>>>>>>>>>>>>>> yet upstream) is 28s on blink_core.dll, compared to 110s with /debug >>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo Santagada < >>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> You probably don't want to go down the same route that >>>>>>>>>>>>>>>>>>>> clang goes through to write the object file. If you think yaml2coff is >>>>>>>>>>>>>>>>>>>> convoluted, the way clang does it will just give you a headache. There are >>>>>>>>>>>>>>>>>>>> multiple abstractions involved to account for different object file formats >>>>>>>>>>>>>>>>>>>> (ELF, COFF, MachO) and output formats (Assembly, binary file). At least >>>>>>>>>>>>>>>>>>>> with yaml2coff >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I think your phrase got cut there, but yeah I just found >>>>>>>>>>>>>>>>>>> AsmPrinter.cpp and it is convoluted. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> It's true that yaml2coff is using the COFFParser >>>>>>>>>>>>>>>>>>>> structure, but if you look at the writeCOFF function >>>>>>>>>>>>>>>>>>>> in yaml2coff it's pretty bare-metal. The logic you need will be almost >>>>>>>>>>>>>>>>>>>> identical, except that instead of checking the COFFParser for the various >>>>>>>>>>>>>>>>>>>> fields, you'll check the existing COFFObjectFile, which should have similar >>>>>>>>>>>>>>>>>>>> fields. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The only thing you need to different is when writing >>>>>>>>>>>>>>>>>>>> the section table and section contents, to insert a new entry. Since >>>>>>>>>>>>>>>>>>>> you're injecting a section into the middle, you'll also probably need to >>>>>>>>>>>>>>>>>>>> push back the file pointer of all subsequent sections so that they don't >>>>>>>>>>>>>>>>>>>> overlap. (e.g. if the original sections are 1, 2, 3, 4, 5 and you insert >>>>>>>>>>>>>>>>>>>> between 2 and 3, then the original sections 3, 4, and 5 would need to have >>>>>>>>>>>>>>>>>>>> their FilePointerToRawData offset by the size of the new section). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have the PE/COFF spec open here and I'm happy that I >>>>>>>>>>>>>>>>>>> read a bit of it so I actually know what you are talking about... yeah it >>>>>>>>>>>>>>>>>>> doesn't seem too complicated. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> If you need to know what values to put for the other >>>>>>>>>>>>>>>>>>>> fields in a section header, run `dumpbin /headers foo.obj` on a >>>>>>>>>>>>>>>>>>>> clang-generated object file that has a .debug$H section already (e.g. run >>>>>>>>>>>>>>>>>>>> clang with -emit-codeview-ghash-section, and look at the properties of the >>>>>>>>>>>>>>>>>>>> .debug$H section and use the same values). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks I will do that and then also look at how the >>>>>>>>>>>>>>>>>>> CodeView part of the code does it if I can't understand some of it. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The only invariant that needs to be maintained is that >>>>>>>>>>>>>>>>>>>> Section[N]->FilePointerOfRawData == Section[N-1]->FilePointerOfRawData + >>>>>>>>>>>>>>>>>>>> Section[N-1]->SizeOfRawData >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Well, that and all the sections need to be on the final >>>>>>>>>>>>>>>>>>> file... But I'm hopeful. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Anyone has times on linking a big project like chrome >>>>>>>>>>>>>>>>>>> with this so that at least I know what kind of performance to expect? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> My numbers are something like: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1 pdb per obj file: link.exe takes ~15 minutes and 16GB >>>>>>>>>>>>>>>>>>> of ram, lld-link.exe takes 2:30 minutes and ~8GB of ram >>>>>>>>>>>>>>>>>>> around 10 pdbs per folder: link.exe takes 1 minute and >>>>>>>>>>>>>>>>>>> 2-3GB of ram, lld-link.exe takes 1:30 minutes and ~6GB of ram >>>>>>>>>>>>>>>>>>> faslink: link.exe takes 40 seconds, but then 20 seconds >>>>>>>>>>>>>>>>>>> of loading at the first break point in the debugger and we lost DIA support >>>>>>>>>>>>>>>>>>> for listing symbols. >>>>>>>>>>>>>>>>>>> incremental: link.exe takes 8 seconds, but it only >>>>>>>>>>>>>>>>>>> happens when very minor changes happen. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> We have an non negligible number of symbols used on some >>>>>>>>>>>>>>>>>>> runtime systems. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo Santagada < >>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks for the tips, I now have something that reads >>>>>>>>>>>>>>>>>>>>> the obj file, finds .debug$T sections and global hashes it (proof of >>>>>>>>>>>>>>>>>>>>> concept kind of code). What I can't find is: how does clang itself writes >>>>>>>>>>>>>>>>>>>>> the coff files with global hashes, as that might help me understand how to >>>>>>>>>>>>>>>>>>>>> create the .debug$H section, how to update the file section count and how >>>>>>>>>>>>>>>>>>>>> to properly write this back. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The code on yaml2coff is expecting to be working on >>>>>>>>>>>>>>>>>>>>> the yaml COFFParser struct and I'm having quite a bit of a headache turning >>>>>>>>>>>>>>>>>>>>> the COFFObjectFile into a COFFParser object or compatible... Tomorrow I >>>>>>>>>>>>>>>>>>>>> might try the very non efficient path of coff2yaml and then yaml2coff with >>>>>>>>>>>>>>>>>>>>> the hashes header... but it seems way too inefficient and convoluted. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM Leonardo Santagada < >>>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29 PM Leonardo Santagada >>>>>>>>>>>>>>>>>>>>>>>> <santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> No I didn't, I used cl.exe from the visual studio >>>>>>>>>>>>>>>>>>>>>>>>> toolchain. What I'm proposing is a tool for processing .obj files in COFF >>>>>>>>>>>>>>>>>>>>>>>>> format, reading them and generating the GHASH part. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> To make our build faster we use hundreds of unity >>>>>>>>>>>>>>>>>>>>>>>>> build files (.cpp's with a lot of other .cpp's in them aka munch files) but >>>>>>>>>>>>>>>>>>>>>>>>> still have a lot of single .cpp's as well (in total something like 3.4k >>>>>>>>>>>>>>>>>>>>>>>>> .obj files). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> ps: sorry for sending to the wrong list, I was >>>>>>>>>>>>>>>>>>>>>>>>> reading about llvm mailing lists and jumped when I saw what I thought was a >>>>>>>>>>>>>>>>>>>>>>>>> lld exclusive list. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> A tool like this would be useful, yes. We've >>>>>>>>>>>>>>>>>>>>>>>> talked about it internally as well and agreed it would be useful, we just >>>>>>>>>>>>>>>>>>>>>>>> haven't prioritized it. If you're interested in submitting a patch along >>>>>>>>>>>>>>>>>>>>>>>> those lines though, I think it would be a good addition. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I'm not sure what the best place for it would be. >>>>>>>>>>>>>>>>>>>>>>>> llvm-readobj and llvm-objdump seem like obvious choices, but they are >>>>>>>>>>>>>>>>>>>>>>>> intended to be read-only, so perhaps they wouldn't be a good fit. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil is kind of a hodgepodge of everything >>>>>>>>>>>>>>>>>>>>>>>> else related to PDBs and symbols, so I wouldn't be opposed to making a new >>>>>>>>>>>>>>>>>>>>>>>> subcommand there called "ghash" or something that could process an object >>>>>>>>>>>>>>>>>>>>>>>> file and output a new object file with a .debug$H section. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> A third option would be to make a new tool for it. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I don't htink it would be that hard to write. If >>>>>>>>>>>>>>>>>>>>>>>> you're interested in trying to make a patch for this, I can offer some >>>>>>>>>>>>>>>>>>>>>>>> guidance on where to look in the code. Otherwise it's something that we'll >>>>>>>>>>>>>>>>>>>>>>>> probably get to, I'm just not sure when. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I would love to write it and contribute it back, >>>>>>>>>>>>>>>>>>>>>>> please do tell, I did find some of the code of ghash in lld, but in fuzzy >>>>>>>>>>>>>>>>>>>>>>> on the llvm codeview part of it and never seen llvm-readobj/objdump or >>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil, but I'm not afraid to look :) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Luckily all of the important code is hidden behind >>>>>>>>>>>>>>>>>>>>>> library calls, and it should already just do the right thing, so I suspect >>>>>>>>>>>>>>>>>>>>>> you won't need to know much about CodeView to do this. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I think Peter has the right idea about putting this >>>>>>>>>>>>>>>>>>>>>> in llvm-objcopy. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> You can look at one of the existing CopyBinary >>>>>>>>>>>>>>>>>>>>>> functions there, which currently only work for ELF, but you can just make a >>>>>>>>>>>>>>>>>>>>>> new overload that accepts a COFFObjectFile. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I would probably start by iterating over each of the >>>>>>>>>>>>>>>>>>>>>> sections (getNumberOfSections / getSectionName) looking for .debug$T and >>>>>>>>>>>>>>>>>>>>>> .debug$H sections. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> If you find a .debug$H section then you can just skip >>>>>>>>>>>>>>>>>>>>>> that object file. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> If you find a .debug$T but not a .debug$H, then >>>>>>>>>>>>>>>>>>>>>> basically do the same thing that LLD does in PDBLinker::mergeDebugT >>>>>>>>>>>>>>>>>>>>>> (create a CVTypeArray, and pass it to GloballyHashedType::hashTypes. That >>>>>>>>>>>>>>>>>>>>>> will return an array of hash values. (the format of .debug$H is the >>>>>>>>>>>>>>>>>>>>>> header, followed by the hash values). Then when you're writing the list of >>>>>>>>>>>>>>>>>>>>>> sections, just add in the .debug$H section right after the .debug$T section. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Currently llvm-objcopy only writes ELF files, so it >>>>>>>>>>>>>>>>>>>>>> would need to be taught to write COFF files. We have code to do this in >>>>>>>>>>>>>>>>>>>>>> the yaml2obj utility (specifically, in yaml2coff.cpp in the function >>>>>>>>>>>>>>>>>>>>>> writeCOFF). There may be a way to move this code to somewhere else >>>>>>>>>>>>>>>>>>>>>> (llvm/Object/COFF.h?) so that it can be re-used by both yaml2coff and >>>>>>>>>>>>>>>>>>>>>> llvm-objcopy, but in the worst case scenario you could copy the code and >>>>>>>>>>>>>>>>>>>>>> re-write it to work with these new structures. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Lastly, you'll probably want to put all of this >>>>>>>>>>>>>>>>>>>>>> behind an option in llvm-objcopy such as -add-codeview-ghash-section >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Leonardo Santagada >>>>>>>>>> >>>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Leonardo Santagada >>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Leonardo Santagada >>>> >>> >> >> >> -- >> >> Leonardo Santagada >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180125/35fd604f/attachment-0001.html>
Leonardo Santagada via llvm-dev
2018-Jan-25 17:56 UTC
[llvm-dev] [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
YES, THANK YOU... I WAS THINKING THIS BUT COMPLETELY FORGOT. sorry for the caps... long day of working on this, and using vs 2017, which adds a new section type .chks64 that I couldn't find documentation anywhere was difficult. I highly recommend everyone to just not using vs 2017 until 15.8 or something, our internal bug list is gigantic. On Thu, Jan 25, 2018 at 6:52 PM, Zachary Turner <zturner at google.com> wrote:> Actually I already have a theory that even though you are adding the > section to the section table, you might not be adding a *symbol* for the > section to the symbol table. So the existing symbols (which reference > sections by index) will all be wrong because you've inserted a new > section. Still though, obj2yaml would expose that. > > On Thu, Jan 25, 2018 at 9:50 AM Zachary Turner <zturner at google.com> wrote: > >> Yea as long as you compare clang-cl object file with automatically >> generated .debug$H section against clang-cl object file without .debug$H >> but added after the fact with llvm-objcopy, that should expose the problem >> I think when you run obj2yaml on them. >> >> On Thu, Jan 25, 2018 at 9:49 AM Leonardo Santagada <santagada at gmail.com> >> wrote: >> >>> I did reorder my sections, so that .debug$H is in the correct place, but >>> now I get some errors on dubplicate symbols, I created a folder with >>> examples: >>> >>> https://www.dropbox.com/sh/nmvzi44pi0boe76/ >>> AAA0f47O5PCJ9JiUc6wVuwBra?dl=0 >>> >>> t.obj is generated by vs 2015 and it links fine with lld-link.exe, but >>> tout.obj gives this errors: >>> >>> lld-link.exe /DEBUG:GHASH tout.obj >>> LLD-LINK.EXE: error: duplicate symbol: __local_stdio_printf_options in >>> tout.obj and in LIBCMT.lib(default_local_stdio_options.obj) >>> LLD-LINK.EXE: error: duplicate symbol: __local_stdio_printf_options in >>> tout.obj and in libvcruntime.lib(undname.obj) >>> >>> I'm using PEView from http://wjradburn.com/software/ to look at the >>> files and can't see anything wrong, except some valid differences in the >>> offsets being used for the data (so pointer to data is different between >>> them). >>> >>> I will look into yaml2obj now to see if I see anything else weird going >>> on. >>> >>> >>> On Thu, Jan 25, 2018 at 6:41 PM, Zachary Turner <zturner at google.com> >>> wrote: >>> >>>> I'm pretty confident that cl is not putting anything strange in the >>>> .debug$T sections. We've done a lot of testing and never seen anything >>>> except CodeView type records in a .debug$T. My hunch is that your objcopy >>>> patch is probably not doing the right thing in one or more of the section >>>> headers, and this is confusing the linker. >>>> >>>> One idea might be to build a simple object file with clang-cl but >>>> without the magic -mllvm -emit-codeview-ghash-section, then run your >>>> llvm-objcopy on it. Then build the same object file passing -mllvm >>>> -emit-codeview-ghash-section. Then run obj2yaml on both and diff the >>>> results. They should be byte-for-byte identical. That should give you a >>>> clue about if objcopy is doing something wrong. >>>> >>>> On Thu, Jan 25, 2018 at 2:21 AM Leonardo Santagada <santagada at gmail.com> >>>> wrote: >>>> >>>>> Don't worry, I definetly want to perfect this to generate legal obj >>>>> files, this is just to speed up testing. >>>>> >>>>> Now after patching all the obj files I get this errors when linking a >>>>> small part of our code base (msvc 2017 15.5.3, lld and llvm-objcopy 7.0.0): >>>>> lld-link.exe : error : relocation against symbol in discarded section: >>>>> $LN8 >>>>> lld-link.exe : error : relocation against symbol in discarded section: >>>>> $LN43 >>>>> lld-link.exe : error : relocation against symbol in discarded section: >>>>> $LN37 >>>>> >>>>> I'm starting to guess that cl.exe might be putting some random comdat >>>>> or other discardable symbols in the .debug$T and clang doesn't? I will try >>>>> to debug this and see what more I can uncover. >>>>> >>>>> Linking works perfectly without my llvm-objcopy pass to add .debug$H? >>>>> >>>>> >>>>> On Thu, Jan 25, 2018 at 1:53 AM, Zachary Turner <zturner at google.com> >>>>> wrote: >>>>> >>>>>> It might not influence LLD, but at the same time we don't want to >>>>>> upstream something that is producing technically illegal COFF files. Also >>>>>> good to hear about the planned changes to your header files. Looking >>>>>> forward to hearing about your experiences with clang-cl. >>>>>> >>>>>> On Wed, Jan 24, 2018 at 10:41 AM Leonardo Santagada < >>>>>> santagada at gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I finally got my first .obj file patched with .debug$H to look >>>>>>> somewhat right. I added the new section at the end of the file so I don't >>>>>>> have to recalculate all sections (although now I probably could position it >>>>>>> in the middle, knowing that each section is: SizeOfRawData + (last.Header.NumberOfRelocations >>>>>>> * (4+4+2)) and the $H needs to come right after $T in the file). That >>>>>>> although illegal based on the coff specs doesn't seem its going to >>>>>>> influence lld. >>>>>>> >>>>>>> Also we talked and we are probably going to do something similar to >>>>>>> a bunch of windows defines and a check for our own define (to guarantee >>>>>>> that no one imported windows.h before win32.h) and drop the namespace and >>>>>>> the conflicting names. >>>>>>> >>>>>>> >>>>>>> On Tue, Jan 23, 2018 at 12:46 AM, Zachary Turner <zturner at google.com >>>>>>> > wrote: >>>>>>> >>>>>>>> That's very possible that a 3rd party indirect header include is >>>>>>>> involved. One idea might be like I suggested where you #define _WINDOWS_ >>>>>>>> in win32.h and guarantee that it's always included first. Then those other >>>>>>>> headers won't be able to #include <windows.h>. but it will probably >>>>>>>> greatly expand the amount of stuff you have to add to win32.h, as you will >>>>>>>> probably find some callers of functions that aren't yet in your win32.h >>>>>>>> that you'd have to add. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Jan 22, 2018 at 3:28 PM Leonardo Santagada < >>>>>>>> santagada at gmail.com> wrote: >>>>>>>> >>>>>>>>> Ok some information was lost on getting this example to you, I'm >>>>>>>>> sorry for not being clear. >>>>>>>>> >>>>>>>>> We have a huge code base, let's say 90% of it doesn't include >>>>>>>>> either header, 9% include win32.h and 1% includes both, I will try to >>>>>>>>> discover why, but my guess is they include both a third party that includes >>>>>>>>> windows.h and some of our libs that use win32.h. >>>>>>>>> >>>>>>>>> I will try to fully understand this tomorrow. >>>>>>>>> >>>>>>>>> I guess clang will not implement this ever so finishing the object >>>>>>>>> copier is the best solution until all code is ported to clang. >>>>>>>>> >>>>>>>>> On 23 Jan 2018 00:02, "Zachary Turner" <zturner at google.com> wrote: >>>>>>>>> >>>>>>>>>> You said win32.h doesn't include windows.h, but main.cpp does. >>>>>>>>>> So what's the disadvantage of just including it in win32.h anyway, since >>>>>>>>>> it's already going to be in every translation unit? (Unless you didn't >>>>>>>>>> mean to #include it in main.cpp) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I guess all I can do is warn you how bad of an idea this is. For >>>>>>>>>> starters, I already found a bug in your code ;-) >>>>>>>>>> >>>>>>>>>> // stdint.h >>>>>>>>>> typedef int int32_t; >>>>>>>>>> >>>>>>>>>> // winnt.h >>>>>>>>>> typedef long LONG; >>>>>>>>>> >>>>>>>>>> // windef.h >>>>>>>>>> typedef struct tagPOINT >>>>>>>>>> { >>>>>>>>>> LONG x; // long x >>>>>>>>>> LONG y; // long y >>>>>>>>>> } POINT, *PPOINT, NEAR *NPPOINT, FAR *LPPOINT; >>>>>>>>>> >>>>>>>>>> // win32.h >>>>>>>>>> typedef int32_t LONG; >>>>>>>>>> >>>>>>>>>> struct POINT >>>>>>>>>> { >>>>>>>>>> LONG x; // int x >>>>>>>>>> LONG y; // int y >>>>>>>>>> }; >>>>>>>>>> >>>>>>>>>> So POINT is defined two different ways. In your minimal >>>>>>>>>> interface, it's declared as 2 int32's, which are int. In the actual >>>>>>>>>> Windows header files, it's declared as 2 longs. >>>>>>>>>> >>>>>>>>>> This might seem like a unimportant bug since int and long are the >>>>>>>>>> same size, but int and long also mangle differently and affect overload >>>>>>>>>> resolution, so you could have weird linker errors or call the wrong >>>>>>>>>> function overload. >>>>>>>>>> >>>>>>>>>> Plus, it illustrates the fact that this struct *actually is* a >>>>>>>>>> different type from the one in the windows header. >>>>>>>>>> >>>>>>>>>> You said at the end that you never intentionally import win32.h >>>>>>>>>> and windows.h from the same translation unit. But then in this example you >>>>>>>>>> did. I wonder if you could enforce that by doing this: >>>>>>>>>> >>>>>>>>>> // win32.h >>>>>>>>>> #pragma once >>>>>>>>>> >>>>>>>>>> // Error if windows.h was included before us. >>>>>>>>>> #if defined(_WINDOWS_) >>>>>>>>>> #error "You're including win32.h after having already included >>>>>>>>>> windows.h. Don't do this!" >>>>>>>>>> #endif >>>>>>>>>> >>>>>>>>>> // And also make sure windows.h can't get included after us >>>>>>>>>> #define _WINDOWS_ >>>>>>>>>> >>>>>>>>>> For the record, I tried the test case you linked when windows.h >>>>>>>>>> is not included in main.cpp and it works (but still has the bug about int >>>>>>>>>> and long). >>>>>>>>>> >>>>>>>>>> On Mon, Jan 22, 2018 at 2:23 PM Leonardo Santagada < >>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> It is super gross, but we copy parts of windows.h because having >>>>>>>>>>> all of it if both gigantic and very very messy. So our win32.h has a couple >>>>>>>>>>> thousands of lines and not 30k+ for windows.h and we try to have zero >>>>>>>>>>> macros. Win32.h doesn't include windows.h so using ::BOOL wouldn't work. We >>>>>>>>>>> don't want to create a namespace, we just want a cleaner interface to >>>>>>>>>>> windows api. The namespace with c linkage is the way to trick cl into >>>>>>>>>>> allowing us to in some files have both windows.h and Win32.h. I really >>>>>>>>>>> don't see any way for us to have this Win32.h without this cl support, so >>>>>>>>>>> maybe we should either put windows.h in a compiled header somewhere and not >>>>>>>>>>> care that it is infecting everything or just have one place we can call to >>>>>>>>>>> clean up after including windows.h (a massive set of undefs). >>>>>>>>>>> >>>>>>>>>>> So using can't work, because we never intentionally import >>>>>>>>>>> windows.h and win32.h on the same translation unit. >>>>>>>>>>> >>>>>>>>>>> On Mon, Jan 22, 2018 at 7:08 PM, Zachary Turner < >>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> This is pretty gross, honestly :) >>>>>>>>>>>> >>>>>>>>>>>> Can't you just use using declarations? >>>>>>>>>>>> >>>>>>>>>>>> namespace Win32 { >>>>>>>>>>>> extern "C" { >>>>>>>>>>>> >>>>>>>>>>>> using ::BOOL; >>>>>>>>>>>> using ::LONG; >>>>>>>>>>>> using ::POINT; >>>>>>>>>>>> using ::LPPOINT; >>>>>>>>>>>> >>>>>>>>>>>> using ::GetCursorPos; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> This works with clang-cl. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jan 22, 2018 at 5:39 AM Leonardo Santagada < >>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Here it is a minimal example, we do this so we don't have to >>>>>>>>>>>>> import the whole windows api everywhere. >>>>>>>>>>>>> >>>>>>>>>>>>> https://gist.github.com/santagada/ >>>>>>>>>>>>> 7977e929d31c629c4bf18ebb987f6be3 >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Jan 21, 2018 at 2:31 AM, Zachary Turner < >>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Clang-cl maintains compatibility with msvc even in cases >>>>>>>>>>>>>> where it’s non standards compliant (eg 2 phase name lookup), but we try to >>>>>>>>>>>>>> keep these cases few and far between. >>>>>>>>>>>>>> >>>>>>>>>>>>>> To help me understand your case, do you mean you copy >>>>>>>>>>>>>> windows.h and modify it? How does this lead to the same struct being >>>>>>>>>>>>>> defined twice? If i were to write this: >>>>>>>>>>>>>> >>>>>>>>>>>>>> struct Foo {}; >>>>>>>>>>>>>> struct Foo {}; >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is this a small repro of the issue you’re talking about? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 3:44 PM Leonardo Santagada < >>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I can totally see something like incremental linking with a >>>>>>>>>>>>>>> simple padding between obj and a mapping file (which can also help with >>>>>>>>>>>>>>> edit and continue, something we also would love to have). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have another developer doing the port to support >>>>>>>>>>>>>>> clang-cl, but although most of our code also goes trough a version of >>>>>>>>>>>>>>> clang, migrating the rest to clang-cl has been a fight. From what I heard >>>>>>>>>>>>>>> the main problem is that we have a copy of parts of windows.h (so not to >>>>>>>>>>>>>>> bring the awful parts of it like lower case macros) and that totally works >>>>>>>>>>>>>>> on cl, but clang (at least 6.0) complains about two struct/vars with the >>>>>>>>>>>>>>> same name, even though they are exactly the same. Making clang-cl as broken >>>>>>>>>>>>>>> as cl.exe is not an option I suppose? I would love to turn on a flag >>>>>>>>>>>>>>> --accept-that-cl-made-bad-decisions-and-live-with-it and >>>>>>>>>>>>>>> have this at least until this is completely fixed in our code base. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> the biggest win with moving to cl would be a better more >>>>>>>>>>>>>>> standards compliant compiler, no 1 minute compiles on heavily templated >>>>>>>>>>>>>>> files and maybe the holy grail of ThinLTO. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner < >>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 10-15s will be hard without true incremental linking. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> At some point that's going to be the only way to get any >>>>>>>>>>>>>>>> faster, but incremental linking is hard (putting it lightly), and since our >>>>>>>>>>>>>>>> full links are already really fast we think we can get reasonably close to >>>>>>>>>>>>>>>> link.exe incremental speeds with full links. But it's never enough and I >>>>>>>>>>>>>>>> will always want it to be faster, so you may see incremental linking in the >>>>>>>>>>>>>>>> future after we hit a performance wall with full link speed :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In any case, I'm definitely interested in seeing what kind >>>>>>>>>>>>>>>> of numbers you get with /debug:ghash after you get this llvm-objcopy >>>>>>>>>>>>>>>> feature implemented. So keep me updated :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As an aside, have you tried building with clang instead of >>>>>>>>>>>>>>>> cl? If you build with clang you wouldn't even have to do this llvm-objcopy >>>>>>>>>>>>>>>> work, because it would "just work". If you've tried but ran into issues >>>>>>>>>>>>>>>> I'm interested in hearing about those too. On the other hand, it's also >>>>>>>>>>>>>>>> reasonable to only switch one thing at a time. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada < >>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> if we get to < 30s I think most users would prefer it to >>>>>>>>>>>>>>>>> link.exe, just hopping there is still some more optimizations to get closer >>>>>>>>>>>>>>>>> to ELF linking times (around 10-15s here). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner < >>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Generally speaking a good rule of thumb is that >>>>>>>>>>>>>>>>>> /debug:ghash will be close to or faster than /debug:fastlink, but with none >>>>>>>>>>>>>>>>>> of the penalties like slow debug time >>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner < >>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Chrome is actually one of my exact benchmark cases. When >>>>>>>>>>>>>>>>>>> building blink_core.dll and browser_tests.exe, i get anywhere from a 20-40% >>>>>>>>>>>>>>>>>>> reduction in link time. We have some other optimizations in the pipeline >>>>>>>>>>>>>>>>>>> but not upstream yet. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> My best time so far (including other optimizations not >>>>>>>>>>>>>>>>>>> yet upstream) is 28s on blink_core.dll, compared to 110s with /debug >>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo Santagada < >>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> You probably don't want to go down the same route that >>>>>>>>>>>>>>>>>>>>> clang goes through to write the object file. If you think yaml2coff is >>>>>>>>>>>>>>>>>>>>> convoluted, the way clang does it will just give you a headache. There are >>>>>>>>>>>>>>>>>>>>> multiple abstractions involved to account for different object file formats >>>>>>>>>>>>>>>>>>>>> (ELF, COFF, MachO) and output formats (Assembly, binary file). At least >>>>>>>>>>>>>>>>>>>>> with yaml2coff >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I think your phrase got cut there, but yeah I just >>>>>>>>>>>>>>>>>>>> found AsmPrinter.cpp and it is convoluted. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> It's true that yaml2coff is using the COFFParser >>>>>>>>>>>>>>>>>>>>> structure, but if you look at the writeCOFF function >>>>>>>>>>>>>>>>>>>>> in yaml2coff it's pretty bare-metal. The logic you need will be almost >>>>>>>>>>>>>>>>>>>>> identical, except that instead of checking the COFFParser for the various >>>>>>>>>>>>>>>>>>>>> fields, you'll check the existing COFFObjectFile, which should have similar >>>>>>>>>>>>>>>>>>>>> fields. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The only thing you need to different is when writing >>>>>>>>>>>>>>>>>>>>> the section table and section contents, to insert a new entry. Since >>>>>>>>>>>>>>>>>>>>> you're injecting a section into the middle, you'll also probably need to >>>>>>>>>>>>>>>>>>>>> push back the file pointer of all subsequent sections so that they don't >>>>>>>>>>>>>>>>>>>>> overlap. (e.g. if the original sections are 1, 2, 3, 4, 5 and you insert >>>>>>>>>>>>>>>>>>>>> between 2 and 3, then the original sections 3, 4, and 5 would need to have >>>>>>>>>>>>>>>>>>>>> their FilePointerToRawData offset by the size of the new section). >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I have the PE/COFF spec open here and I'm happy that I >>>>>>>>>>>>>>>>>>>> read a bit of it so I actually know what you are talking about... yeah it >>>>>>>>>>>>>>>>>>>> doesn't seem too complicated. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> If you need to know what values to put for the other >>>>>>>>>>>>>>>>>>>>> fields in a section header, run `dumpbin /headers foo.obj` on a >>>>>>>>>>>>>>>>>>>>> clang-generated object file that has a .debug$H section already (e.g. run >>>>>>>>>>>>>>>>>>>>> clang with -emit-codeview-ghash-section, and look at the properties of the >>>>>>>>>>>>>>>>>>>>> .debug$H section and use the same values). >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks I will do that and then also look at how the >>>>>>>>>>>>>>>>>>>> CodeView part of the code does it if I can't understand some of it. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The only invariant that needs to be maintained is that >>>>>>>>>>>>>>>>>>>>> Section[N]->FilePointerOfRawData == Section[N-1]->FilePointerOfRawData + >>>>>>>>>>>>>>>>>>>>> Section[N-1]->SizeOfRawData >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Well, that and all the sections need to be on the final >>>>>>>>>>>>>>>>>>>> file... But I'm hopeful. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Anyone has times on linking a big project like chrome >>>>>>>>>>>>>>>>>>>> with this so that at least I know what kind of performance to expect? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> My numbers are something like: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 1 pdb per obj file: link.exe takes ~15 minutes and 16GB >>>>>>>>>>>>>>>>>>>> of ram, lld-link.exe takes 2:30 minutes and ~8GB of ram >>>>>>>>>>>>>>>>>>>> around 10 pdbs per folder: link.exe takes 1 minute and >>>>>>>>>>>>>>>>>>>> 2-3GB of ram, lld-link.exe takes 1:30 minutes and ~6GB of ram >>>>>>>>>>>>>>>>>>>> faslink: link.exe takes 40 seconds, but then 20 seconds >>>>>>>>>>>>>>>>>>>> of loading at the first break point in the debugger and we lost DIA support >>>>>>>>>>>>>>>>>>>> for listing symbols. >>>>>>>>>>>>>>>>>>>> incremental: link.exe takes 8 seconds, but it only >>>>>>>>>>>>>>>>>>>> happens when very minor changes happen. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> We have an non negligible number of symbols used on >>>>>>>>>>>>>>>>>>>> some runtime systems. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo Santagada < >>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for the tips, I now have something that reads >>>>>>>>>>>>>>>>>>>>>> the obj file, finds .debug$T sections and global hashes it (proof of >>>>>>>>>>>>>>>>>>>>>> concept kind of code). What I can't find is: how does clang itself writes >>>>>>>>>>>>>>>>>>>>>> the coff files with global hashes, as that might help me understand how to >>>>>>>>>>>>>>>>>>>>>> create the .debug$H section, how to update the file section count and how >>>>>>>>>>>>>>>>>>>>>> to properly write this back. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The code on yaml2coff is expecting to be working on >>>>>>>>>>>>>>>>>>>>>> the yaml COFFParser struct and I'm having quite a bit of a headache turning >>>>>>>>>>>>>>>>>>>>>> the COFFObjectFile into a COFFParser object or compatible... Tomorrow I >>>>>>>>>>>>>>>>>>>>>> might try the very non efficient path of coff2yaml and then yaml2coff with >>>>>>>>>>>>>>>>>>>>>> the hashes header... but it seems way too inefficient and convoluted. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM Leonardo Santagada < >>>>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29 PM Leonardo >>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> No I didn't, I used cl.exe from the visual studio >>>>>>>>>>>>>>>>>>>>>>>>>> toolchain. What I'm proposing is a tool for processing .obj files in COFF >>>>>>>>>>>>>>>>>>>>>>>>>> format, reading them and generating the GHASH part. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> To make our build faster we use hundreds of unity >>>>>>>>>>>>>>>>>>>>>>>>>> build files (.cpp's with a lot of other .cpp's in them aka munch files) but >>>>>>>>>>>>>>>>>>>>>>>>>> still have a lot of single .cpp's as well (in total something like 3.4k >>>>>>>>>>>>>>>>>>>>>>>>>> .obj files). >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> ps: sorry for sending to the wrong list, I was >>>>>>>>>>>>>>>>>>>>>>>>>> reading about llvm mailing lists and jumped when I saw what I thought was a >>>>>>>>>>>>>>>>>>>>>>>>>> lld exclusive list. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> A tool like this would be useful, yes. We've >>>>>>>>>>>>>>>>>>>>>>>>> talked about it internally as well and agreed it would be useful, we just >>>>>>>>>>>>>>>>>>>>>>>>> haven't prioritized it. If you're interested in submitting a patch along >>>>>>>>>>>>>>>>>>>>>>>>> those lines though, I think it would be a good addition. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I'm not sure what the best place for it would be. >>>>>>>>>>>>>>>>>>>>>>>>> llvm-readobj and llvm-objdump seem like obvious choices, but they are >>>>>>>>>>>>>>>>>>>>>>>>> intended to be read-only, so perhaps they wouldn't be a good fit. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil is kind of a hodgepodge of everything >>>>>>>>>>>>>>>>>>>>>>>>> else related to PDBs and symbols, so I wouldn't be opposed to making a new >>>>>>>>>>>>>>>>>>>>>>>>> subcommand there called "ghash" or something that could process an object >>>>>>>>>>>>>>>>>>>>>>>>> file and output a new object file with a .debug$H section. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> A third option would be to make a new tool for it. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I don't htink it would be that hard to write. If >>>>>>>>>>>>>>>>>>>>>>>>> you're interested in trying to make a patch for this, I can offer some >>>>>>>>>>>>>>>>>>>>>>>>> guidance on where to look in the code. Otherwise it's something that we'll >>>>>>>>>>>>>>>>>>>>>>>>> probably get to, I'm just not sure when. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I would love to write it and contribute it back, >>>>>>>>>>>>>>>>>>>>>>>> please do tell, I did find some of the code of ghash in lld, but in fuzzy >>>>>>>>>>>>>>>>>>>>>>>> on the llvm codeview part of it and never seen llvm-readobj/objdump or >>>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil, but I'm not afraid to look :) >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Luckily all of the important code is hidden behind >>>>>>>>>>>>>>>>>>>>>>> library calls, and it should already just do the right thing, so I suspect >>>>>>>>>>>>>>>>>>>>>>> you won't need to know much about CodeView to do this. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I think Peter has the right idea about putting this >>>>>>>>>>>>>>>>>>>>>>> in llvm-objcopy. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> You can look at one of the existing CopyBinary >>>>>>>>>>>>>>>>>>>>>>> functions there, which currently only work for ELF, but you can just make a >>>>>>>>>>>>>>>>>>>>>>> new overload that accepts a COFFObjectFile. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I would probably start by iterating over each of the >>>>>>>>>>>>>>>>>>>>>>> sections (getNumberOfSections / getSectionName) looking for .debug$T and >>>>>>>>>>>>>>>>>>>>>>> .debug$H sections. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$H section then you can just >>>>>>>>>>>>>>>>>>>>>>> skip that object file. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$T but not a .debug$H, then >>>>>>>>>>>>>>>>>>>>>>> basically do the same thing that LLD does in PDBLinker::mergeDebugT >>>>>>>>>>>>>>>>>>>>>>> (create a CVTypeArray, and pass it to GloballyHashedType::hashTypes. >>>>>>>>>>>>>>>>>>>>>>> That will return an array of hash values. (the format of .debug$H is the >>>>>>>>>>>>>>>>>>>>>>> header, followed by the hash values). Then when you're writing the list of >>>>>>>>>>>>>>>>>>>>>>> sections, just add in the .debug$H section right after the .debug$T section. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Currently llvm-objcopy only writes ELF files, so it >>>>>>>>>>>>>>>>>>>>>>> would need to be taught to write COFF files. We have code to do this in >>>>>>>>>>>>>>>>>>>>>>> the yaml2obj utility (specifically, in yaml2coff.cpp in the function >>>>>>>>>>>>>>>>>>>>>>> writeCOFF). There may be a way to move this code to somewhere else >>>>>>>>>>>>>>>>>>>>>>> (llvm/Object/COFF.h?) so that it can be re-used by both yaml2coff and >>>>>>>>>>>>>>>>>>>>>>> llvm-objcopy, but in the worst case scenario you could copy the code and >>>>>>>>>>>>>>>>>>>>>>> re-write it to work with these new structures. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Lastly, you'll probably want to put all of this >>>>>>>>>>>>>>>>>>>>>>> behind an option in llvm-objcopy such as -add-codeview-ghash-section >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Leonardo Santagada >>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Leonardo Santagada >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Leonardo Santagada >>>>> >>>> >>> >>> >>> -- >>> >>> Leonardo Santagada >>> >>-- Leonardo Santagada -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180125/767a5ffb/attachment.html>
Leonardo Santagada via llvm-dev
2018-Jan-25 17:57 UTC
[llvm-dev] [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
Any idea on how to create this new symbol there? I saw that there is a symbol pointing to each section, but didn't understand the format, and yaml2obj doesn't check it or do anything with the list. On Thu, Jan 25, 2018 at 6:56 PM, Leonardo Santagada <santagada at gmail.com> wrote:> YES, THANK YOU... I WAS THINKING THIS BUT COMPLETELY FORGOT. > > sorry for the caps... long day of working on this, and using vs 2017, > which adds a new section type .chks64 that I couldn't find documentation > anywhere was difficult. I highly recommend everyone to just not using vs > 2017 until 15.8 or something, our internal bug list is gigantic. > > On Thu, Jan 25, 2018 at 6:52 PM, Zachary Turner <zturner at google.com> > wrote: > >> Actually I already have a theory that even though you are adding the >> section to the section table, you might not be adding a *symbol* for the >> section to the symbol table. So the existing symbols (which reference >> sections by index) will all be wrong because you've inserted a new >> section. Still though, obj2yaml would expose that. >> >> On Thu, Jan 25, 2018 at 9:50 AM Zachary Turner <zturner at google.com> >> wrote: >> >>> Yea as long as you compare clang-cl object file with automatically >>> generated .debug$H section against clang-cl object file without .debug$H >>> but added after the fact with llvm-objcopy, that should expose the problem >>> I think when you run obj2yaml on them. >>> >>> On Thu, Jan 25, 2018 at 9:49 AM Leonardo Santagada <santagada at gmail.com> >>> wrote: >>> >>>> I did reorder my sections, so that .debug$H is in the correct place, >>>> but now I get some errors on dubplicate symbols, I created a folder with >>>> examples: >>>> >>>> https://www.dropbox.com/sh/nmvzi44pi0boe76/AAA0f47O5PCJ9JiUc >>>> 6wVuwBra?dl=0 >>>> >>>> t.obj is generated by vs 2015 and it links fine with lld-link.exe, but >>>> tout.obj gives this errors: >>>> >>>> lld-link.exe /DEBUG:GHASH tout.obj >>>> LLD-LINK.EXE: error: duplicate symbol: __local_stdio_printf_options in >>>> tout.obj and in LIBCMT.lib(default_local_stdio_options.obj) >>>> LLD-LINK.EXE: error: duplicate symbol: __local_stdio_printf_options in >>>> tout.obj and in libvcruntime.lib(undname.obj) >>>> >>>> I'm using PEView from http://wjradburn.com/software/ to look at the >>>> files and can't see anything wrong, except some valid differences in the >>>> offsets being used for the data (so pointer to data is different between >>>> them). >>>> >>>> I will look into yaml2obj now to see if I see anything else weird going >>>> on. >>>> >>>> >>>> On Thu, Jan 25, 2018 at 6:41 PM, Zachary Turner <zturner at google.com> >>>> wrote: >>>> >>>>> I'm pretty confident that cl is not putting anything strange in the >>>>> .debug$T sections. We've done a lot of testing and never seen anything >>>>> except CodeView type records in a .debug$T. My hunch is that your objcopy >>>>> patch is probably not doing the right thing in one or more of the section >>>>> headers, and this is confusing the linker. >>>>> >>>>> One idea might be to build a simple object file with clang-cl but >>>>> without the magic -mllvm -emit-codeview-ghash-section, then run your >>>>> llvm-objcopy on it. Then build the same object file passing -mllvm >>>>> -emit-codeview-ghash-section. Then run obj2yaml on both and diff the >>>>> results. They should be byte-for-byte identical. That should give you a >>>>> clue about if objcopy is doing something wrong. >>>>> >>>>> On Thu, Jan 25, 2018 at 2:21 AM Leonardo Santagada < >>>>> santagada at gmail.com> wrote: >>>>> >>>>>> Don't worry, I definetly want to perfect this to generate legal obj >>>>>> files, this is just to speed up testing. >>>>>> >>>>>> Now after patching all the obj files I get this errors when linking a >>>>>> small part of our code base (msvc 2017 15.5.3, lld and llvm-objcopy 7.0.0): >>>>>> lld-link.exe : error : relocation against symbol in discarded >>>>>> section: $LN8 >>>>>> lld-link.exe : error : relocation against symbol in discarded >>>>>> section: $LN43 >>>>>> lld-link.exe : error : relocation against symbol in discarded >>>>>> section: $LN37 >>>>>> >>>>>> I'm starting to guess that cl.exe might be putting some random comdat >>>>>> or other discardable symbols in the .debug$T and clang doesn't? I will try >>>>>> to debug this and see what more I can uncover. >>>>>> >>>>>> Linking works perfectly without my llvm-objcopy pass to add .debug$H? >>>>>> >>>>>> >>>>>> On Thu, Jan 25, 2018 at 1:53 AM, Zachary Turner <zturner at google.com> >>>>>> wrote: >>>>>> >>>>>>> It might not influence LLD, but at the same time we don't want to >>>>>>> upstream something that is producing technically illegal COFF files. Also >>>>>>> good to hear about the planned changes to your header files. Looking >>>>>>> forward to hearing about your experiences with clang-cl. >>>>>>> >>>>>>> On Wed, Jan 24, 2018 at 10:41 AM Leonardo Santagada < >>>>>>> santagada at gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I finally got my first .obj file patched with .debug$H to look >>>>>>>> somewhat right. I added the new section at the end of the file so I don't >>>>>>>> have to recalculate all sections (although now I probably could position it >>>>>>>> in the middle, knowing that each section is: SizeOfRawData + >>>>>>>> (last.Header.NumberOfRelocations * (4+4+2)) and the $H needs to >>>>>>>> come right after $T in the file). That although illegal based on the coff >>>>>>>> specs doesn't seem its going to influence lld. >>>>>>>> >>>>>>>> Also we talked and we are probably going to do something similar to >>>>>>>> a bunch of windows defines and a check for our own define (to guarantee >>>>>>>> that no one imported windows.h before win32.h) and drop the namespace and >>>>>>>> the conflicting names. >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jan 23, 2018 at 12:46 AM, Zachary Turner < >>>>>>>> zturner at google.com> wrote: >>>>>>>> >>>>>>>>> That's very possible that a 3rd party indirect header include is >>>>>>>>> involved. One idea might be like I suggested where you #define _WINDOWS_ >>>>>>>>> in win32.h and guarantee that it's always included first. Then those other >>>>>>>>> headers won't be able to #include <windows.h>. but it will probably >>>>>>>>> greatly expand the amount of stuff you have to add to win32.h, as you will >>>>>>>>> probably find some callers of functions that aren't yet in your win32.h >>>>>>>>> that you'd have to add. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jan 22, 2018 at 3:28 PM Leonardo Santagada < >>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Ok some information was lost on getting this example to you, I'm >>>>>>>>>> sorry for not being clear. >>>>>>>>>> >>>>>>>>>> We have a huge code base, let's say 90% of it doesn't include >>>>>>>>>> either header, 9% include win32.h and 1% includes both, I will try to >>>>>>>>>> discover why, but my guess is they include both a third party that includes >>>>>>>>>> windows.h and some of our libs that use win32.h. >>>>>>>>>> >>>>>>>>>> I will try to fully understand this tomorrow. >>>>>>>>>> >>>>>>>>>> I guess clang will not implement this ever so finishing the >>>>>>>>>> object copier is the best solution until all code is ported to clang. >>>>>>>>>> >>>>>>>>>> On 23 Jan 2018 00:02, "Zachary Turner" <zturner at google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> You said win32.h doesn't include windows.h, but main.cpp does. >>>>>>>>>>> So what's the disadvantage of just including it in win32.h anyway, since >>>>>>>>>>> it's already going to be in every translation unit? (Unless you didn't >>>>>>>>>>> mean to #include it in main.cpp) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I guess all I can do is warn you how bad of an idea this is. >>>>>>>>>>> For starters, I already found a bug in your code ;-) >>>>>>>>>>> >>>>>>>>>>> // stdint.h >>>>>>>>>>> typedef int int32_t; >>>>>>>>>>> >>>>>>>>>>> // winnt.h >>>>>>>>>>> typedef long LONG; >>>>>>>>>>> >>>>>>>>>>> // windef.h >>>>>>>>>>> typedef struct tagPOINT >>>>>>>>>>> { >>>>>>>>>>> LONG x; // long x >>>>>>>>>>> LONG y; // long y >>>>>>>>>>> } POINT, *PPOINT, NEAR *NPPOINT, FAR *LPPOINT; >>>>>>>>>>> >>>>>>>>>>> // win32.h >>>>>>>>>>> typedef int32_t LONG; >>>>>>>>>>> >>>>>>>>>>> struct POINT >>>>>>>>>>> { >>>>>>>>>>> LONG x; // int x >>>>>>>>>>> LONG y; // int y >>>>>>>>>>> }; >>>>>>>>>>> >>>>>>>>>>> So POINT is defined two different ways. In your minimal >>>>>>>>>>> interface, it's declared as 2 int32's, which are int. In the actual >>>>>>>>>>> Windows header files, it's declared as 2 longs. >>>>>>>>>>> >>>>>>>>>>> This might seem like a unimportant bug since int and long are >>>>>>>>>>> the same size, but int and long also mangle differently and affect overload >>>>>>>>>>> resolution, so you could have weird linker errors or call the wrong >>>>>>>>>>> function overload. >>>>>>>>>>> >>>>>>>>>>> Plus, it illustrates the fact that this struct *actually is* a >>>>>>>>>>> different type from the one in the windows header. >>>>>>>>>>> >>>>>>>>>>> You said at the end that you never intentionally import win32.h >>>>>>>>>>> and windows.h from the same translation unit. But then in this example you >>>>>>>>>>> did. I wonder if you could enforce that by doing this: >>>>>>>>>>> >>>>>>>>>>> // win32.h >>>>>>>>>>> #pragma once >>>>>>>>>>> >>>>>>>>>>> // Error if windows.h was included before us. >>>>>>>>>>> #if defined(_WINDOWS_) >>>>>>>>>>> #error "You're including win32.h after having already included >>>>>>>>>>> windows.h. Don't do this!" >>>>>>>>>>> #endif >>>>>>>>>>> >>>>>>>>>>> // And also make sure windows.h can't get included after us >>>>>>>>>>> #define _WINDOWS_ >>>>>>>>>>> >>>>>>>>>>> For the record, I tried the test case you linked when windows.h >>>>>>>>>>> is not included in main.cpp and it works (but still has the bug about int >>>>>>>>>>> and long). >>>>>>>>>>> >>>>>>>>>>> On Mon, Jan 22, 2018 at 2:23 PM Leonardo Santagada < >>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> It is super gross, but we copy parts of windows.h because >>>>>>>>>>>> having all of it if both gigantic and very very messy. So our win32.h has a >>>>>>>>>>>> couple thousands of lines and not 30k+ for windows.h and we try to have >>>>>>>>>>>> zero macros. Win32.h doesn't include windows.h so using ::BOOL wouldn't >>>>>>>>>>>> work. We don't want to create a namespace, we just want a cleaner interface >>>>>>>>>>>> to windows api. The namespace with c linkage is the way to trick cl into >>>>>>>>>>>> allowing us to in some files have both windows.h and Win32.h. I really >>>>>>>>>>>> don't see any way for us to have this Win32.h without this cl support, so >>>>>>>>>>>> maybe we should either put windows.h in a compiled header somewhere and not >>>>>>>>>>>> care that it is infecting everything or just have one place we can call to >>>>>>>>>>>> clean up after including windows.h (a massive set of undefs). >>>>>>>>>>>> >>>>>>>>>>>> So using can't work, because we never intentionally import >>>>>>>>>>>> windows.h and win32.h on the same translation unit. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jan 22, 2018 at 7:08 PM, Zachary Turner < >>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> This is pretty gross, honestly :) >>>>>>>>>>>>> >>>>>>>>>>>>> Can't you just use using declarations? >>>>>>>>>>>>> >>>>>>>>>>>>> namespace Win32 { >>>>>>>>>>>>> extern "C" { >>>>>>>>>>>>> >>>>>>>>>>>>> using ::BOOL; >>>>>>>>>>>>> using ::LONG; >>>>>>>>>>>>> using ::POINT; >>>>>>>>>>>>> using ::LPPOINT; >>>>>>>>>>>>> >>>>>>>>>>>>> using ::GetCursorPos; >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> This works with clang-cl. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jan 22, 2018 at 5:39 AM Leonardo Santagada < >>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Here it is a minimal example, we do this so we don't have to >>>>>>>>>>>>>> import the whole windows api everywhere. >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://gist.github.com/santagada/7977e929d31c629c4bf18ebb98 >>>>>>>>>>>>>> 7f6be3 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Jan 21, 2018 at 2:31 AM, Zachary Turner < >>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Clang-cl maintains compatibility with msvc even in cases >>>>>>>>>>>>>>> where it’s non standards compliant (eg 2 phase name lookup), but we try to >>>>>>>>>>>>>>> keep these cases few and far between. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To help me understand your case, do you mean you copy >>>>>>>>>>>>>>> windows.h and modify it? How does this lead to the same struct being >>>>>>>>>>>>>>> defined twice? If i were to write this: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> struct Foo {}; >>>>>>>>>>>>>>> struct Foo {}; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is this a small repro of the issue you’re talking about? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 3:44 PM Leonardo Santagada < >>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I can totally see something like incremental linking with a >>>>>>>>>>>>>>>> simple padding between obj and a mapping file (which can also help with >>>>>>>>>>>>>>>> edit and continue, something we also would love to have). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We have another developer doing the port to support >>>>>>>>>>>>>>>> clang-cl, but although most of our code also goes trough a version of >>>>>>>>>>>>>>>> clang, migrating the rest to clang-cl has been a fight. From what I heard >>>>>>>>>>>>>>>> the main problem is that we have a copy of parts of windows.h (so not to >>>>>>>>>>>>>>>> bring the awful parts of it like lower case macros) and that totally works >>>>>>>>>>>>>>>> on cl, but clang (at least 6.0) complains about two struct/vars with the >>>>>>>>>>>>>>>> same name, even though they are exactly the same. Making clang-cl as broken >>>>>>>>>>>>>>>> as cl.exe is not an option I suppose? I would love to turn on a flag >>>>>>>>>>>>>>>> --accept-that-cl-made-bad-decisions-and-live-with-it and >>>>>>>>>>>>>>>> have this at least until this is completely fixed in our code base. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> the biggest win with moving to cl would be a better more >>>>>>>>>>>>>>>> standards compliant compiler, no 1 minute compiles on heavily templated >>>>>>>>>>>>>>>> files and maybe the holy grail of ThinLTO. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner < >>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 10-15s will be hard without true incremental linking. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> At some point that's going to be the only way to get any >>>>>>>>>>>>>>>>> faster, but incremental linking is hard (putting it lightly), and since our >>>>>>>>>>>>>>>>> full links are already really fast we think we can get reasonably close to >>>>>>>>>>>>>>>>> link.exe incremental speeds with full links. But it's never enough and I >>>>>>>>>>>>>>>>> will always want it to be faster, so you may see incremental linking in the >>>>>>>>>>>>>>>>> future after we hit a performance wall with full link speed :) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In any case, I'm definitely interested in seeing what kind >>>>>>>>>>>>>>>>> of numbers you get with /debug:ghash after you get this llvm-objcopy >>>>>>>>>>>>>>>>> feature implemented. So keep me updated :) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As an aside, have you tried building with clang instead of >>>>>>>>>>>>>>>>> cl? If you build with clang you wouldn't even have to do this llvm-objcopy >>>>>>>>>>>>>>>>> work, because it would "just work". If you've tried but ran into issues >>>>>>>>>>>>>>>>> I'm interested in hearing about those too. On the other hand, it's also >>>>>>>>>>>>>>>>> reasonable to only switch one thing at a time. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada < >>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> if we get to < 30s I think most users would prefer it to >>>>>>>>>>>>>>>>>> link.exe, just hopping there is still some more optimizations to get closer >>>>>>>>>>>>>>>>>> to ELF linking times (around 10-15s here). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner < >>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Generally speaking a good rule of thumb is that >>>>>>>>>>>>>>>>>>> /debug:ghash will be close to or faster than /debug:fastlink, but with none >>>>>>>>>>>>>>>>>>> of the penalties like slow debug time >>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner < >>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Chrome is actually one of my exact benchmark cases. >>>>>>>>>>>>>>>>>>>> When building blink_core.dll and browser_tests.exe, i get anywhere from a >>>>>>>>>>>>>>>>>>>> 20-40% reduction in link time. We have some other optimizations in the >>>>>>>>>>>>>>>>>>>> pipeline but not upstream yet. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> My best time so far (including other optimizations not >>>>>>>>>>>>>>>>>>>> yet upstream) is 28s on blink_core.dll, compared to 110s with /debug >>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo Santagada < >>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> You probably don't want to go down the same route >>>>>>>>>>>>>>>>>>>>>> that clang goes through to write the object file. If you think yaml2coff >>>>>>>>>>>>>>>>>>>>>> is convoluted, the way clang does it will just give you a headache. There >>>>>>>>>>>>>>>>>>>>>> are multiple abstractions involved to account for different object file >>>>>>>>>>>>>>>>>>>>>> formats (ELF, COFF, MachO) and output formats (Assembly, binary file). At >>>>>>>>>>>>>>>>>>>>>> least with yaml2coff >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I think your phrase got cut there, but yeah I just >>>>>>>>>>>>>>>>>>>>> found AsmPrinter.cpp and it is convoluted. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> It's true that yaml2coff is using the COFFParser >>>>>>>>>>>>>>>>>>>>>> structure, but if you look at the writeCOFF function >>>>>>>>>>>>>>>>>>>>>> in yaml2coff it's pretty bare-metal. The logic you need will be almost >>>>>>>>>>>>>>>>>>>>>> identical, except that instead of checking the COFFParser for the various >>>>>>>>>>>>>>>>>>>>>> fields, you'll check the existing COFFObjectFile, which should have similar >>>>>>>>>>>>>>>>>>>>>> fields. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The only thing you need to different is when writing >>>>>>>>>>>>>>>>>>>>>> the section table and section contents, to insert a new entry. Since >>>>>>>>>>>>>>>>>>>>>> you're injecting a section into the middle, you'll also probably need to >>>>>>>>>>>>>>>>>>>>>> push back the file pointer of all subsequent sections so that they don't >>>>>>>>>>>>>>>>>>>>>> overlap. (e.g. if the original sections are 1, 2, 3, 4, 5 and you insert >>>>>>>>>>>>>>>>>>>>>> between 2 and 3, then the original sections 3, 4, and 5 would need to have >>>>>>>>>>>>>>>>>>>>>> their FilePointerToRawData offset by the size of the new section). >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I have the PE/COFF spec open here and I'm happy that I >>>>>>>>>>>>>>>>>>>>> read a bit of it so I actually know what you are talking about... yeah it >>>>>>>>>>>>>>>>>>>>> doesn't seem too complicated. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> If you need to know what values to put for the other >>>>>>>>>>>>>>>>>>>>>> fields in a section header, run `dumpbin /headers foo.obj` on a >>>>>>>>>>>>>>>>>>>>>> clang-generated object file that has a .debug$H section already (e.g. run >>>>>>>>>>>>>>>>>>>>>> clang with -emit-codeview-ghash-section, and look at the properties of the >>>>>>>>>>>>>>>>>>>>>> .debug$H section and use the same values). >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks I will do that and then also look at how the >>>>>>>>>>>>>>>>>>>>> CodeView part of the code does it if I can't understand some of it. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The only invariant that needs to be maintained is >>>>>>>>>>>>>>>>>>>>>> that Section[N]->FilePointerOfRawData =>>>>>>>>>>>>>>>>>>>>>> Section[N-1]->FilePointerOfRawData + >>>>>>>>>>>>>>>>>>>>>> Section[N-1]->SizeOfRawData >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Well, that and all the sections need to be on the >>>>>>>>>>>>>>>>>>>>> final file... But I'm hopeful. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Anyone has times on linking a big project like chrome >>>>>>>>>>>>>>>>>>>>> with this so that at least I know what kind of performance to expect? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> My numbers are something like: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1 pdb per obj file: link.exe takes ~15 minutes and >>>>>>>>>>>>>>>>>>>>> 16GB of ram, lld-link.exe takes 2:30 minutes and ~8GB of ram >>>>>>>>>>>>>>>>>>>>> around 10 pdbs per folder: link.exe takes 1 minute and >>>>>>>>>>>>>>>>>>>>> 2-3GB of ram, lld-link.exe takes 1:30 minutes and ~6GB of ram >>>>>>>>>>>>>>>>>>>>> faslink: link.exe takes 40 seconds, but then 20 >>>>>>>>>>>>>>>>>>>>> seconds of loading at the first break point in the debugger and we lost DIA >>>>>>>>>>>>>>>>>>>>> support for listing symbols. >>>>>>>>>>>>>>>>>>>>> incremental: link.exe takes 8 seconds, but it only >>>>>>>>>>>>>>>>>>>>> happens when very minor changes happen. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> We have an non negligible number of symbols used on >>>>>>>>>>>>>>>>>>>>> some runtime systems. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo Santagada < >>>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the tips, I now have something that reads >>>>>>>>>>>>>>>>>>>>>>> the obj file, finds .debug$T sections and global hashes it (proof of >>>>>>>>>>>>>>>>>>>>>>> concept kind of code). What I can't find is: how does clang itself writes >>>>>>>>>>>>>>>>>>>>>>> the coff files with global hashes, as that might help me understand how to >>>>>>>>>>>>>>>>>>>>>>> create the .debug$H section, how to update the file section count and how >>>>>>>>>>>>>>>>>>>>>>> to properly write this back. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> The code on yaml2coff is expecting to be working on >>>>>>>>>>>>>>>>>>>>>>> the yaml COFFParser struct and I'm having quite a bit of a headache turning >>>>>>>>>>>>>>>>>>>>>>> the COFFObjectFile into a COFFParser object or compatible... Tomorrow I >>>>>>>>>>>>>>>>>>>>>>> might try the very non efficient path of coff2yaml and then yaml2coff with >>>>>>>>>>>>>>>>>>>>>>> the hashes header... but it seems way too inefficient and convoluted. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM Leonardo Santagada < >>>>>>>>>>>>>>>>>>>>>>>> santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM, Zachary Turner < >>>>>>>>>>>>>>>>>>>>>>>>> zturner at google.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29 PM Leonardo >>>>>>>>>>>>>>>>>>>>>>>>>> Santagada <santagada at gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> No I didn't, I used cl.exe from the visual >>>>>>>>>>>>>>>>>>>>>>>>>>> studio toolchain. What I'm proposing is a tool for processing .obj files in >>>>>>>>>>>>>>>>>>>>>>>>>>> COFF format, reading them and generating the GHASH part. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> To make our build faster we use hundreds of >>>>>>>>>>>>>>>>>>>>>>>>>>> unity build files (.cpp's with a lot of other .cpp's in them aka munch >>>>>>>>>>>>>>>>>>>>>>>>>>> files) but still have a lot of single .cpp's as well (in total something >>>>>>>>>>>>>>>>>>>>>>>>>>> like 3.4k .obj files). >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> ps: sorry for sending to the wrong list, I was >>>>>>>>>>>>>>>>>>>>>>>>>>> reading about llvm mailing lists and jumped when I saw what I thought was a >>>>>>>>>>>>>>>>>>>>>>>>>>> lld exclusive list. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> A tool like this would be useful, yes. We've >>>>>>>>>>>>>>>>>>>>>>>>>> talked about it internally as well and agreed it would be useful, we just >>>>>>>>>>>>>>>>>>>>>>>>>> haven't prioritized it. If you're interested in submitting a patch along >>>>>>>>>>>>>>>>>>>>>>>>>> those lines though, I think it would be a good addition. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I'm not sure what the best place for it would >>>>>>>>>>>>>>>>>>>>>>>>>> be. llvm-readobj and llvm-objdump seem like obvious choices, but they are >>>>>>>>>>>>>>>>>>>>>>>>>> intended to be read-only, so perhaps they wouldn't be a good fit. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil is kind of a hodgepodge of >>>>>>>>>>>>>>>>>>>>>>>>>> everything else related to PDBs and symbols, so I wouldn't be opposed to >>>>>>>>>>>>>>>>>>>>>>>>>> making a new subcommand there called "ghash" or something that could >>>>>>>>>>>>>>>>>>>>>>>>>> process an object file and output a new object file with a .debug$H section. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> A third option would be to make a new tool for it. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I don't htink it would be that hard to write. If >>>>>>>>>>>>>>>>>>>>>>>>>> you're interested in trying to make a patch for this, I can offer some >>>>>>>>>>>>>>>>>>>>>>>>>> guidance on where to look in the code. Otherwise it's something that we'll >>>>>>>>>>>>>>>>>>>>>>>>>> probably get to, I'm just not sure when. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I would love to write it and contribute it back, >>>>>>>>>>>>>>>>>>>>>>>>> please do tell, I did find some of the code of ghash in lld, but in fuzzy >>>>>>>>>>>>>>>>>>>>>>>>> on the llvm codeview part of it and never seen llvm-readobj/objdump or >>>>>>>>>>>>>>>>>>>>>>>>> llvm-pdbutil, but I'm not afraid to look :) >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Luckily all of the important code is hidden behind >>>>>>>>>>>>>>>>>>>>>>>> library calls, and it should already just do the right thing, so I suspect >>>>>>>>>>>>>>>>>>>>>>>> you won't need to know much about CodeView to do this. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I think Peter has the right idea about putting this >>>>>>>>>>>>>>>>>>>>>>>> in llvm-objcopy. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> You can look at one of the existing CopyBinary >>>>>>>>>>>>>>>>>>>>>>>> functions there, which currently only work for ELF, but you can just make a >>>>>>>>>>>>>>>>>>>>>>>> new overload that accepts a COFFObjectFile. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I would probably start by iterating over each of >>>>>>>>>>>>>>>>>>>>>>>> the sections (getNumberOfSections / getSectionName) looking for .debug$T >>>>>>>>>>>>>>>>>>>>>>>> and .debug$H sections. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$H section then you can just >>>>>>>>>>>>>>>>>>>>>>>> skip that object file. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> If you find a .debug$T but not a .debug$H, then >>>>>>>>>>>>>>>>>>>>>>>> basically do the same thing that LLD does in PDBLinker::mergeDebugT >>>>>>>>>>>>>>>>>>>>>>>> (create a CVTypeArray, and pass it to GloballyHashedType::hashTypes. >>>>>>>>>>>>>>>>>>>>>>>> That will return an array of hash values. (the format of .debug$H is the >>>>>>>>>>>>>>>>>>>>>>>> header, followed by the hash values). Then when you're writing the list of >>>>>>>>>>>>>>>>>>>>>>>> sections, just add in the .debug$H section right after the .debug$T section. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Currently llvm-objcopy only writes ELF files, so it >>>>>>>>>>>>>>>>>>>>>>>> would need to be taught to write COFF files. We have code to do this in >>>>>>>>>>>>>>>>>>>>>>>> the yaml2obj utility (specifically, in yaml2coff.cpp in the function >>>>>>>>>>>>>>>>>>>>>>>> writeCOFF). There may be a way to move this code to somewhere else >>>>>>>>>>>>>>>>>>>>>>>> (llvm/Object/COFF.h?) so that it can be re-used by both yaml2coff and >>>>>>>>>>>>>>>>>>>>>>>> llvm-objcopy, but in the worst case scenario you could copy the code and >>>>>>>>>>>>>>>>>>>>>>>> re-write it to work with these new structures. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Lastly, you'll probably want to put all of this >>>>>>>>>>>>>>>>>>>>>>>> behind an option in llvm-objcopy such as -add-codeview-ghash-section >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Leonardo Santagada >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Leonardo Santagada >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Leonardo Santagada >>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Leonardo Santagada >>>> >>> > > > -- > > Leonardo Santagada >-- Leonardo Santagada -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180125/e5c3a53e/attachment-0001.html>
Seemingly Similar Threads
- [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
- [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
- [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
- [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)
- [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)