via llvm-dev
2018-Nov-20 14:28 UTC
[llvm-dev] Ninja build (on Windows anyway) may be doing redundant work
(resend to the list) And of course, just as I say that, my next ninja build shows the line only once. On reflection I am less sure that the lack of a [N/M] line means they are from the same invocation. Surely ninja could spawn two links, which then independently report "Creating library" after ninja emits the [N/M] lines. --paulr From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of via llvm-dev Sent: Tuesday, November 20, 2018 9:01 AM To: thakis at chromium.org Cc: llvm-dev at lists.llvm.org Subject: Re: [llvm-dev] Ninja build (on Windows anyway) may be doing redundant work OK, worth knowing. Thanks. --paulr From: Nico Weber [mailto:thakis at chromium.org] Sent: Monday, November 19, 2018 7:11 PM To: Robinson, Paul Cc: Zachary Turner; llvm-dev Subject: Re: [llvm-dev] Ninja build (on Windows anyway) may be doing redundant work Since there's no "[2663/3121] " line between the two messages, the two lines are from the same link.exe invocation. I don't know why link.exe thinks it needs to print this line twice, ninja doesn't have anything to do with it. On Mon, Nov 19, 2018 at 6:57 PM <paul.robinson at sony.com<mailto:paul.robinson at sony.com>> wrote: I'm more concerned about seeing the message come out twice, which implies Ninja is executing a rule twice when it doesn't need to. --paulr From: Nico Weber [mailto:thakis at chromium.org<mailto:thakis at chromium.org>] Sent: Monday, November 19, 2018 6:09 PM To: Zachary Turner Cc: Robinson, Paul; llvm-dev Subject: Re: [llvm-dev] Ninja build (on Windows anyway) may be doing redundant work Do you still see this if you use lld-link for linking? The "corrupt obj file" is something we saw on chrome's bots every now and then before we switched to lld. On Mon, Nov 19, 2018 at 5:27 PM Zachary Turner <zturner at google.com<mailto:zturner at google.com>> wrote: +Nico Weber<mailto:thakis at google.com> On Mon, Nov 19, 2018 at 12:25 PM via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: This afternoon I did a clean build of llvm using ninja and MSVC 2015 on Windows 10. I saw something curious and wondered if any of the ninja or CMake experts out there can explain/fix it. [2663/3121] Linking CXX shared library bin\LTO.dll Creating library lib\LTO.dll and object lib\LTO.exp Creating library lib\LTO.dll and object lib\LTO.exp [3120/3121] Running the LLVM regression tests [[[ etc ]]] I'm curious about the "creating library" message popping out twice. Sometimes ninja builds will die on me, complaining about a corrupted object file, usually related to LTO. Rerunning ninja usually completes successfully. So I wonder if there's some mishap in the dependencies somewhere that actually causes the build to try to link the DLL twice, which of course might corrupt a file if the links are overlapping in time. Thought I'd throw this out there in case someone wants to take a look. --paulr _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/0fd8f3ec/attachment-0001.html>
Nico Weber via llvm-dev
2018-Nov-20 14:49 UTC
[llvm-dev] Ninja build (on Windows anyway) may be doing redundant work
Ninja makes subprocesses write to a pipe and prints their output preceded by the [N/M] line before printing their output the process is done, to not get interleaving output from parallel processes and to make it easy to see which process a given output belongs to. So link.exe definitely prints this line twice. On Tue, Nov 20, 2018 at 9:28 AM <paul.robinson at sony.com> wrote:> (resend to the list) > > And of course, just as I say that, my next ninja build shows the line only > once. > > On reflection I am less sure that the lack of a [N/M] line means they are > from the same invocation. Surely ninja could spawn two links, which then > independently report "Creating library" after ninja emits the [N/M] lines. > > --paulr > > > > > > *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of *via > llvm-dev > *Sent:* Tuesday, November 20, 2018 9:01 AM > *To:* thakis at chromium.org > *Cc:* llvm-dev at lists.llvm.org > *Subject:* Re: [llvm-dev] Ninja build (on Windows anyway) may be doing > redundant work > > > > OK, worth knowing. Thanks. > > --paulr > > > > *From:* Nico Weber [mailto:thakis at chromium.org] > *Sent:* Monday, November 19, 2018 7:11 PM > *To:* Robinson, Paul > *Cc:* Zachary Turner; llvm-dev > *Subject:* Re: [llvm-dev] Ninja build (on Windows anyway) may be doing > redundant work > > > > Since there's no "[2663/3121] " line between the two messages, the two > lines are from the same link.exe invocation. I don't know why link.exe > thinks it needs to print this line twice, ninja doesn't have anything to do > with it. > > > > On Mon, Nov 19, 2018 at 6:57 PM <paul.robinson at sony.com> wrote: > > I'm more concerned about seeing the message come out twice, which implies > Ninja is executing a rule twice when it doesn't need to. > > --paulr > > > > *From:* Nico Weber [mailto:thakis at chromium.org] > *Sent:* Monday, November 19, 2018 6:09 PM > *To:* Zachary Turner > *Cc:* Robinson, Paul; llvm-dev > *Subject:* Re: [llvm-dev] Ninja build (on Windows anyway) may be doing > redundant work > > > > Do you still see this if you use lld-link for linking? > > > > The "corrupt obj file" is something we saw on chrome's bots every now and > then before we switched to lld. > > > > On Mon, Nov 19, 2018 at 5:27 PM Zachary Turner <zturner at google.com> wrote: > > +Nico Weber <thakis at google.com> > > On Mon, Nov 19, 2018 at 12:25 PM via llvm-dev <llvm-dev at lists.llvm.org> > wrote: > > This afternoon I did a clean build of llvm using ninja and MSVC 2015 on > Windows 10. I saw something curious and wondered if any of the ninja or > CMake experts out there can explain/fix it. > > [2663/3121] Linking CXX shared library bin\LTO.dll > Creating library lib\LTO.dll and object lib\LTO.exp > Creating library lib\LTO.dll and object lib\LTO.exp > [3120/3121] Running the LLVM regression tests > [[[ etc ]]] > > I'm curious about the "creating library" message popping out twice. > Sometimes ninja builds will die on me, complaining about a corrupted > object file, usually related to LTO. Rerunning ninja usually completes > successfully. So I wonder if there's some mishap in the dependencies > somewhere that actually causes the build to try to link the DLL twice, > which of course might corrupt a file if the links are overlapping in > time. > > Thought I'd throw this out there in case someone wants to take a look. > --paulr > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/2724c5c1/attachment.html>
via llvm-dev
2018-Nov-20 18:32 UTC
[llvm-dev] Ninja build (on Windows anyway) may be doing redundant work
So when ninja says "[N/M] Building foobar" that means it's actually *all done* building foobar. Excellent. Re. the double message, "blame Microsoft" as I'm not motivated to dig into it any further. Thanks for the info! It did help. --paulr From: Nico Weber [mailto:thakis at chromium.org] Sent: Tuesday, November 20, 2018 9:49 AM To: Robinson, Paul Cc: llvm-dev Subject: Re: [llvm-dev] Ninja build (on Windows anyway) may be doing redundant work Ninja makes subprocesses write to a pipe and prints their output preceded by the [N/M] line before printing their output the process is done, to not get interleaving output from parallel processes and to make it easy to see which process a given output belongs to. So link.exe definitely prints this line twice. On Tue, Nov 20, 2018 at 9:28 AM <paul.robinson at sony.com<mailto:paul.robinson at sony.com>> wrote: (resend to the list) And of course, just as I say that, my next ninja build shows the line only once. On reflection I am less sure that the lack of a [N/M] line means they are from the same invocation. Surely ninja could spawn two links, which then independently report "Creating library" after ninja emits the [N/M] lines. --paulr From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of via llvm-dev Sent: Tuesday, November 20, 2018 9:01 AM To: thakis at chromium.org<mailto:thakis at chromium.org> Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] Ninja build (on Windows anyway) may be doing redundant work OK, worth knowing. Thanks. --paulr From: Nico Weber [mailto:thakis at chromium.org<mailto:thakis at chromium.org>] Sent: Monday, November 19, 2018 7:11 PM To: Robinson, Paul Cc: Zachary Turner; llvm-dev Subject: Re: [llvm-dev] Ninja build (on Windows anyway) may be doing redundant work Since there's no "[2663/3121] " line between the two messages, the two lines are from the same link.exe invocation. I don't know why link.exe thinks it needs to print this line twice, ninja doesn't have anything to do with it. On Mon, Nov 19, 2018 at 6:57 PM <paul.robinson at sony.com<mailto:paul.robinson at sony.com>> wrote: I'm more concerned about seeing the message come out twice, which implies Ninja is executing a rule twice when it doesn't need to. --paulr From: Nico Weber [mailto:thakis at chromium.org<mailto:thakis at chromium.org>] Sent: Monday, November 19, 2018 6:09 PM To: Zachary Turner Cc: Robinson, Paul; llvm-dev Subject: Re: [llvm-dev] Ninja build (on Windows anyway) may be doing redundant work Do you still see this if you use lld-link for linking? The "corrupt obj file" is something we saw on chrome's bots every now and then before we switched to lld. On Mon, Nov 19, 2018 at 5:27 PM Zachary Turner <zturner at google.com<mailto:zturner at google.com>> wrote: +Nico Weber<mailto:thakis at google.com> On Mon, Nov 19, 2018 at 12:25 PM via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: This afternoon I did a clean build of llvm using ninja and MSVC 2015 on Windows 10. I saw something curious and wondered if any of the ninja or CMake experts out there can explain/fix it. [2663/3121] Linking CXX shared library bin\LTO.dll Creating library lib\LTO.dll and object lib\LTO.exp Creating library lib\LTO.dll and object lib\LTO.exp [3120/3121] Running the LLVM regression tests [[[ etc ]]] I'm curious about the "creating library" message popping out twice. Sometimes ninja builds will die on me, complaining about a corrupted object file, usually related to LTO. Rerunning ninja usually completes successfully. So I wonder if there's some mishap in the dependencies somewhere that actually causes the build to try to link the DLL twice, which of course might corrupt a file if the links are overlapping in time. Thought I'd throw this out there in case someone wants to take a look. --paulr _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/bb59467b/attachment.html>