<rongqing.li@windriver.com>
2013-Sep-18 01:11 UTC
[PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue
From: Roy Li <rongqing.li@windriver.com> The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file. #grep ''#include "version.h"'' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h" Signed-off-by: Roy Li <rongqing.li@windriver.com> --- Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3 can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again. Makefile | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Makefile b/Makefile index c43cb68..a7c259c 100644 --- a/Makefile +++ b/Makefile @@ -81,6 +81,12 @@ endif @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $< +btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h +btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h +btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h +btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h +btrfs-debug-tree.o:version.h + %.static.o: %.c @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ -- 1.7.10.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Eric Sandeen
2013-Sep-18 01:51 UTC
[PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue
On 9/17/13 8:11 PM, rongqing.li@windriver.com wrote:> From: Roy Li <rongqing.li@windriver.com> > > The dependencies of "all: version.h" or other similar ones can not > fix the parallel build failure, only reduce the times; In fact, > many *.o files require version.h file. > > #grep ''#include "version.h"'' ./ -r > ./btrfs-corrupt-block.c:#include "version.h" > ./btrfs.c:#include "version.h" > ./btrfs-image.c:#include "version.h" > ./cmds-filesystem.c:#include "version.h" > ./btrfs-show-super.c:#include "version.h" > ./btrfs-select-super.c:#include "version.h" > ./cmds-restore.c:#include "version.h" > ./btrfs-find-root.c:#include "version.h" > ./mkfs.c:#include "version.h" > ./btrfs-zero-log.c:#include "version.h" > ./btrfs-defrag.c:#include "version.h" > ./cmds-chunk.c:#include "version.h" > ./btrfstune.c:#include "version.h" > ./btrfs-calc-size.c:#include "version.h" > ./btrfs-map-logical.c:#include "version.h" > ./cmds-check.c:#include "version.h" > ./btrfs-debug-tree.c:#include "version.h" > > Signed-off-by: Roy Li <rongqing.li@windriver.com> > --- > Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3 > can not fix the build failure, when build enough times on a 16 core cpu, > the build failure happens again, so I refix it again. > > > Makefile | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/Makefile b/Makefile > index c43cb68..a7c259c 100644 > --- a/Makefile > +++ b/Makefile > @@ -81,6 +81,12 @@ endif > @echo " [CC] $@" > $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $< > > +btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h > +btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h > +btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h > +btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h > +btrfs-debug-tree.o:version.h > + > %.static.o: %.c > @echo " [CC] $@" > $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ >I think this can be done more cleanly, I''ll send a patch. -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Rongqing Li
2013-Sep-18 01:55 UTC
Re: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue
On 09/18/2013 09:51 AM, Eric Sandeen wrote:>> @echo " [CC] $@" >> > $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ >> > > I think this can be done more cleanly, I''ll send a patch. > > -Eric > >Thanks, expect your patch -Roy -- Best Reagrds, Roy | RongQing Li -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Eric Sandeen
2013-Sep-18 01:56 UTC
[PATCH] btrfs-progs: Add version.h dependency to fix parallel builds
Change the suffix rule to ensure that version.h is built before we try to create any .o file. Reported-by: Roy Li <rongqing.li@windriver.com> Signed-off-by: Eric Sandeen <sandeen@redhat.com> --- I dunno, does this work? My gnu make is rusty. diff --git a/Makefile b/Makefile index 3d715d8..d25054f 100644 --- a/Makefile +++ b/Makefile @@ -107,13 +107,13 @@ endif %.o.d: %.c $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< -.c.o: +%.o: %.c version.h @$(check_echo) " [SP] $<" $(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $< @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $< -%.static.o: %.c +%.static.o: %.c version.h @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Eric Sandeen
2013-Sep-18 02:26 UTC
Re: [PATCH] btrfs-progs: Add version.h dependency to fix parallel builds
On 9/17/13 8:56 PM, Eric Sandeen wrote:> Change the suffix rule to ensure that version.h is > built before we try to create any .o file. > > Reported-by: Roy Li <rongqing.li@windriver.com> > Signed-off-by: Eric Sandeen <sandeen@redhat.com> > --- > > I dunno, does this work? My gnu make is rusty.I''m actually confused by what the makefile is doing with the dependency generation, maybe my patch isn''t right. this: %.o.d: %.c $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< completely confuses me :) -Eric> diff --git a/Makefile b/Makefile > index 3d715d8..d25054f 100644 > --- a/Makefile > +++ b/Makefile > @@ -107,13 +107,13 @@ endif > %.o.d: %.c > $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< > > -.c.o: > +%.o: %.c version.h > @$(check_echo) " [SP] $<" > $(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $< > @echo " [CC] $@" > $(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $< > > -%.static.o: %.c > +%.static.o: %.c version.h > @echo " [CC] $@" > $(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Rongqing Li
2013-Sep-18 06:39 UTC
Re: [PATCH] btrfs-progs: Add version.h dependency to fix parallel builds
On 09/18/2013 10:26 AM, Eric Sandeen wrote:> On 9/17/13 8:56 PM, Eric Sandeen wrote: >> Change the suffix rule to ensure that version.h is >> built before we try to create any .o file. >> >> Reported-by: Roy Li <rongqing.li@windriver.com> >> Signed-off-by: Eric Sandeen <sandeen@redhat.com> >> --- >> >> I dunno, does this work? My gnu make is rusty. > > I''m actually confused by what the makefile is doing with the > dependency generation, maybe my patch isn''t right. > > this: > > %.o.d: %.c > $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< > > completely confuses me :) > > -EricIt lead to the below building error: n file included from ctree.c:18:0: | ctree.h:30:24: fatal error: btrfs/list.h: No such file or directory | #include <btrfs/list.h> | ^ | compilation terminated. | In file included from extent-tree.c:22:0: | radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory | #include <btrfs/kerncompat.h> | ^ | compilation terminated. | In file included from disk-io.c:29:0: | radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory | #include <btrfs/kerncompat.h> | ^ | compilation terminated. | In file included from radix-tree.c:40:0: | radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory | #include <btrfs/kerncompat.h> | ^ | compilation terminated. | In file included from root-tree.c:19:0: | ctree.h:30:24: fatal error: btrfs/list.h: No such file or directory | #include <btrfs/list.h> | ^ | compilation terminated. -Roy> >> diff --git a/Makefile b/Makefile >> index 3d715d8..d25054f 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -107,13 +107,13 @@ endif >> %.o.d: %.c >> $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< >> >> -.c.o: >> +%.o: %.c version.h >> @$(check_echo) " [SP] $<" >> $(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $< >> @echo " [CC] $@" >> $(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $< >> >> -%.static.o: %.c >> +%.static.o: %.c version.h >> @echo " [CC] $@" >> $(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > >-- Best Reagrds, Roy | RongQing Li -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
David Sterba
2013-Sep-18 14:51 UTC
Re: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue
On Wed, Sep 18, 2013 at 09:11:01AM +0800, rongqing.li@windriver.com wrote:> From: Roy Li <rongqing.li@windriver.com> > > The dependencies of "all: version.h" or other similar ones can not > fix the parallel build failure, only reduce the times; In fact, > many *.o files require version.h file. > > #grep ''#include "version.h"'' ./ -r > ./btrfs-corrupt-block.c:#include "version.h" > ./btrfs.c:#include "version.h" > ./btrfs-image.c:#include "version.h" > ./cmds-filesystem.c:#include "version.h" > ./btrfs-show-super.c:#include "version.h" > ./btrfs-select-super.c:#include "version.h" > ./cmds-restore.c:#include "version.h" > ./btrfs-find-root.c:#include "version.h" > ./mkfs.c:#include "version.h" > ./btrfs-zero-log.c:#include "version.h" > ./btrfs-defrag.c:#include "version.h" > ./cmds-chunk.c:#include "version.h" > ./btrfstune.c:#include "version.h" > ./btrfs-calc-size.c:#include "version.h" > ./btrfs-map-logical.c:#include "version.h" > ./cmds-check.c:#include "version.h" > ./btrfs-debug-tree.c:#include "version.h" > > Signed-off-by: Roy Li <rongqing.li@windriver.com> > --- > Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3 > can not fix the build failure, when build enough times on a 16 core cpu, > the build failure happens again, so I refix it again.I''m running make -j all the time but haven''t seen the build fail due to missing version.h for a long time. With this patch or the previous one or with Eric''s, all fine. The generated dependency files contain version.h so make has complete information how to order the rules. The dependency files are generated by an implicit rule .c -> .o.d, so there should be no problem for any of the files listed above. If you think the patch "btrfs-progs: fix parallel build" does not help much I''ll drop it. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Rongqing Li
2013-Sep-22 01:06 UTC
Re: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue
On 09/18/2013 10:51 PM, David Sterba wrote:> On Wed, Sep 18, 2013 at 09:11:01AM +0800, rongqing.li@windriver.com wrote: >> From: Roy Li <rongqing.li@windriver.com> >> >> The dependencies of "all: version.h" or other similar ones can not >> fix the parallel build failure, only reduce the times; In fact, >> many *.o files require version.h file. >> >> #grep ''#include "version.h"'' ./ -r >> ./btrfs-corrupt-block.c:#include "version.h" >> ./btrfs.c:#include "version.h" >> ./btrfs-image.c:#include "version.h" >> ./cmds-filesystem.c:#include "version.h" >> ./btrfs-show-super.c:#include "version.h" >> ./btrfs-select-super.c:#include "version.h" >> ./cmds-restore.c:#include "version.h" >> ./btrfs-find-root.c:#include "version.h" >> ./mkfs.c:#include "version.h" >> ./btrfs-zero-log.c:#include "version.h" >> ./btrfs-defrag.c:#include "version.h" >> ./cmds-chunk.c:#include "version.h" >> ./btrfstune.c:#include "version.h" >> ./btrfs-calc-size.c:#include "version.h" >> ./btrfs-map-logical.c:#include "version.h" >> ./cmds-check.c:#include "version.h" >> ./btrfs-debug-tree.c:#include "version.h" >> >> Signed-off-by: Roy Li <rongqing.li@windriver.com> >> --- >> Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3 >> can not fix the build failure, when build enough times on a 16 core cpu, >> the build failure happens again, so I refix it again. > > I''m running make -j all the time but haven''t seen the build fail due to > missing version.h for a long time. With this patch or the previous one > or with Eric''s, all fine. The generated dependency files contain > version.h so make has complete information how to order the rules. >I want to know how many cores your cpu has? I can not reproduce it on my 2 cores cpu, but it always happens when run on a server which is a 16 cores cpu and "make -j20"> The dependency files are generated by an implicit rule .c -> .o.d, so > there should be no problem for any of the files listed above. >Do you means the below: .c.o: $(Q)$(check) $< @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $< -Roy> If you think the patch "btrfs-progs: fix parallel build" does not help > much I''ll drop it. > > > david > >-- Best Reagrds, Roy | RongQing Li -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Rongqing Li
2013-Sep-23 07:17 UTC
Re: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue
On 09/18/2013 09:51 AM, Eric Sandeen wrote:> On 9/17/13 8:11 PM, rongqing.li@windriver.com wrote: >> From: Roy Li <rongqing.li@windriver.com> >> >> The dependencies of "all: version.h" or other similar ones can not >> fix the parallel build failure, only reduce the times; In fact, >> many *.o files require version.h file. >> >> #grep ''#include "version.h"'' ./ -r >> ./btrfs-corrupt-block.c:#include "version.h" >> ./btrfs.c:#include "version.h" >> ./btrfs-image.c:#include "version.h" >> ./cmds-filesystem.c:#include "version.h" >> ./btrfs-show-super.c:#include "version.h" >> ./btrfs-select-super.c:#include "version.h" >> ./cmds-restore.c:#include "version.h" >> ./btrfs-find-root.c:#include "version.h" >> ./mkfs.c:#include "version.h" >> ./btrfs-zero-log.c:#include "version.h" >> ./btrfs-defrag.c:#include "version.h" >> ./cmds-chunk.c:#include "version.h" >> ./btrfstune.c:#include "version.h" >> ./btrfs-calc-size.c:#include "version.h" >> ./btrfs-map-logical.c:#include "version.h" >> ./cmds-check.c:#include "version.h" >> ./btrfs-debug-tree.c:#include "version.h" >> >> Signed-off-by: Roy Li <rongqing.li@windriver.com> >> --- >> Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3 >> can not fix the build failure, when build enough times on a 16 core cpu, >> the build failure happens again, so I refix it again. >> >> >> Makefile | 6 ++++++ >> 1 file changed, 6 insertions(+) >> >> diff --git a/Makefile b/Makefile >> index c43cb68..a7c259c 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -81,6 +81,12 @@ endif >> @echo " [CC] $@" >> $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $< >> >> +btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h >> +btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h >> +btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h >> +btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h >> +btrfs-debug-tree.o:version.h >> + >> %.static.o: %.c >> @echo " [CC] $@" >> $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ >> > > I think this can be done more cleanly, I''ll send a patch. > > -Eric > >If this one is not cleanly, how is the below one From a56ac083a789605904507b602d9ebf196ae6746d Mon Sep 17 00:00:00 2001 From: Roy Li <rongqing.li@windriver.com> Date: Mon, 23 Sep 2013 09:53:30 +0800 Subject: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file. #grep ''#include "version.h"'' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h" Signed-off-by: Roy Li <rongqing.li@windriver.com> --- Makefile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Makefile b/Makefile index c43cb68..901ffa2 100644 --- a/Makefile +++ b/Makefile @@ -81,6 +81,8 @@ endif @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $< +%.o: version.h + %.static.o: %.c @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ -- 1.7.10.4 -- Best Reagrds, Roy | RongQing Li -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
David Sterba
2013-Sep-23 12:26 UTC
Re: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel build issue
On Sun, Sep 22, 2013 at 09:06:39AM +0800, Rongqing Li wrote:> I want to know how many cores your cpu has? I can not reproduce it on > my 2 cores cpu, but it always happens when run on a server which is > a 16 cores cpu and "make -j20"Depends on what you call a core, I''ve tested this on a box with 8 cpu cores and 64 logical cpus. I can clearly see that the build is stalled for a few moments at ''[SH] version.h'' before it proceeds to ''[CC] ...''.> >The dependency files are generated by an implicit rule .c -> .o.d, so > >there should be no problem for any of the files listed above. > > Do you means the below: > > .c.o: > $(Q)$(check) $< > @echo " [CC] $@" > $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<Rather %.o.d: %.c $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< the .o.d files are included at the end of Makefile, I''m not completely sure that they get included before the .c.o rule is processed. That way the .o.d files would be empty and the dependency on version.h missing. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html