Matthijs Kooijman
2008-May-21 13:09 UTC
[LLVMdev] MultiSource/Applications/lemon slow in JIT
Hi, I've been toying around with the testsuite for a bit, and after recompiling llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually got it to run and pass most tests. When running, I noticed a very long runtime on the lemon test. At first I suspected an infinite loop, but it turned out the test simply needed around 1000 seconds to run. Some investigation turned out that the lemon program does a lot of forking (it runs every input file 20 times, so that's around 200 forks). However, since the forking happens quite early in the main function, it happens before most of the program is JITted. So, every one of these forked process is taking a few seconds to compile the other needed functions, over and over again. A solution for this problem seems to be to run the lemon test with the -no-lazy option, so things get compiled before forking. In my case, this reduced runtime from 1000+ seconds to only 16 seconds. Intrestingly enough, I can't seem to find similar behaviour in the test results on llvm.org/nightlytest (though I only checked two machines [since the results take ages to load...] of which only one actually ran the test). Am I the only one that is experiencing this? Anyhow, the below patch fixes the problem for me, good to commit? Gr. Matthijs Index: MultiSource/Applications/lemon/Makefile ==================================================================--- MultiSource/Applications/lemon/Makefile (revision 51189) +++ MultiSource/Applications/lemon/Makefile (working copy) @@ -8,5 +8,8 @@ include $(LEVEL)/Makefile.config include ../../Makefile.multisrc + +JIT_OPTS += -no-lazy + DIFFPROG := $(PROGDIR)/DiffOutput.sh "diff " -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20080521/8533d9a0/attachment.sig>
Matthijs Kooijman
2008-May-21 13:55 UTC
[LLVMdev] MultiSource/Applications/lemon slow in JIT
Hi all,> Intrestingly enough, I can't seem to find similar behaviour in the test > results on llvm.org/nightlytest (though I only checked two machines [since the > results take ages to load...] of which > only one actually ran the test). Am I the only one that is experiencing this?After finally being able to view the full test results of another machine, I noticed that grue-x86-64 also needs 142 seconds for the test, which is not as slow as here, but also way too slow... Gr. Matthijs -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20080521/42942408/attachment.sig>
Dale Johannesen
2008-May-21 17:57 UTC
[LLVMdev] MultiSource/Applications/lemon slow in JIT
On May 21, 2008, at 6:09 AM, Matthijs Kooijman wrote:> Hi, > > I've been toying around with the testsuite for a bit, and after > recompiling > llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually > got it to > run and pass most tests. > > When running, I noticed a very long runtime on the lemon test. At > first I > suspected an infinite loop, but it turned out the test simply needed > around > 1000 seconds to run. Some investigation turned out that the lemon > program does > a lot of forking (it runs every input file 20 times, so that's > around 200 > forks). However, since the forking happens quite early in the main > function, > it happens before most of the program is JITted. So, every one of > these forked > process is taking a few seconds to compile the other needed > functions, over > and over again. > > A solution for this problem seems to be to run the lemon test with the > -no-lazy option, so things get compiled before forking. In my case, > this > reduced runtime from 1000+ seconds to only 16 seconds. > > Intrestingly enough, I can't seem to find similar behaviour in the > test > results on llvm.org/nightlytest (though I only checked two machines > [since the > results take ages to load...] of which > only one actually ran the test). Am I the only one that is > experiencing this?I have noticed a timeout on lemon JIT, hadn't investigated to see whether looping or just slow, thanks! It's true it's not failing in the nightly logs, but if the number in the "Today's JIT" column has anything to do with runtime (another Feature of that display is that it nowhere explains what any of the numbers mean), it is taking a very long time, about 6x as long as any other Multisource JIT run. Is the timeout disabled in those runs somehow? I don't know anything about the test, but your patch works for me.> Anyhow, the below patch fixes the problem for me, good to commit? > > Gr. > > Matthijs > > > Index: MultiSource/Applications/lemon/Makefile > ==================================================================> --- MultiSource/Applications/lemon/Makefile (revision 51189) > +++ MultiSource/Applications/lemon/Makefile (working copy) > @@ -8,5 +8,8 @@ > include $(LEVEL)/Makefile.config > > include ../../Makefile.multisrc > + > +JIT_OPTS += -no-lazy > + > DIFFPROG := $(PROGDIR)/DiffOutput.sh "diff " > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On May 21, 2008, at 6:09 AM, Matthijs Kooijman wrote:> Hi, > > I've been toying around with the testsuite for a bit, and after > recompiling > llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually > got it to > run and pass most tests. > > When running, I noticed a very long runtime on the lemon test. At > first I > suspected an infinite loop, but it turned out the test simply needed > around > 1000 seconds to run. Some investigation turned out that the lemon > program does > a lot of forking (it runs every input file 20 times, so that's > around 200 > forks). However, since the forking happens quite early in the main > function, > it happens before most of the program is JITted. So, every one of > these forked > process is taking a few seconds to compile the other needed > functions, over > and over again. > > A solution for this problem seems to be to run the lemon test with the > -no-lazy option, so things get compiled before forking. In my case, > this > reduced runtime from 1000+ seconds to only 16 seconds. > > Intrestingly enough, I can't seem to find similar behaviour in the > test > results on llvm.org/nightlytest (though I only checked two machines > [since the > results take ages to load...] of which > only one actually ran the test). Am I the only one that is > experiencing this? > > Anyhow, the below patch fixes the problem for me, good to commit?Ouch. I didn't notice this. I don't think passing this option is the right fix though. Any test that forks a lot is simply not a good compiler benchmark. The right thing to do is to eliminate the forking. Is that cannot be easily done. We should remove it from our test suite. Thanks, Evan> > > Gr. > > Matthijs > > > Index: MultiSource/Applications/lemon/Makefile > ==================================================================> --- MultiSource/Applications/lemon/Makefile (revision 51189) > +++ MultiSource/Applications/lemon/Makefile (working copy) > @@ -8,5 +8,8 @@ > include $(LEVEL)/Makefile.config > > include ../../Makefile.multisrc > + > +JIT_OPTS += -no-lazy > + > DIFFPROG := $(PROGDIR)/DiffOutput.sh "diff " > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Evan Cheng wrote:> On May 21, 2008, at 6:09 AM, Matthijs Kooijman wrote: > > >> Hi, >> >> I've been toying around with the testsuite for a bit, and after >> recompiling >> llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually >> got it to >> run and pass most tests. >> >> When running, I noticed a very long runtime on the lemon test. At >> first I >> suspected an infinite loop, but it turned out the test simply needed >> around >> 1000 seconds to run. Some investigation turned out that the lemon >> program does >> a lot of forking (it runs every input file 20 times, so that's >> around 200 >> forks). However, since the forking happens quite early in the main >> function, >> it happens before most of the program is JITted. So, every one of >> these forked >> process is taking a few seconds to compile the other needed >> functions, over >> and over again. >> >> A solution for this problem seems to be to run the lemon test with the >> -no-lazy option, so things get compiled before forking. In my case, >> this >> reduced runtime from 1000+ seconds to only 16 seconds. >> >> Intrestingly enough, I can't seem to find similar behaviour in the >> test >> results on llvm.org/nightlytest (though I only checked two machines >> [since the >> results take ages to load...] of which >> only one actually ran the test). Am I the only one that is >> experiencing this? >> >> Anyhow, the below patch fixes the problem for me, good to commit? >> > > Ouch. I didn't notice this. I don't think passing this option is the > right fix though. Any test that forks a lot is simply not a good > compiler benchmark. The right thing to do is to eliminate the forking. > Is that cannot be easily done. We should remove it from our test suite. >The forking was introduced because the runtime was too small; the program keeps global state, so there is no easy way to reset state. I would prefer to remove the forking (and get short runtimes again), rather than removing it completely from the testsuite. What do you think? Best regards. --Edwin