A longstanding project on the LLVM "open projects" page is to run random C code through random LLVM passes. We have found many LLVM bugs by doing the first of these, but not the second: we test only -O[0123s]. Slowly but surely, LLVM is evolving resistance to our random tester. Of course this is good, but it means that we need to keep expanding our efforts if we want to continue to get solid return on investment for our bugfinding hours. My first question is: When we start running random combinations of passes using opt, which ones should we leave out, because you folks don't want to hear that they're wrong? For example, the other day I learned that ABCD is not yet ready for this sort of testing. A while ago someone on the list said something similar about predsimplify (though it looks like that pass has disappeared from the tree?). Also I'd appreciate some advice about probabilities for each pass. For example, it seems like a lot of passes will have little to do if Mem2Reg doesn't run, so perhaps that pass should get >0.5 probability of running. Does that make sense and do other examples come to mind? Frankly I don't expect a zillion bugs to come out of this but it is super easy and you asked for it :). Thanks, John Regehr
On 2009-11-19 22:16, John Regehr wrote:> A longstanding project on the LLVM "open projects" page is to run random > C code through random LLVM passes. > > We have found many LLVM bugs by doing the first of these, but not the > second: we test only -O[0123s]. > > Slowly but surely, LLVM is evolving resistance to our random tester. Of > course this is good, but it means that we need to keep expanding our > efforts if we want to continue to get solid return on investment for our > bugfinding hours. > > My first question is: When we start running random combinations of > passes using opt, which ones should we leave out, because you folks > don't want to hear that they're wrong? For example, the other day I > learned that ABCD is not yet ready for this sort of testing. A while > ago someone on the list said something similar about predsimplify > (though it looks like that pass has disappeared from the tree?). >I think that if a pass is not part of -O[0123s] or -std-compile-opts, or -std-link-opts then it is not ready for random testing.> Also I'd appreciate some advice about probabilities for each pass. For > example, it seems like a lot of passes will have little to do if Mem2Reg > doesn't run, so perhaps that pass should get >0.5 probability of > running. Does that make sense and do other examples come to mind? >You should have loops and nested loops (in some tests), when you want to test loop optimizers. You should have some tests with EH, some without. Also I think you should also try to test LLVM IR features that are less often used: byval, weak linkage, ssp, structs with unusual layout/alignment, vector operations, multiple return values, indirectbr, atomic ops, lifetime/invariant annotations, etc. Best regards, --Edwin
Hi John,> My first question is: When we start running random combinations of > passes using opt, which ones should we leave out, because you folks > don't want to hear that they're wrong? For example, the other day I > learned that ABCD is not yet ready for this sort of testing. A while > ago someone on the list said something similar about predsimplify > (though it looks like that pass has disappeared from the tree?).check out include/llvm/Support/StandardPasses.h. All of the passes scheduled there need to be solid because they are really being used.> Also I'd appreciate some advice about probabilities for each pass. For > example, it seems like a lot of passes will have little to do if Mem2Reg > doesn't run, so perhaps that pass should get >0.5 probability of > running. Does that make sense and do other examples come to mind?You could start with the list of passes you get at -O3, then randomly alter it, shifting the pass order, dropping passes and introducing more. Each run might take too long though due to scheduling many passes.> Frankly I don't expect a zillion bugs to come out of this but it is > super easy and you asked for it :).Well, we will soon know! Thanks for doing this, Duncan.
Thanks Duncan and Edwin-- looking at the passes used everyday sounds like the right approach.> Also I think you should also try to test LLVM IR features that are less > often used: > byval, weak linkage, ssp, structs with unusual layout/alignment, vector > operations, > multiple return values, indirectbr, atomic ops, lifetime/invariant > annotations, etc.Well, we are actually testing C compilers, not specifically LLVM, so we're not going to generate anything that clang or llvm-gcc doesn't use already. We do generate packed structs, bitfields, etc. in an attempt to stress some of these things. John Regehr