Hi all, Looking into ClamAV, it seems that the output is highly asynchronous, so a simple diff is obviously bogus. Since the output is mainly a report of what's happening, without any logical sequence, I can sort both outputs and diff them, for a similar (though not perfect) result. If you sort clamscan.out-nat and clamscan.out-simple, you'll see that they're identical. I'm not sure if there is an option to run the program in sequential mode (if there is, the test will pass), but if not, the simplest way to fix the diff is to sort the reference_output and the program's output on execution. Adding "| sort" to the RUN_OPTIONS didn't have the desired effect (that was a long shot, anyway), does anyone know of a way to sort the output of a program on those Makefiles? Maybe a rule that can be intermediary of the final rule (I'm guessing $PROGRAM) that I can add a few steps before the diff? cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130402/829bf126/attachment.html>
Hi Renato, What is it that makes the output of the program asynchronous? The output is deterministic on Darwin, so it seems like it should be possible to make it more stable. I'm not really a fan of doing any more transformations on the output than we have to, because it just pushes complexity into the test suite infrastructure. - Daniel On Tue, Apr 2, 2013 at 10:05 AM, Renato Golin <renato.golin at linaro.org>wrote:> Hi all, > > Looking into ClamAV, it seems that the output is highly asynchronous, so a > simple diff is obviously bogus. Since the output is mainly a report of > what's happening, without any logical sequence, I can sort both outputs and > diff them, for a similar (though not perfect) result. If you sort > clamscan.out-nat and clamscan.out-simple, you'll see that they're identical. > > I'm not sure if there is an option to run the program in sequential mode > (if there is, the test will pass), but if not, the simplest way to fix the > diff is to sort the reference_output and the program's output on execution. > > Adding "| sort" to the RUN_OPTIONS didn't have the desired effect (that > was a long shot, anyway), does anyone know of a way to sort the output of a > program on those Makefiles? Maybe a rule that can be intermediary of the > final rule (I'm guessing $PROGRAM) that I can add a few steps before the > diff? > > cheers, > --renato >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130402/9bbb1e99/attachment.html>
On 2 April 2013 19:20, Daniel Dunbar <daniel at zuster.org> wrote:> What is it that makes the output of the program asynchronous? The output > is deterministic on Darwin, so it seems like it should be possible to make > it more stable. >This is a virus scan and, AFAICS, depends on the order in which the INODEs are laid out in the directory. I'm not sure there is a way to sort the files before, I'll look into that.> I'm not really a fan of doing any more transformations on the output than > we have to, because it just pushes complexity into the test suite > infrastructure. >I agree, that is the last resort. cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130402/72090793/attachment.html>