Hi, As I mentioned the other day, Whamcloud is hosting a Hudson build server and producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for both 1.8.x and 2.x branches. Our intention is for this to be a resource for the Lustre community to find recent Lustre packages for variety of Linux distributions. Early next year we''ll connect this to our test system so at least some of the packages can be tested, as well. We would be interested in hearing from anyone that would like to participate producing builds. Hudson is an distributed system, and it''s easy to add more build nodes, even behind firewalls (some of us are running build VMs on our home machines). If you would like add another distribution or architecture we don''t have yet, or even one we do have (the more the merrier), we''d be happy to work with you to do that. Please contact me if you are interested. cheers, robert
Hi Robert, That''s very interesting. At CEA we also have a Hudson platform and I''m running acceptance-small for several Lustre branches in it. Hudson is a great tool but it was not design to test tools that run kernel-space that can crash your nodes or, at least, put your kernel in a bad shape. I will be very interested to share Hudson experience testing Lustre and see how you''ve configured it for your own tests. Aur?lien Robert Read a ?crit :> Hi, > > As I mentioned the other day, Whamcloud is hosting a Hudson build server and producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for both 1.8.x and 2.x branches. Our intention is for this to be a resource for the Lustre community to find recent Lustre packages for variety of Linux distributions. Early next year we''ll connect this to our test system so at least some of the packages can be tested, as well. > > We would be interested in hearing from anyone that would like to participate producing builds. Hudson is an distributed system, and it''s easy to add more build nodes, even behind firewalls (some of us are running build VMs on our home machines). If you would like add another distribution or architecture we don''t have yet, or even one we do have (the more the merrier), we''d be happy to work with you to do that. Please contact me if you are interested. > > cheers, > robert > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Hi Aur?lien, Yes, we''ve noticed Hudson''s support for testing is not quite what we need, so we''re planning to use Hudson to trigger our testing system, but not necessarily to manage it. We''d definitely be interested in learning more about your experiences, though. robert On Dec 16, 2010, at 1:22 , DEGREMONT Aurelien wrote:> Hi Robert, > > That''s very interesting. > At CEA we also have a Hudson platform and I''m running acceptance-small for several Lustre branches in it. Hudson is a great tool but it was not design to test tools that run kernel-space that can crash your nodes or, at least, put your kernel in a bad shape. I will be very interested to share Hudson experience testing Lustre and see how you''ve configured it for your own tests. > > > Aur?lien > > Robert Read a ?crit : >> Hi, >> >> As I mentioned the other day, Whamcloud is hosting a Hudson build server and producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for both 1.8.x and 2.x branches. Our intention is for this to be a resource for the Lustre community to find recent Lustre packages for variety of Linux distributions. Early next year we''ll connect this to our test system so at least some of the packages can be tested, as well. >> >> We would be interested in hearing from anyone that would like to participate producing builds. Hudson is an distributed system, and it''s easy to add more build nodes, even behind firewalls (some of us are running build VMs on our home machines). If you would like add another distribution or architecture we don''t have yet, or even one we do have (the more the merrier), we''d be happy to work with you to do that. Please contact me if you are interested. >> cheers, >> robert >> >> >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >
Hi Robert Read a ?crit :> Hi Aur?lien, > > Yes, we''ve noticed Hudson''s support for testing is not quite what we need, so we''re planning to use Hudson to trigger our testing system, but not necessarily to manage it. We''d definitely be interested in learning more about your experiences, though. >I do not know what you mean by triggering your testing system. But here is what I set up. Hudson has only 1 slave node dedicated to testing Lustre 2. Hudson will launch a shell script through ssh to it. This script: - retrieves Lustre source (managed by Hudson git plugin) - compiles it. - launches acceptance-small with several parameters. - acceptance-small will connect to other nodes dedicated for these tests. acc-sm have been patched: - to be more error resilient (does not stop at first failure) - to generate a test report in JUNIT format. Hudson fetch the junit report and parse it thanks to its plugin. Hudson can display in its interface all tests successes and failures. Everything goes fine as long as: - the testsuite leaves the node in a good shape. It is difficult to have a automatic way to put the node back. Currently, we need to manualy fix that. - Hudson does not know about the other nodes used by acc-sm. And so can trigger tests even if some sattelites nodes are unavailable. How is you do this on your side? Aur?lien> robert > > > > > On Dec 16, 2010, at 1:22 , DEGREMONT Aurelien wrote: > > >> Hi Robert, >> >> That''s very interesting. >> At CEA we also have a Hudson platform and I''m running acceptance-small for several Lustre branches in it. Hudson is a great tool but it was not design to test tools that run kernel-space that can crash your nodes or, at least, put your kernel in a bad shape. I will be very interested to share Hudson experience testing Lustre and see how you''ve configured it for your own tests. >> >> >> Aur?lien >> >> Robert Read a ?crit : >> >>> Hi, >>> >>> As I mentioned the other day, Whamcloud is hosting a Hudson build server and producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for both 1.8.x and 2.x branches. Our intention is for this to be a resource for the Lustre community to find recent Lustre packages for variety of Linux distributions. Early next year we''ll connect this to our test system so at least some of the packages can be tested, as well. >>> >>> We would be interested in hearing from anyone that would like to participate producing builds. Hudson is an distributed system, and it''s easy to add more build nodes, even behind firewalls (some of us are running build VMs on our home machines). If you would like add another distribution or architecture we don''t have yet, or even one we do have (the more the merrier), we''d be happy to work with you to do that. Please contact me if you are interested. >>> cheers, >>> robert >>> >>> >>> >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >>> > >
Hi, On Dec 16, 2010, at 9:42 , DEGREMONT Aurelien wrote:> Hi > > Robert Read a ?crit : >> Hi Aur?lien, >> >> Yes, we''ve noticed Hudson''s support for testing is not quite what we need, so we''re planning to use Hudson to trigger our testing system, but not necessarily to manage it. We''d definitely be interested in learning more about your experiences, though. > I do not know what you mean by triggering your testing system. But here is what I set up.I mean that once the build is complete we will notify the test system that new build is ready to be picked up and tested. We haven''t yet implemented that part of it.> Hudson has only 1 slave node dedicated to testing Lustre 2. > Hudson will launch a shell script through ssh to it. > > This script: > - retrieves Lustre source (managed by Hudson git plugin) > - compiles it. > - launches acceptance-small with several parameters. > - acceptance-small will connect to other nodes dedicated for these tests. > > acc-sm have been patched: > - to be more error resilient (does not stop at first failure) > - to generate a test report in JUNIT format. > > Hudson fetch the junit report and parse it thanks to its plugin. > Hudson can display in its interface all tests successes and failures. > > Everything goes fine as long as: > - the testsuite leaves the node in a good shape. It is difficult to have a automatic way to put the node back. Currently, we need to manualy fix that. > - Hudson does not know about the other nodes used by acc-sm. And so can trigger tests even if some sattelites nodes are unavailable. > > How is you do this on your side?We don''t plan to use Hudson to manage our testing results as I don''t think it would scale very well for all the testing we might do for each build. We''re currently building a more custom results server that''s similar (in spirit at least) to the kinds of tools we had at Oracle. We''ll make it available once it''s in presentable form. Actually, our first step was to replace the acceptance-small.sh driver script with one that has a more sensible user interface for running the standard tests. Since the test-framework.sh on master has already been changed to produce test results in yaml format, the new script collects these with the logs, and is capable of submitting them to the test results server. Currently this is being run manually, though. Automating the test execution and connecting all the pieces will be next step. cheers, robert> > > Aur?lien > >> robert >> >> >> >> >> On Dec 16, 2010, at 1:22 , DEGREMONT Aurelien wrote: >> >> >>> Hi Robert, >>> >>> That''s very interesting. >>> At CEA we also have a Hudson platform and I''m running acceptance-small for several Lustre branches in it. Hudson is a great tool but it was not design to test tools that run kernel-space that can crash your nodes or, at least, put your kernel in a bad shape. I will be very interested to share Hudson experience testing Lustre and see how you''ve configured it for your own tests. >>> >>> >>> Aur?lien >>> >>> Robert Read a ?crit : >>> >>>> Hi, >>>> >>>> As I mentioned the other day, Whamcloud is hosting a Hudson build server and producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for both 1.8.x and 2.x branches. Our intention is for this to be a resource for the Lustre community to find recent Lustre packages for variety of Linux distributions. Early next year we''ll connect this to our test system so at least some of the packages can be tested, as well. >>>> >>>> We would be interested in hearing from anyone that would like to participate producing builds. Hudson is an distributed system, and it''s easy to add more build nodes, even behind firewalls (some of us are running build VMs on our home machines). If you would like add another distribution or architecture we don''t have yet, or even one we do have (the more the merrier), we''d be happy to work with you to do that. Please contact me if you are interested. cheers, >>>> robert >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>> >> >> >
Robert Read a ?crit :> We don''t plan to use Hudson to manage our testing results as I don''t think it would scale very well for all the testing we might do for each build. We''re currently building a more custom results server that''s similar (in spirit at least) to the kinds of tools we had at Oracle. We''ll make it available once it''s in presentable form. > > Actually, our first step was to replace the acceptance-small.sh driver script with one that has a more sensible user interface for running the standard tests. Since the test-framework.sh on master has already been changed to produce test results in yaml format, the new script collects these with the logs, and is capable of submitting them to the test results server. Currently this is being run manually, though. Automating the test execution and connecting all the pieces will be next step. >Ok. I will be very interested in seeing the final result. But I think it is a good idea to stick to standard format and tools as much as possible. This could be a pity if all your new work will be only usable by your tool. Junit is quite standard. PerfPublisher has its own format due to junit limitations. There is other ones. It could be really good if you do not create a new one. And indeed, acc-sm is a bit limited and improve it could be really interesting. Aur?lien
Hi Aur?lien, Do you have a specification for the junit test results you produce, or an example of one of your test results sets. We would be more than willing to pick up and go with something that can be used with a wider set of tools, with the obvious caveat that it provides everything needed to completely capture the test results for Lustre today and in the future. If you have some example results set''s that you can forward please mail them to chris whamcloud.com Thanks Chris I see that PerfPublisher uses xml, although this seems to be the only specification. On 17/12/2010 20:11, Aur?lien wrote:>> Robert Read a ?crit : >>> We don''t plan to use Hudson to manage our testing results as I don''t >>> think it would scale very well for all the testing we might do for >>> each build. We''re currently building a more custom results server >>> that''s similar (in spirit at least) to the kinds of tools we had at >>> Oracle. We''ll make it available once it''s in presentable form. >>> Actually, our first step was to replace the acceptance-small.sh >>> driver script with one that has a more sensible user interface for >>> running the standard tests. Since the test-framework.sh on master >>> has already been changed to produce test results in yaml format, >>> the new script collects these with the logs, and is capable of >>> submitting them to the test results server. Currently this is >>> being run manually, though. Automating the test execution and >>> connecting all the pieces will be next step. >> Ok. I will be very interested in seeing the final result. >> But I think it is a good idea to stick to standard format and tools >> as much as possible. This could be a pity if all your new work will >> be only usable by your tool. >> >> Junit is quite standard. >> PerfPublisher has its own format due to junit limitations. There is >> other ones. It could be really good if you do not create a new one. >> >> And indeed, acc-sm is a bit limited and improve it could be really >> interesting. >> >> >> Aur?lien > >
Hi Chris, Here is a rough example of junit report we produce. This was a quick and dirty implementation of Junit that could be improved. Some part of junit report content was limited due to lack of information acceptance small upcalls have. <?xml version="1.0" encoding="UTF-8" ?> <testsuite name="Lustre.acceptance-small"> <testcase classname="Lustre.sanity" name="Test #0: touch .../ ; rm .../" time="1" /> <testcase classname="Lustre.sanity" name="Test #0b: chmod 0755 /mnt/lustre" time="1" /> <testcase classname="Lustre.sanity" name="Test #0c: check import proc" time="1" /> <testcase classname="Lustre.sanity" name="Test #1a: mkdir .../d1; mkdir .../d1/d2" time="1" /> <testcase classname="Lustre.sanity" name="Test #1b: rmdir .../d1/d2; rmdir .../d1" time="1" /> <testcase classname="Lustre.sanity" name="Test #2a: mkdir .../d2; touch .../d2/f" time="1" /> <testcase classname="Lustre.sanity" name="Test #2b: rm -r .../d2; checkstat .../d2/f" time="1" /> <testcase classname="Lustre.sanity" name="Test #3a: mkdir .../d3" time="1" /> <testcase classname="Lustre.sanity" name="Test #3b: touch .../d3/f" time="1" /> <testcase classname="Lustre.sanity" name="Test #3c: rm -r .../d3" time="1" /> <testcase classname="Lustre.sanity" name="Test #4a: mkdir .../d4" time="1" /> <testcase classname="Lustre.sanity" name="Test #4b: mkdir .../d4/d2" time="1" /> ... <testcase classname="Lustre.sanity" name="Test #180a: test obdecho on osc" time="80"> <failure type="FAIL"><![CDATA[rc=1 test_180a failed with 1]]></failure> </testcase> ... <testcase classname="Lustre.sanity-benchmark" name="Test #dbench: test_dbench" time="0"> <skipped/> </testcase> <testcase classname="Lustre.sanity-benchmark" name="Test #dbench: dbench" time="2" /> ... </testsuite> This is a imple junit report. This could be improved. Partly in improving acceptance-small.sh and test-framework.sh. I''m open to switch to any other standard format, supported by Hudson, if this can helps. What''s your needs/plan? Aur?lien Chris Gearing a ?crit :> Hi Aur?lien, > > Do you have a specification for the junit test results you produce, or > an example of one of your test results sets. We would be more than > willing to pick up and go with something that can be used with a wider > set of tools, with the obvious caveat that it provides everything needed > to completely capture the test results for Lustre today and in the future. > > If you have some example results set''s that you can forward please mail > them to chris whamcloud.com > > Thanks > > Chris > > I see that PerfPublisher uses xml, although this seems to be the only > specification. > > On 17/12/2010 20:11, Aur?lien wrote: > >>> Robert Read a ?crit : >>> >>>> We don''t plan to use Hudson to manage our testing results as I don''t >>>> think it would scale very well for all the testing we might do for >>>> each build. We''re currently building a more custom results server >>>> that''s similar (in spirit at least) to the kinds of tools we had at >>>> Oracle. We''ll make it available once it''s in presentable form. >>>> Actually, our first step was to replace the acceptance-small.sh >>>> driver script with one that has a more sensible user interface for >>>> running the standard tests. Since the test-framework.sh on master >>>> has already been changed to produce test results in yaml format, >>>> the new script collects these with the logs, and is capable of >>>> submitting them to the test results server. Currently this is >>>> being run manually, though. Automating the test execution and >>>> connecting all the pieces will be next step. >>>> >>> Ok. I will be very interested in seeing the final result. >>> But I think it is a good idea to stick to standard format and tools >>> as much as possible. This could be a pity if all your new work will >>> be only usable by your tool. >>> >>> Junit is quite standard. >>> PerfPublisher has its own format due to junit limitations. There is >>> other ones. It could be really good if you do not create a new one. >>> >>> And indeed, acc-sm is a bit limited and improve it could be really >>> interesting. >>> >>> >>> Aur?lien >>> >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Hi Aurelien, That''s useful thanks. Our plan is to create a community database that enables every valuable facet of a lustre test to be stored. This means all the results, all the logs, stack traces etc etc. Sometimes we will only store failing logs, but for acceptance and release testing we will store passing and failing(!) logs. The aim is that when someone is looking into a test failure, nothing they require will have been left behind. A further use of this approach is that we hope to be able to deliver landing collateral to reviewers as a standard part of our process. For example a patch could have a link to an example of the failing test that isolated the bug or missing feature, the code that fixes the failure or adds the feature and an example of the passing test showing the bug fix or feature addition working. This additional usage is not entirely near term, but is a direction that would add great value. As for yml vs. xml, the reality is that we use the yml as a way of transporting the results from the test machine, which could be a test-cluster or an individuals laptop running VMs, to the database. If xml made this data more widely usable and the requirements of junit do not require us to leave information out I can''t really see a problem in switching to it. Chris On 20/12/2010 14:50, DEGREMONT Aurelien wrote:> Hi Chris, > > Here is a rough example of junit report we produce. This was a quick > and dirty implementation of Junit that could be improved. Some part of > junit report content was limited due to lack of information acceptance > small upcalls have. > > <?xml version="1.0" encoding="UTF-8" ?> > <testsuite name="Lustre.acceptance-small"> > <testcase classname="Lustre.sanity" name="Test #0: touch .../ ; rm > .../" time="1" /> > <testcase classname="Lustre.sanity" name="Test #0b: chmod 0755 > /mnt/lustre" time="1" /> > <testcase classname="Lustre.sanity" name="Test #0c: check import proc" > time="1" /> > <testcase classname="Lustre.sanity" name="Test #1a: mkdir .../d1; > mkdir .../d1/d2" time="1" /> > <testcase classname="Lustre.sanity" name="Test #1b: rmdir .../d1/d2; > rmdir .../d1" time="1" /> > <testcase classname="Lustre.sanity" name="Test #2a: mkdir .../d2; > touch .../d2/f" time="1" /> > <testcase classname="Lustre.sanity" name="Test #2b: rm -r .../d2; > checkstat .../d2/f" time="1" /> > <testcase classname="Lustre.sanity" name="Test #3a: mkdir .../d3" > time="1" /> > <testcase classname="Lustre.sanity" name="Test #3b: touch .../d3/f" > time="1" /> > <testcase classname="Lustre.sanity" name="Test #3c: rm -r .../d3" > time="1" /> > <testcase classname="Lustre.sanity" name="Test #4a: mkdir .../d4" > time="1" /> > <testcase classname="Lustre.sanity" name="Test #4b: mkdir .../d4/d2" > time="1" /> > ... > <testcase classname="Lustre.sanity" name="Test #180a: test obdecho on > osc" time="80"> > <failure type="FAIL"><![CDATA[rc=1 > test_180a failed with 1]]></failure> > </testcase> > ... > <testcase classname="Lustre.sanity-benchmark" name="Test #dbench: > test_dbench" time="0"> > <skipped/> > </testcase> > <testcase classname="Lustre.sanity-benchmark" name="Test #dbench: > dbench" time="2" /> > ... > </testsuite> > > This is a imple junit report. This could be improved. Partly in > improving acceptance-small.sh and test-framework.sh. > I''m open to switch to any other standard format, supported by Hudson, > if this can helps. > > What''s your needs/plan? > > Aur?lien > > Chris Gearing a ?crit : >> Hi Aur?lien, >> >> Do you have a specification for the junit test results you produce, >> or an example of one of your test results sets. We would be more than >> willing to pick up and go with something that can be used with a >> wider set of tools, with the obvious caveat that it provides >> everything needed to completely capture the test results for Lustre >> today and in the future. >> >> If you have some example results set''s that you can forward please >> mail them to chris whamcloud.com >> >> Thanks >> >> Chris >> >> I see that PerfPublisher uses xml, although this seems to be the only >> specification. >> >> On 17/12/2010 20:11, Aur?lien wrote: >>>> Robert Read a ?crit : >>>>> We don''t plan to use Hudson to manage our testing results as I >>>>> don''t think it would scale very well for all the testing we might >>>>> do for each build. We''re currently building a more custom results >>>>> server that''s similar (in spirit at least) to the kinds of tools >>>>> we had at Oracle. We''ll make it available once it''s in >>>>> presentable form. >>>>> Actually, our first step was to replace the acceptance-small.sh >>>>> driver script with one that has a more sensible user interface for >>>>> running the standard tests. Since the test-framework.sh on master >>>>> has already been changed to produce test results in yaml format, >>>>> the new script collects these with the logs, and is capable of >>>>> submitting them to the test results server. Currently this is >>>>> being run manually, though. Automating the test execution and >>>>> connecting all the pieces will be next step. >>>> Ok. I will be very interested in seeing the final result. >>>> But I think it is a good idea to stick to standard format and tools >>>> as much as possible. This could be a pity if all your new work will >>>> be only usable by your tool. >>>> >>>> Junit is quite standard. >>>> PerfPublisher has its own format due to junit limitations. There is >>>> other ones. It could be really good if you do not create a new one. >>>> >>>> And indeed, acc-sm is a bit limited and improve it could be really >>>> interesting. >>>> >>>> >>>> Aur?lien >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >