Lustre discuss - Jan 2011 - Lustre community build server

Hi Aurelien, Robert  - 

We also use Hudson and are interested in using it to do Lustre builds and
testing.
> Hi
> 
> Robert Read a ?crit :
> > Hi Aur?lien,
> >
> > Yes, we''ve noticed Hudson''s support for testing is
not quite what we need, so
> > we''re planning to use Hudson to trigger our testing system,
but not
> > necessarily to manage it.  We''d definitely be interested in
learning more
> > about your experiences, though. 
> >   
> I do not know what you mean by triggering your testing system. But here 
> is what I set up.
> Hudson has only 1 slave node dedicated to testing Lustre 2.
> Hudson will launch a shell script through ssh to it.
> 
> This script:
>  - retrieves Lustre source (managed by Hudson git plugin)
>  - compiles it.
>  - launches acceptance-small with several parameters.
>  - acceptance-small will connect to other nodes dedicated for these tests.
> 
> acc-sm have been patched:
> - to be more error resilient (does not stop at first failure)
> - to generate a test report in JUNIT format.
Is this the yaml acc-sm that Robert was talking about, or an older
one?
> 
> Hudson fetch the junit report and parse it thanks to its plugin.
> Hudson can display in its interface all tests successes and failures.
> 
> Everything goes fine as long as:
>  - the testsuite leaves the node in a good shape. It is difficult to 
> have a automatic way to put the node back. Currently, we need to manualy 
> fix that.
Would it be helpful to run the test in a VM?  Hudson has a libvirt-slave plugin
that
seems like it can start and stop a VM for you.  Another point I like about
VM''s is
that they can be suspended and shipped to an investigator for local debugging.
>  - Hudson does not know about the other nodes used by acc-sm. And so can 
> trigger tests even if some sattelites nodes are unavailable.
Don''t know if libvirt-slave can handle multiple
VM''s for a multi-node Lustre test, but maybe it can be
extended.
> 
> How is you do this on your side?
It seems that like you, we are more interested in reporting the results
within Hudson as opposed to a different custom tool.

Nathan
> 
> 
> Aur?lien
> 
> > robert
> >
> >
> >
> >
> > On Dec 16, 2010, at 1:22 , DEGREMONT Aurelien wrote:
> >
> >   
> >> Hi Robert,
> >>
> >> That''s very interesting.
> >> At CEA we also have a Hudson platform and I''m running
acceptance-small for
> >> several Lustre branches in it. Hudson is a great tool but it was
not design
> >> to test tools that run kernel-space that can crash your nodes or,
at least,
> >> put your kernel in a bad shape. I will be very interested to share
Hudson
> >> experience testing Lustre and see how you''ve configured
it for your own
> >> tests.
> >>
> >>
> >> Aur?lien
> >>
> >> Robert Read a ?crit :
> >>     
> >>> Hi,
> >>>
> >>> As I mentioned the other day, Whamcloud is hosting a Hudson
build server
> >>> and producing snapshot builds for Centos 5.x (and Ubuntu 10.4
when it
> >>> works) for both 1.8.x and 2.x branches. Our intention is for
this to be a
> >>> resource for the Lustre community to find recent Lustre
packages for
> >>> variety of Linux distributions. Early next year we''ll
connect this to our
> >>> test system so at least some of the packages can be tested, as
well.
> >>>
> >>> We would be interested in hearing from anyone that would like
to
> >>> participate producing builds. Hudson is an distributed system,
and it''s
> >>> easy to add more build nodes, even behind firewalls (some of
us are running
> >>> build VMs on our home machines). If you would like add another
distribution
> >>> or architecture we don''t have yet, or even one we do
have (the more the
> >>> merrier), we''d be happy to work with you to do that. 
Please contact me if
> >>> you are interested.  
> >>> cheers,
> >>> robert
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Lustre-discuss mailing list
> >>> Lustre-discuss at lists.lustre.org
> >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >>>  
> >>>       
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20110119/015cfefb/attachment.html

Hi

Nathan Rutman a ?crit :
> Hi Aurelien, Robert  - 
>
> We also use Hudson and are interested in using it to do Lustre builds 
> and testing.
>> Hi
>>
>> Robert Read a ?crit :
>> > Hi Aur?lien,
>> >
>> > Yes, we''ve noticed Hudson''s support for testing
is not quite what we need, so
>> > we''re planning to use Hudson to trigger our testing
system, but not
>> > necessarily to manage it.  We''d definitely be interested
in learning more
>> > about your experiences, though. 
>> >   
>> I do not know what you mean by triggering your testing system. But here
>> is what I set up.
>> Hudson has only 1 slave node dedicated to testing Lustre 2.
>> Hudson will launch a shell script through ssh to it.
>>
>> This script:
>>  - retrieves Lustre source (managed by Hudson git plugin)
>>  - compiles it.
>>  - launches acceptance-small with several parameters.
>>  - acceptance-small will connect to other nodes dedicated for these
tests.
>>
>> acc-sm have been patched:
>> - to be more error resilient (does not stop at first failure)
>> - to generate a test report in JUNIT format.
>>     
> Is this the yaml acc-sm that Robert was talking about, or an older one?
I think so.
We modified the current test-framework.sh  and acceptance-small.sh, from 
master.
We reused the call introduced to produce a result.yml script to produce 
a junit-report.xml script
>> Hudson fetch the junit report and parse it thanks to its plugin.
>> Hudson can display in its interface all tests successes and failures.
>>
>> Everything goes fine as long as:
>>  - the testsuite leaves the node in a good shape. It is difficult to 
>> have a automatic way to put the node back. Currently, we need to
manualy
>> fix that.
>>     
> Would it be helpful to run the test in a VM?  Hudson has a 
> libvirt-slave plugin that
> seems like it can start and stop a VM for you.  Another point I like 
> about VM''s is
> that they can be suspended and shipped to an investigator for local 
> debugging.
I notice this plugin for Hudson and also had the same analysis.
I did not try to setup it yet, but this as some side effects anyway and 
won''t be a perfect solution for sure.

I find your idea to send the VM interesting. Not sure it could be done 
easily, but could be a great help for debugging if this is
doable.
>>  - Hudson does not know about the other nodes used by acc-sm. And so
can
>> trigger tests even if some sattelites nodes are unavailable.
>>     
> Don''t know if libvirt-slave can handle multiple
> VM''s for a multi-node Lustre test, but maybe it can be extended.
Even without testing Lustre I will be very pleased if Hudson can 
allocate several nodes for 1 run and give the node list to the test 
script through variable by example.
This could be very interesting for us, as we are working on clustering 
stuff, we always need group of nodes whatever the tests are. This is not 
easy to handle in Hudson. We have to cheat :p
>> How is you do this on your side?
>>     
> It seems that like you, we are more interested in reporting the results
> within Hudson as opposed to a different custom tool.
In this case, I''m quite sure we can converge toward the same patch for
that.

Robert''s ideas are more ambitious and want to handle more generic uses.
I will not be displeased by that as long as the test results could be 
analyzed by Hudson plugins.



Aurelien