Hi all, you probably have all heard by now that the Xen Project Advisory Board (a group of vendors who provide funds to the Xen Project that are intended to be used for the good of the community) recently created the Test Framework Working Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains more information about the group. The working group had its first meeting a few weeks ago and one of the actions I had was to kick off a thread on development lists to figure out what would help the developer community. I was planning to kick off this thread with some questions and options, which reflect some discussions I had with individuals in the community, various meetings (WG and AB meetings), etc. which I condensed into a picture. This reflects my personal opinion (not a Citrix opinion) and is merely intended to get a discussion going. Feel free to pick it apart: I won’t be upset. First, I wanted to clear up a few misconceptions that I have heard from a few people: * The Advisory Board has funds that can be used to create an independently hosted test infrastructure to help the developer community. However, funds are limited. Thus, it is important that we do what is right for the Xen community in the short term and the longer term. Otherwise, we will burn funds that could be used to help the Xen community in other ways. * The Test Framework Working Group is made up of people employed by vendors who have some experience in testing. * There is no intention to prescribe a test environment that you then have to use. Advisory Board members made clear to me that they want to make sure that what we end up with a solution that works for you. * At the Xen Developer Summit two different solutions for system testing were presented. The intention was to explain what is there and what we can use going forward. A presentation on OSSTest which runs regularly today was given. And one for XenRT, for which there is a plan to get a small 3 box system up and running that can be used for you to look at. Citrix volunteered to set this up at its own cost. * Just to be clear: what works for you may be one of these, none of these, both of them, … * There may also be different answers in the short and the long run. * At the end of the day, different community members will have different views. Also the Advisory Board members who provide the funds, will also have specific interests that they will push for. Thus, in all likelihood, we will have to find a good enough compromise. * The vast majority of Advisory Board members care about the Hypervisor (and not so much about XAPI and Mirage OS). Thus, it is likely that the focus of the test system would be the Hypervisor. So let me try and condense some of the arguments and opinions I heard and information that is around. This list may be incomplete. == Work Flow = I added this section, because some members of the community and the working group had prior experience with attempts to introduce a test infrastructure for an open source community in the past, and these may not have worked as well as hoped. I made up some of the terminology. *Local testing*: the basic idea here is for a developer to write their code and be able to run some tests on their machine locally as part of their development workflow. We do have same capability to do this with OSSTest (seehttp://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/), but the drawback is that this only works on the developers machine. This may be good enough. We could “emulate” different environments using QEMU, but that is likely to be slow. *System testing*: both OSSTest and XenRT are essentially system test environments, which would be run on code that is already in the Xen mainline (or a staging branch). This is really great for regression testing on different hardware platforms, but does not match the typical workflow in the community: a) code developed locally b) code submitted to list for review c) code reviewed and then submitted by the committer to the staging branch/mainline d) only then tests are run e) if tests fail, the whole process starts again One of the issues here is that the elapsed time between development and the tests being run. Another issue we have with the current set-up is that OSSTest is currently hosted at Citrix, which makes difficult for community members to log into a machine and investigate issues. Admittedly this can fixed by having the system test hosted somewhere neutral (some sort of access control would still remain). *Test on demand:* this would be a mixture between local testing and system testing. It can probably be implemented in both OSSTest and XenRT, although I believe XenRT is a closer to this model already. The workflow would look like this: a) code developed locally b) developer checks working prototype into their personal git branch (which would need to be accessible from the web) c) developer submits a job to the test farm which checks out and builds code and runs a set of interesting (or new) tests on some machines of different architectures. New tests would probably need to be on the git branch d) if the test fail, the cycle starts again d) if all works well, code is submitted for review as normal (and test results could be attached) This would only really work, if the interface into the system is easy to use and tests are run fast enough. Access control, etc. would be similar to system test. IMHO, this would be a nice mid to long-term goal, assuming it could be made to work with the funds we have. == OSSTest = What runs now and thus easiest to get started on More Info *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/ *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/ *http://www.youtube.com/watch?v=JxTFZIwZzJ8 Problems: * Runs on Citrix premises (thus general access is an issue) * Ian Jackson is acting as sys-admin in his spare time. But, the Advisory Board could provide resource to fix this * Basic test coverage * Not a lot of documentation right now (which is a bit of a barrier to adoption) Risks * Not well understood (maybe you guys can fill the gaps) == XenRT = Used by Citrix for XenServer testing. Tarballs have been made available by Citrix under a BSD license. But the code has not been put into live repos: my understanding is that Citrix would do this, if the Xen community believes this is valuable. More Info *http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT *http://www.youtube.com/watch?v=s11_Iw7AI_U Problems: * No publicly accessible demo instance (this is being worked on – to be hosted on a small test bed at http://osuosl.org/ – work sponsored by Citrix) * Currently does not yet support “xl” (a “xl” connector is being worked on – sponsored by Citrix) * Code not in yet public repo Potentially Interesting Properties: * Very large test coverage (including performance, security and other tests). Most of them should work once an “xl” connector is in place * Been in production at scale for a long time: thus well understood * XenRT has a lot of provisioning functionality and supports a distributed architecture: aka the ability to manage machines in different locations (data centres). The detail is abstracted away from users. This creates some interesting possibilities. For example: ** Hardware Vendors on the Advisory Board could provide hardware to the community on their site (assuming that these can be hosted outside a firewall). Some HW vendors on the AB indicated that this would indeed be doable. ** This would open up the opportunity to make available cutting edge or “unusual” HW for testing to the community. ** It would also mean that machines that would be expensive to ship and host by the project, could be hosted on premise by AB vendors * XenRT has the capability to “inject” some test code on the fly (i.e. the test code is attached to a job that is submitted). * I checked this with the XenRT devs and the *Test on demand* approach should be relatively easy to implement, but does not exist. I do not know what of the above would apply to OSSTest. Risks * Complexity * The cost of supporting such a system may be too high * Not in use by the community today * Not clear whether a *local test* version of XenRT is feasible == Support and Ownership = Whatever solution we go for, needs to be properly funded and looked after. This is understood and the intention would be for the Xen Project (aka Advisory Board) to fund a Linux Foundation employee to do this on behalf of the Xen Project: this is a bit like Greg KH and others being LF employees working on the kernel. Some vendors on the Advisory Board indicated that providing Colo/hosting space and HW would be possible in principle, which could help keeping the cost manageable. == Access = Any central system, has of course the issue of access control and managing users. This is obviously a barrier to entry (if we do not have also a local test mechanism). Am wondering how other FOSS communities handle this. This should certainly be the job of the Test Framework owner (see above). Best Regards Lars
Premise/Disclaimer: I reply because I''m one of the ones that played (is playing, actually) with OSSTest, since when standalone mode become available, so I think I may have an opinion on it. Unfortunately, I don''t have direct experience with XenRT, and about it, I only know what I''ve learn in talks and presentations, and talking with people that uses it at Citrix. For that reason, I wouldn''t go as far as providing any sort of comparison, as I only know one of the twos, and that would be unfair, if at all possible. On mar, 2013-11-26 at 13:21 +0000, Lars Kurth wrote:> == Work Flow = > *Local testing*: the basic idea here is for a developer to write their > code and be able to run some tests on their machine locally as part of > their development workflow. We do have same capability to do this with > OSSTest > (seehttp://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/), > but the drawback is that this only works on the developers machine. This > may be good enough. We could “emulate” different environments using > QEMU, but that is likely to be slow. >Honestly, I don''t believe this kind of testing needs to be that broad and/or that general, which is a good thing, as I don''t think it would be possible for it to be that way! ;-P What I mean is, before starting to work on feature X, I usually make sure that I will have access to a platform reasonably sensitive to feature X. Before sending, I do test feature X on such platform, to make sure the code serves its purpose / match with the original design, in an testing environment (which in this case includes both the test platform and the kind of test I execute) meant at highlight exactly that. I don''t see how the job of checking, as thoroughly as possible, hat feature X does not break other --even completely unrelated-- features, does what is expected on a number of platform and architecture that are either similar or totally different from my own one can''t be demanded to ''system testing''. I don''t think there is a way for every developer to have every of its patch series tested on 5 or 10 different boxes, in all the PV, HVM and PVH variants, etc.! :-O All this to say that, I think ''local testing'' is ok to stay on the developers'' machine. What should be done, from a workflow perspective, is (and we''ve started trying to pull in that direction) require that, along with feature X, a testcase for whatever the ''system testing'' will end up being to be submited. That way, breakage of feature X on weird archs, as well as feature X either breaking or being broken (in future) by other features would come, again as a part of ''system testing''. About the actual tools, I''ve been able to install and use OSSTest standalone mode inside the Citrix network (i.e., the easy part :-)) and am halfway through setting it up here at my place, for using it to manage my (only!) testbox. It''s not super easy yet, it probably can be improved, but it''s _there_ right now and it''s _working_. The fact that such a thing is possible is, I think, of paramount importance to allow people to familiarize with the testing framework if they want, and becomes absolutely fundamental if we want to encourage (not to mention request, more or less formally) developers to submit test cases for their stuff.> *Test on demand:* this would be a mixture between local testing and > system testing. It can probably be implemented in both OSSTest and > XenRT, although I believe XenRT is a closer to this model already. >Well, if this will become a real chance, considering that *a*lot* of people would possibly wan to run complex and lengthy tests, unless the amount of hardware we''ll have is unlimited, it looks to me that a job scheduler would be quite a nice thing to have. As I stated already, I know next to nothing of XenRT, and what I know about OSSTest is mostly about standalone mode, so I don''t know who''s ahead, I''ll just say that I really think we need something good at doing this (test job scheduling) in he final solution.> The > workflow would look like this: > a) code developed locally > b) developer checks working prototype into their personal git branch > (which would need to be accessible from the web) > c) developer submits a job to the test farm which checks out and builds > code and runs a set of interesting (or new) tests on some machines of > different architectures. New tests would probably need to be on the git > branch > d) if the test fail, the cycle starts again > d) if all works well, code is submitted for review as normal (and test > results could be attached) >Modulo scheduling test jobs (covered above), for what I know OSSTest can handle all of this.> == OSSTest => > What runs now and thus easiest to get started on > > More Info > *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/ > *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/ > *http://www.youtube.com/watch?v=JxTFZIwZzJ8 > > Problems: > * Runs on Citrix premises (thus general access is an issue) > * Ian Jackson is acting as sys-admin in his spare time. But, the > Advisory Board could provide resource to fix this >Well, these are not real issues, are they? I mean, yes, that''s the current situation, but if we decide to go for OSSTest, both infrastructure and people will be allocated properly, making these disappear.> * Basic test coverage >True, but I never checked how may and what test coverage XenRT would provide, perhaps someone could put this information somewhere. For OSSTest, it''s all here (isn''t that Ian?): http://www.chiark.greenend.org.uk/~xensrcts/logs/22099/> * Not a lot of documentation right now (which is a bit of a barrier to > adoption) >Yes, that is definitely true, the biggest issue with it (but again, I can''t really compare, since I never really went checking how much docs there is for XenRT). I think, a similar very strict policy about improving and keeping updated the doc, similar to what we introduced for Xen, would be required here too, if we go for OSSTest. If we do that, I''m sure things will improve.> Risks > * Not well understood (maybe you guys can fill the gaps) >Again, true, although the fact that the code is there, and that is in an actual repo, with all the history, etc., and the fact that we''re starting to see patches on xen-devel, is already and will continue to clarify what OSSTest does and how it does it quite a lot. Looking back at my experience, for instance, something that would have helped me / saved me some time (and some questions to Ian on IRC) is a clear definition of what ''flights'', ''jobs'', ''recipes'' and ''testcases'' are, and what relationships they share among each others.> == XenRT =>And I''m not saying anything about it, as I don''t like speaking of things I don''t know. :-(> == Support and Ownership => > Whatever solution we go for, needs to be properly funded and looked > after. This is understood and the intention would be for the Xen Project > (aka Advisory Board) to fund a Linux Foundation employee to do this on > behalf of the Xen Project: this is a bit like Greg KH and others being > LF employees working on the kernel. >Or, perhaps a better example, like John Hawley which, when acting as the chief of maintenance of the kernel.org infrastructure, is also payed by the Linux Foundation (https://www.linkedin.com/in/warthog9). Ok... Done. Hope my small contribution can be of help in the discussion. :-) Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Dario, that makes sense. This shouldn't be a OSSTest vs. XenRT discussion. I added a few of comments... Lars On 27/11/2013 03:56, Dario Faggioli wrote:>> *Local testing*: the basic idea here is for a developer to write their >> code and be able to run some tests on their machine locally as part of >> their development workflow. We do have same capability to do this with >> OSSTest >> (seehttp://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/), >> but the drawback is that this only works on the developers machine. This >> may be good enough. We could “emulate” different environments using >> QEMU, but that is likely to be slow. >> > Honestly, I don't believe this kind of testing needs to be that broad > and/or that general, which is a good thing, as I don't think it would be > possible for it to be that way! ;-P > > What I mean is, before starting to work on feature X, I usually make > sure that I will have access to a platform reasonably sensitive to > feature X. Before sending, I do test feature X on such platform, to make > sure the code serves its purpose / match with the original design, in an > testing environment (which in this case includes both the test platform > and the kind of test I execute) meant at highlight exactly that.This is certainly true, for new feature development such as NUMA.> I don't see how the job of checking, as thoroughly as possible, hat > feature X does not break other --even completely unrelated-- features, > does what is expected on a number of platform and architecture that are > either similar or totally different from my own one can't be demanded to > 'system testing'. I don't think there is a way for every developer to > have every of its patch series tested on 5 or 10 different boxes, in all > the PV, HVM and PVH variants, etc.! :-OI don't think I am suggesting that we should always run every patch against all machines before it is submitted. But maybe it is a way to de-risk feature development that is inherently more risky, such as HVM, or other low-level stuff. Or for areas, where we do not have as many reviewers as we would like. It may make HW accessible to some people, that they may not normally have access to. There could be other potential uses. If we had the capability of running some tests on different hardware before a patch is submitted, wouldn't that speed up the review-commit cycle. Each bug/issue that is found after a system test failure, will lead to a re-review and will consume time and effort. As we are moving to shorter release cycles, this sounds like something that would be very valuable. It is also quite conceivable that this line of argument is a "red herring" (and that I stated a problem which does not actually exist). I guess it depends on how often we break functionality unintentionally in practice. I do think that our community is actually quite good and thorough when it comes to reviewing patches.> All this to say that, I think 'local testing' is ok to stay on the > developers' machine. What should be done, from a workflow perspective, > is (and we've started trying to pull in that direction) require that, > along with feature X, a testcase for whatever the 'system testing' will > end up being to be submited. That way, breakage of feature X on weird > archs, as well as feature X either breaking or being broken (in future) > by other features would come, again as a part of 'system testing'. > > About the actual tools, I've been able to install and use OSSTest > standalone mode inside the Citrix network (i.e., the easy part :-)) and > am halfway through setting it up here at my place, for using it to > manage my (only!) testbox. It's not super easy yet, it probably can be > improved, but it's _there_ right now and it's _working_. The fact that > such a thing is possible is, I think, of paramount importance to allow > people to familiarize with the testing framework if they want, and > becomes absolutely fundamental if we want to encourage (not to mention > request, more or less formally) developers to submit test cases for > their stuff.Totally agreed.>> [snip] >> == OSSTest =>> >> What runs now and thus easiest to get started on >> >> [snip] >> >> Problems: >> * Runs on Citrix premises (thus general access is an issue) >> * Ian Jackson is acting as sys-admin in his spare time. But, the >> Advisory Board could provide resource to fix this > Well, these are not real issues, are they? I mean, yes, that's the > current situation, but if we decide to go for OSSTest, both > infrastructure and people will be allocated properly, making these > disappear.Agreed. Which is why I added this, such that it doesn't get forgotten.>> * Basic test coverage >> > True, but I never checked how may and what test coverage XenRT would > provide, perhaps someone could put this information somewhere. > > For OSSTest, it's all here (isn't that Ian?): > http://www.chiark.greenend.org.uk/~xensrcts/logs/22099/ >> * Not a lot of documentation right now (which is a bit of a barrier to >> adoption) >> > Yes, that is definitely true, the biggest issue with it (but again, I > can't really compare, since I never really went checking how much docs > there is for XenRT). > > I think, a similar very strict policy about improving and keeping > updated the doc, similar to what we introduced for Xen, would be > required here too, if we go for OSSTest. If we do that, I'm sure things > will improve.Same point: it is an issue today and it could be resolved with adequate funds, focus and will. But the issue ought to be mentioned.>> Risks >> * Not well understood (maybe you guys can fill the gaps) > Again, true, although the fact that the code is there, and that is in an > actual repo, with all the history, etc., and the fact that we're > starting to see patches on xen-devel, is already and will continue to > clarify what OSSTest does and how it does it quite a lot. > > Looking back at my experience, for instance, something that would have > helped me / saved me some time (and some questions to Ian on IRC) is a > clear definition of what 'flights', 'jobs', 'recipes' and 'testcases' > are, and what relationships they share among each others.Again, I raised this such that the existing users of OSSTest can flesh this section out and provide a bit more clarity. It's basically a place-holder that you guys should flesh out. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On ven, 2013-11-29 at 16:52 +0000, Lars Kurth wrote:> Dario, > that makes sense. >Good to know. :-)> This shouldn''t be a OSSTest vs. XenRT discussion. >Sure, and I neither read/interpreted it that way, nor I wanted to turn it into something like that. If it sounded like I did, sorry, that wasn''t my intention. As I said, I only have experience with OSSTest, and I felt like I waned to share thoughts and opinions coming out from such experience, about what it can and can''t (right now) do and be used for, that''s it. :-)> On 27/11/2013 03:56, Dario Faggioli wrote: > > I don''t see how the job of checking, as thoroughly as possible, hat > > feature X does not break other --even completely unrelated-- features, > > does what is expected on a number of platform and architecture that are > > either similar or totally different from my own one can''t be demanded to > > ''system testing''. I don''t think there is a way for every developer to > > have every of its patch series tested on 5 or 10 different boxes, in all > > the PV, HVM and PVH variants, etc.! :-O > I don''t think I am suggesting that we should always run every patch > against all machines before it is submitted. >Right, I got that in the first place. Point is, as you suggest yourself below, if we go that way, there''s some either some formal policing or ''rule-of-thumb''-ing needed, and I just think it''s going to be an hard call, to the point that I''m not sure it''s even worth. As I tired to say, I''d rather invest that effort in making OSSTest (but I suspect that would apply to XenRT too) easier to "deploy", or, if you want, in teaching people how to do that, so that they can run it in *their* homes/offices, perform their own testing by writing actual testcases for it and, perhaps, submit them to the "official" testing service / push gate (e.g., as something that runs once in a while, instead of every night, depending on the feature being tested).> But maybe it is a way to de-risk feature development that is inherently > more risky, such as HVM, or other low-level stuff. > Or for areas, where we do not have as many reviewers as we would like. > It may make HW accessible to some people, that they may not normally > have access to. >Exactly, and there are some nice examples of policy items. Again, I''m certainly not against it, but I don''t thing this is the main point of this discussion (and so, sorry for making some fuss about it myself in the first place), so let''s leave it and go ahead. :-)> >> [snip] > >> == OSSTest => >> > >> What runs now and thus easiest to get started on > >> > >> [snip] > >> > >> Problems: > >> * Runs on Citrix premises (thus general access is an issue) > >> * Ian Jackson is acting as sys-admin in his spare time. But, the > >> Advisory Board could provide resource to fix this > > Well, these are not real issues, are they? I mean, yes, that''s the > > current situation, but if we decide to go for OSSTest, both > > infrastructure and people will be allocated properly, making these > > disappear. > Agreed. Which is why I added this, such that it doesn''t get forgotten. >Perfect then. Given that, I probably would move this and the other similar arguments away from the ''Problems:'' section (and of course do the same for the same arguments in the XenRT paragraph), as, although it''s important not to forget them, they''re not that relevant in the discussion/decision phase, are they? That is, BTW, all I meant with this (and the other similar) comment(s).> >> Risks > >> * Not well understood (maybe you guys can fill the gaps) > > Again, true, although the fact that the code is there, and that is in an > > actual repo, with all the history, etc., and the fact that we''re > > starting to see patches on xen-devel, is already and will continue to > > clarify what OSSTest does and how it does it quite a lot. > > > > Looking back at my experience, for instance, something that would have > > helped me / saved me some time (and some questions to Ian on IRC) is a > > clear definition of what ''flights'', ''jobs'', ''recipes'' and ''testcases'' > > are, and what relationships they share among each others. > Again, I raised this such that the existing users of OSSTest can flesh > this section out and provide a bit more clarity. It''s basically a > place-holder that you guys should flesh out. >Which is exactly what I tried to do with my e-mail, all of it, but particularly here. :-) Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Tue, Nov 26, 2013 at 1:21 PM, Lars Kurth <lars.kurth@xen.org> wrote:> Hi all, > > you probably have all heard by now that the Xen Project Advisory Board (a > group of vendors who provide funds to the Xen Project that are intended to > be used for the good of the community) recently created the Test Framework > Working Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains > more information about the group. The working group had its first meeting a > few weeks ago and one of the actions I had was to kick off a thread on > development lists to figure out what would help the developer community. > > I was planning to kick off this thread with some questions and options, > which reflect some discussions I had with individuals in the community, > various meetings (WG and AB meetings), etc. which I condensed into a > picture. > > This reflects my personal opinion (not a Citrix opinion) and is merely > intended to get a discussion going. Feel free to pick it apart: I won’t be > upset.While I have a few quibbles, what you wrote seems to be a fair reflection of the situation as a whole. But I don''t really see any action items, or questions. So other than quibble with little details, I don''t see a way to engage with this and actually have a discussion. :-) What do you need from us right now? -George
On Tue, 2013-11-26 at 13:21 +0000, Lars Kurth wrote:> Hi all, > > you probably have all heard by now that the Xen Project Advisory Board > (a group of vendors who provide funds to the Xen Project that are > intended to be used for the good of the community) recently created the > Test Framework Working > Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains more > information about the group. The working group had its first meeting a > few weeks ago and one of the actions I had was to kick off a thread on > development lists to figure out what would help the developer community. > > I was planning to kick off this thread with some questions and options, > which reflect some discussions I had with individuals in the community, > various meetings (WG and AB meetings), etc. which I condensed into a > picture. > > This reflects my personal opinion (not a Citrix opinion) and is merely > intended to get a discussion going. Feel free to pick it apart: I won’t > be upset. > > First, I wanted to clear up a few misconceptions that I have heard from > a few people: > > * The Advisory Board has funds that can be used to create an > independently hosted test infrastructure to help the developer > community. However, funds are limited. Thus, it is important that we do > what is right for the Xen community in the short term and the longer > term. Otherwise, we will burn funds that could be used to help the Xen > community in other ways. > * The Test Framework Working Group is made up of people employed by > vendors who have some experience in testing. > * There is no intention to prescribe a test environment that you then > have to use. Advisory Board members made clear to me that they want to > make sure that what we end up with a solution that works for you. > * At the Xen Developer Summit two different solutions for system testing > were presented. The intention was to explain what is there and what we > can use going forward. A presentation on OSSTest which runs regularly > today was given. And one for XenRT, for which there is a plan to get a > small 3 box system up and running that can be used for you to look at. > Citrix volunteered to set this up at its own cost. > * Just to be clear: what works for you may be one of these, none of > these, both of them, … > * There may also be different answers in the short and the long run. > * At the end of the day, different community members will have different > views. Also the Advisory Board members who provide the funds, will also > have specific interests that they will push for. Thus, in all > likelihood, we will have to find a good enough compromise. > * The vast majority of Advisory Board members care about the Hypervisor > (and not so much about XAPI and Mirage OS). Thus, it is likely that the > focus of the test system would be the Hypervisor. > > So let me try and condense some of the arguments and opinions I heard > and information that is around. This list may be incomplete. > > == Work Flow => > I added this section, because some members of the community and the > working group had prior experience with attempts to introduce a test > infrastructure for an open source community in the past, and these may > not have worked as well as hoped. I made up some of the terminology. > > *Local testing*: the basic idea here is for a developer to write their[...] We have a small amount of local test suites in the tree (e.g. vif and disk config parsing have little test suites) but it could do with tying together with some infrastructure into something which is simple to run (currently it requires an installed Xen system and there is no one single way to run something). As you correctly suggest there is a limit to how much local testing can cover in terms of elapsed time, available resources, the configurations which can be reasonably set up, running on real hardware etc. IMHO This could benefit from an enthusiastic (or press-ganged by their manager ;-)) community member putting some time into tying it all together into something which we can ask people to run before submitting with a straight face.> *System testing*: both OSSTest and XenRT are essentially system test[...] I think most people use "system testing" to mean testing of the integrated whole, as opposed to e.g. unit testing. The current "automated test" which we have covers some aspects of both whole system and unit testing. Anyway, terminology aside, the existing osstest stuff is *extremely* valuable IMHO, and the system testing has been very useful over the majority of the lifetime of the xen project, at least as long as I've been involved. The main limitation is the amount of resources dedicated to it, in terms of hardware (and its location within citrix infrastructure doesn't help here) and test coverage. Even with its current set of tests and limited hardware it already tests far more than we could ever realistically ask people to do locally before submitting and it catches real issues on real hardware. Any local test stuff should obviously be integrated into the system tests as a step as well. I notice that your description of system test omits the targeted local testing which we expect contributors to do before submitting a patch -- by targeted I mean you are changing $FOO therefore you should be trying $FOO! And if you think you might have an impact on $BAR you should be testing that too. I just mention it because your description seemed to imply (inadvertently I expect) that there was no testing at all between writing the code and the system tests running, which is not quite accurate. IMHO both local and system test are valuable. I think the local testing situation can be improved by people working within the community to do the work (in particular building out the infrastructure), whereas the system testing side of things would benefit greatly from any resourcing which the AB can provide in terms of hardware, hosting and sysadmin time etc. There is no doubt in my mind that this would be beneficial to the community in both the short and long term. It might also be worth considering spending some money kickstarting the actual tests (i.e. fleshing out the suites) in both cases, but I think ultimately I think this needs to be driven by community member (AB or otherwise) who care about particular functionality making sure the tests exist, probably by writing them. So in terms of budget I think that would be secondary to sorting out the hosting etc> *Test on demand:* this would be a mixture between local testing and[...] I think it would be nice long term goal to aim for this but short term the other two types of testing are more important.> IMHO, this would be a nice mid to long-term goal, > assuming it could be made to work with the funds we have.Heh, I should read right to the end ;-)> == OSSTest => > What runs now and thus easiest to get started on > > More Info > *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/ > *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/ > *http://www.youtube.com/watch?v=JxTFZIwZzJ8 > > Problems: > * Runs on Citrix premises (thus general access is an issue) > * Ian Jackson is acting as sys-admin in his spare time. But, the > Advisory Board could provide resource to fix this > * Basic test coverage > * Not a lot of documentation right now (which is a bit of a barrier to > adoption) > > Risks > * Not well understood (maybe you guys can fill the gaps)This is slowly changing, Wei, Roger and myself have all done development with osstest and contributed (or are in the process of doing so) new bits of testing. I think Dario and Anthony have played with it too. There is certainly more which could be done here in terms of documentation. I at least was planning to make this part of my focus on future documentation and/or test days. I think all of the above applies equally to XenRT, either system is going to have a learning curve and is going to need documentation for the community etc.> == XenRT => > Used by Citrix for XenServer testing. Tarballs have been made available > by Citrix under a BSD license. But the code has not been put into live > repos: my understanding is that Citrix would do this, if the Xen > community believes this is valuable. > > More Info > *http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT > *http://www.youtube.com/watch?v=s11_Iw7AI_U > > Problems: > * No publicly accessible demo instance (this is being worked on – to be > hosted on a small test bed at http://osuosl.org/ – work sponsored by Citrix) > * Currently does not yet support “xl” (a “xl” connector is being worked > on – sponsored by Citrix) > * Code not in yet public repo > > Potentially Interesting Properties: > * Very large test coverage (including performance, security and other > tests). Most of them should work once an “xl” connector is in placeI think that's rather optimistic. I would expect that a reasonable proportion of the interesting tests will require features of xapi to work, e.g. pools of hosts, storage management, networking etc and/or require some amount of reworking to function with xl.> * Been in production at scale for a long time: thus well understood > * XenRT has a lot of provisioning functionality and supports a > distributed architecture: aka the ability to manage machines in > different locations (data centres). The detail is abstracted away from > users. This creates some interesting possibilities. For example: > ** Hardware Vendors on the Advisory Board could provide hardware to the > community on their site (assuming that these can be hosted outside a > firewall). Some HW vendors on the AB indicated that this would indeed be > doable. > ** This would open up the opportunity to make available cutting edge or > “unusual” HW for testing to the community. > ** It would also mean that machines that would be expensive to ship and > host by the project, could be hosted on premise by AB vendors > * XenRT has the capability to “inject” some test code on the fly (i.e. > the test code is attached to a job that is submitted). > * I checked this with the XenRT devs and the *Test on demand* approach > should be relatively easy to implement, but does not exist. > > I do not know what of the above would apply to OSSTest.I think it is all equally doable for either.> Risks > * Complexity > * The cost of supporting such a system may be too high > * Not in use by the community today > * Not clear whether a *local test* version of XenRT is feasible > > == Support and Ownership => > Whatever solution we go for, needs to be properly funded and looked > after.From the remainder of the paragraph I think you are talking specifically about hiring a test person here I think? I think this is essential, the current testing is done on a shoe string and that is one of its main limiting factors.> This is understood and the intention would be for the Xen Project > (aka Advisory Board) to fund a Linux Foundation employee to do this on > behalf of the Xen Project: this is a bit like Greg KH and others being > LF employees working on the kernel. Some vendors on the Advisory Board > indicated that providing Colo/hosting space and HW would be possible in > principle, which could help keeping the cost manageable.We should certainly be taking them up on those offers IMHO.> == Access => > Any central system, has of course the issue of access control and > managing users. This is obviously a barrier to entry (if we do not have > also a local test mechanism). Am wondering how other FOSS communities > handle this. This should certainly be the job of the Test Framework > owner (see above).At a minimum it ought to be possible to allow access to any employee of a project member, since we have the opportunity through the membership process to put whatever paperwork and agreements (acceptable use etc) in place. Unfettered access for anyone who rocks up and asks is a bit trickier. I'm quite happy to let that be the framework owner's problem ;-) Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 02/12/2013 15:55, George Dunlap wrote:> On Tue, Nov 26, 2013 at 1:21 PM, Lars Kurth <lars.kurth@xen.org> wrote: >> Hi all, >> >> you probably have all heard by now that the Xen Project Advisory Board (a >> group of vendors who provide funds to the Xen Project that are intended to >> be used for the good of the community) recently created the Test Framework >> Working Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains >> more information about the group. The working group had its first meeting a >> few weeks ago and one of the actions I had was to kick off a thread on >> development lists to figure out what would help the developer community. >> >> I was planning to kick off this thread with some questions and options, >> which reflect some discussions I had with individuals in the community, >> various meetings (WG and AB meetings), etc. which I condensed into a >> picture. >> >> This reflects my personal opinion (not a Citrix opinion) and is merely >> intended to get a discussion going. Feel free to pick it apart: I won’t be >> upset. > While I have a few quibbles, what you wrote seems to be a fair > reflection of the situation as a whole. But I don''t really see any > action items, or questions. So other than quibble with little > details, I don''t see a way to engage with this and actually have a > discussion. :-) > > What do you need from us right now?Sorry, I missed this reply. Maybe this was all a bit too abstract What I am looking for is: * Are there any specific needs that you have that should be considered in this discussion * Is there anything in the way we do tests now that really sucks and thus should be fixed * Are there any red lines - aka any solutions, approaches, etc. - that in your opinion would prevent yourself and the community from using a common test framework * I am also looking for one (or maybe two) volunteers who can represent the developer communities view within the test framework WG Lars
Ian, I think that is a good assessment and thanks for pointing out some gaps in my understanding. So to summarize: * We have some local test suites which could be run, but are probably not as they are poorly documented * We expect people to do some targetted local testing (presumably performed in a manual manner) of the features they developed and of those which may be impacted. Bt we don't actually always know whether they do. * osstest (or system testing in general) is *extremely* valuable * building out the infrastructure for system testing (aka number and diversity of boxes) would be extremely valuable - this really means funding hardware, hosting and sysadmin time * It might also be worth considering spending some money kickstarting the actual tests (i.e. fleshing out the suites) * *Test on demand* would be a nice long term goal * Some members of the community are intending to make OSSTest more accessible by improving docs and sharing their experience * At a minimum it ought to be possible to allow access to any employee of, a project member, since we have the opportunity through the membership, process to put whatever paperwork and agreements (acceptable use etc) in place. Lars On 09/12/2013 13:20, Ian Campbell wrote:> On Tue, 2013-11-26 at 13:21 +0000, Lars Kurth wrote: >> Hi all, >> >> you probably have all heard by now that the Xen Project Advisory Board >> (a group of vendors who provide funds to the Xen Project that are >> intended to be used for the good of the community) recently created the >> Test Framework Working >> Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains more >> information about the group. The working group had its first meeting a >> few weeks ago and one of the actions I had was to kick off a thread on >> development lists to figure out what would help the developer community. >> >> I was planning to kick off this thread with some questions and options, >> which reflect some discussions I had with individuals in the community, >> various meetings (WG and AB meetings), etc. which I condensed into a >> picture. >> >> This reflects my personal opinion (not a Citrix opinion) and is merely >> intended to get a discussion going. Feel free to pick it apart: I won’t >> be upset. >> >> First, I wanted to clear up a few misconceptions that I have heard from >> a few people: >> >> * The Advisory Board has funds that can be used to create an >> independently hosted test infrastructure to help the developer >> community. However, funds are limited. Thus, it is important that we do >> what is right for the Xen community in the short term and the longer >> term. Otherwise, we will burn funds that could be used to help the Xen >> community in other ways. >> * The Test Framework Working Group is made up of people employed by >> vendors who have some experience in testing. >> * There is no intention to prescribe a test environment that you then >> have to use. Advisory Board members made clear to me that they want to >> make sure that what we end up with a solution that works for you. >> * At the Xen Developer Summit two different solutions for system testing >> were presented. The intention was to explain what is there and what we >> can use going forward. A presentation on OSSTest which runs regularly >> today was given. And one for XenRT, for which there is a plan to get a >> small 3 box system up and running that can be used for you to look at. >> Citrix volunteered to set this up at its own cost. >> * Just to be clear: what works for you may be one of these, none of >> these, both of them, … >> * There may also be different answers in the short and the long run. >> * At the end of the day, different community members will have different >> views. Also the Advisory Board members who provide the funds, will also >> have specific interests that they will push for. Thus, in all >> likelihood, we will have to find a good enough compromise. >> * The vast majority of Advisory Board members care about the Hypervisor >> (and not so much about XAPI and Mirage OS). Thus, it is likely that the >> focus of the test system would be the Hypervisor. >> >> So let me try and condense some of the arguments and opinions I heard >> and information that is around. This list may be incomplete. >> >> == Work Flow =>> >> I added this section, because some members of the community and the >> working group had prior experience with attempts to introduce a test >> infrastructure for an open source community in the past, and these may >> not have worked as well as hoped. I made up some of the terminology. >> >> *Local testing*: the basic idea here is for a developer to write their > [...] > > We have a small amount of local test suites in the tree (e.g. vif and > disk config parsing have little test suites) but it could do with tying > together with some infrastructure into something which is simple to run > (currently it requires an installed Xen system and there is no one > single way to run something). > > As you correctly suggest there is a limit to how much local testing can > cover in terms of elapsed time, available resources, the configurations > which can be reasonably set up, running on real hardware etc. IMHO This > could benefit from an enthusiastic (or press-ganged by their > manager ;-)) community member putting some time into tying it all > together into something which we can ask people to run before submitting > with a straight face. > > >> *System testing*: both OSSTest and XenRT are essentially system test > [...] > > I think most people use "system testing" to mean testing of the > integrated whole, as opposed to e.g. unit testing. The current > "automated test" which we have covers some aspects of both whole system > and unit testing. > > Anyway, terminology aside, the existing osstest stuff is *extremely* > valuable IMHO, and the system testing has been very useful over the > majority of the lifetime of the xen project, at least as long as I've > been involved. The main limitation is the amount of resources dedicated > to it, in terms of hardware (and its location within citrix > infrastructure doesn't help here) and test coverage. > > Even with its current set of tests and limited hardware it already tests > far more than we could ever realistically ask people to do locally > before submitting and it catches real issues on real hardware. > > Any local test stuff should obviously be integrated into the system > tests as a step as well. > > I notice that your description of system test omits the targeted local > testing which we expect contributors to do before submitting a patch -- > by targeted I mean you are changing $FOO therefore you should be trying > $FOO! And if you think you might have an impact on $BAR you should be > testing that too. I just mention it because your description seemed to > imply (inadvertently I expect) that there was no testing at all between > writing the code and the system tests running, which is not quite > accurate. > > IMHO both local and system test are valuable. I think the local testing > situation can be improved by people working within the community to do > the work (in particular building out the infrastructure), whereas the > system testing side of things would benefit greatly from any resourcing > which the AB can provide in terms of hardware, hosting and sysadmin time > etc. There is no doubt in my mind that this would be beneficial to the > community in both the short and long term. > > It might also be worth considering spending some money kickstarting the > actual tests (i.e. fleshing out the suites) in both cases, but I think > ultimately I think this needs to be driven by community member (AB or > otherwise) who care about particular functionality making sure the tests > exist, probably by writing them. So in terms of budget I think that > would be secondary to sorting out the hosting etc > >> *Test on demand:* this would be a mixture between local testing and > [...] > > I think it would be nice long term goal to aim for this but short term > the other two types of testing are more important. > >> IMHO, this would be a nice mid to long-term goal, >> assuming it could be made to work with the funds we have. > Heh, I should read right to the end ;-) > >> == OSSTest =>> >> What runs now and thus easiest to get started on >> >> More Info >> *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/ >> *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/ >> *http://www.youtube.com/watch?v=JxTFZIwZzJ8 >> >> Problems: >> * Runs on Citrix premises (thus general access is an issue) >> * Ian Jackson is acting as sys-admin in his spare time. But, the >> Advisory Board could provide resource to fix this >> * Basic test coverage >> * Not a lot of documentation right now (which is a bit of a barrier to >> adoption) >> >> Risks >> * Not well understood (maybe you guys can fill the gaps) > This is slowly changing, Wei, Roger and myself have all done development > with osstest and contributed (or are in the process of doing so) new > bits of testing. I think Dario and Anthony have played with it too. > There is certainly more which could be done here in terms of > documentation. I at least was planning to make this part of my focus on > future documentation and/or test days. > > I think all of the above applies equally to XenRT, either system is > going to have a learning curve and is going to need documentation for > the community etc. > >> == XenRT =>> >> Used by Citrix for XenServer testing. Tarballs have been made available >> by Citrix under a BSD license. But the code has not been put into live >> repos: my understanding is that Citrix would do this, if the Xen >> community believes this is valuable. >> >> More Info >> *http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT >> *http://www.youtube.com/watch?v=s11_Iw7AI_U >> >> Problems: >> * No publicly accessible demo instance (this is being worked on – to be >> hosted on a small test bed at http://osuosl.org/ – work sponsored by Citrix) >> * Currently does not yet support “xl” (a “xl” connector is being worked >> on – sponsored by Citrix) >> * Code not in yet public repo >> >> Potentially Interesting Properties: >> * Very large test coverage (including performance, security and other >> tests). Most of them should work once an “xl” connector is in place > I think that's rather optimistic. I would expect that a reasonable > proportion of the interesting tests will require features of xapi to > work, e.g. pools of hosts, storage management, networking etc and/or > require some amount of reworking to function with xl. > >> * Been in production at scale for a long time: thus well understood >> * XenRT has a lot of provisioning functionality and supports a >> distributed architecture: aka the ability to manage machines in >> different locations (data centres). The detail is abstracted away from >> users. This creates some interesting possibilities. For example: >> ** Hardware Vendors on the Advisory Board could provide hardware to the >> community on their site (assuming that these can be hosted outside a >> firewall). Some HW vendors on the AB indicated that this would indeed be >> doable. >> ** This would open up the opportunity to make available cutting edge or >> “unusual” HW for testing to the community. >> ** It would also mean that machines that would be expensive to ship and >> host by the project, could be hosted on premise by AB vendors >> * XenRT has the capability to “inject” some test code on the fly (i.e. >> the test code is attached to a job that is submitted). >> * I checked this with the XenRT devs and the *Test on demand* approach >> should be relatively easy to implement, but does not exist. >> >> I do not know what of the above would apply to OSSTest. > I think it is all equally doable for either. > >> Risks >> * Complexity >> * The cost of supporting such a system may be too high >> * Not in use by the community today >> * Not clear whether a *local test* version of XenRT is feasible >> >> == Support and Ownership =>> >> Whatever solution we go for, needs to be properly funded and looked >> after. > From the remainder of the paragraph I think you are talking specifically > about hiring a test person here I think? > > I think this is essential, the current testing is done on a shoe string > and that is one of its main limiting factors. > >> This is understood and the intention would be for the Xen Project >> (aka Advisory Board) to fund a Linux Foundation employee to do this on >> behalf of the Xen Project: this is a bit like Greg KH and others being >> LF employees working on the kernel. Some vendors on the Advisory Board >> indicated that providing Colo/hosting space and HW would be possible in >> principle, which could help keeping the cost manageable. > We should certainly be taking them up on those offers IMHO. > >> == Access =>> >> Any central system, has of course the issue of access control and >> managing users. This is obviously a barrier to entry (if we do not have >> also a local test mechanism). Am wondering how other FOSS communities >> handle this. This should certainly be the job of the Test Framework >> owner (see above). > At a minimum it ought to be possible to allow access to any employee of > a project member, since we have the opportunity through the membership > process to put whatever paperwork and agreements (acceptable use etc) in > place. > > Unfettered access for anyone who rocks up and asks is a bit trickier. > I'm quite happy to let that be the framework owner's problem ;-) > > Ian. > > > > > > > > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Tue, 2013-12-10 at 18:29 +0000, Lars Kurth wrote:> Ian, > > I think that is a good assessment and thanks for pointing out some gaps > in my understanding.No problem.> So to summarize: > * We have some local test suites which could be run, but are probably > not as they are poorly documentedMore documentation is always good. adding a top level script or make target to actually run them in a trivial way would IMHO be even more valuable in the short term. Having that target/script be easily extensible to other suites would be a must as well since we have some but not many of these sorts of tests.> * We expect people to do some targetted local testing (presumably > performed in a manual manner) of the features they developed and of > those which may be impacted. Bt we don't actually always know whether > they do.It's occasionally obvious that someone hasn't tested (or sometimes even compiled) their patches, but for the most part it does appear that people are actually doing this bit and I think it is well understood as an expectation in all development environments (i.e. OSS communities but also commercial settings etc), not just ours. (as well as a common understanding that it's expected there's probably some element of not wanting to look stupid by posting patches which don't compile, or haven't been tested)> * osstest (or system testing in general) is *extremely* valuable > * building out the infrastructure for system testing (aka number and > diversity of boxes) would be extremely valuable - this really means > funding hardware, hosting and sysadmin time > * It might also be worth considering spending some money kickstarting > the actual tests (i.e. fleshing out the suites) > * *Test on demand* would be a nice long term goal > * Some members of the community are intending to make OSSTest more > accessible by improving docs and sharing their experience > * At a minimum it ought to be possible to allow access to any employee > of, a project member, since we have the opportunity through the > membership, process to put whatever paperwork and agreements (acceptable > use etc) in place.Ack to all that. Ian.> > Lars > > On 09/12/2013 13:20, Ian Campbell wrote: > > On Tue, 2013-11-26 at 13:21 +0000, Lars Kurth wrote: > >> Hi all, > >> > >> you probably have all heard by now that the Xen Project Advisory Board > >> (a group of vendors who provide funds to the Xen Project that are > >> intended to be used for the good of the community) recently created the > >> Test Framework Working > >> Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains more > >> information about the group. The working group had its first meeting a > >> few weeks ago and one of the actions I had was to kick off a thread on > >> development lists to figure out what would help the developer community. > >> > >> I was planning to kick off this thread with some questions and options, > >> which reflect some discussions I had with individuals in the community, > >> various meetings (WG and AB meetings), etc. which I condensed into a > >> picture. > >> > >> This reflects my personal opinion (not a Citrix opinion) and is merely > >> intended to get a discussion going. Feel free to pick it apart: I won’t > >> be upset. > >> > >> First, I wanted to clear up a few misconceptions that I have heard from > >> a few people: > >> > >> * The Advisory Board has funds that can be used to create an > >> independently hosted test infrastructure to help the developer > >> community. However, funds are limited. Thus, it is important that we do > >> what is right for the Xen community in the short term and the longer > >> term. Otherwise, we will burn funds that could be used to help the Xen > >> community in other ways. > >> * The Test Framework Working Group is made up of people employed by > >> vendors who have some experience in testing. > >> * There is no intention to prescribe a test environment that you then > >> have to use. Advisory Board members made clear to me that they want to > >> make sure that what we end up with a solution that works for you. > >> * At the Xen Developer Summit two different solutions for system testing > >> were presented. The intention was to explain what is there and what we > >> can use going forward. A presentation on OSSTest which runs regularly > >> today was given. And one for XenRT, for which there is a plan to get a > >> small 3 box system up and running that can be used for you to look at. > >> Citrix volunteered to set this up at its own cost. > >> * Just to be clear: what works for you may be one of these, none of > >> these, both of them, … > >> * There may also be different answers in the short and the long run. > >> * At the end of the day, different community members will have different > >> views. Also the Advisory Board members who provide the funds, will also > >> have specific interests that they will push for. Thus, in all > >> likelihood, we will have to find a good enough compromise. > >> * The vast majority of Advisory Board members care about the Hypervisor > >> (and not so much about XAPI and Mirage OS). Thus, it is likely that the > >> focus of the test system would be the Hypervisor. > >> > >> So let me try and condense some of the arguments and opinions I heard > >> and information that is around. This list may be incomplete. > >> > >> == Work Flow => >> > >> I added this section, because some members of the community and the > >> working group had prior experience with attempts to introduce a test > >> infrastructure for an open source community in the past, and these may > >> not have worked as well as hoped. I made up some of the terminology. > >> > >> *Local testing*: the basic idea here is for a developer to write their > > [...] > > > > We have a small amount of local test suites in the tree (e.g. vif and > > disk config parsing have little test suites) but it could do with tying > > together with some infrastructure into something which is simple to run > > (currently it requires an installed Xen system and there is no one > > single way to run something). > > > > As you correctly suggest there is a limit to how much local testing can > > cover in terms of elapsed time, available resources, the configurations > > which can be reasonably set up, running on real hardware etc. IMHO This > > could benefit from an enthusiastic (or press-ganged by their > > manager ;-)) community member putting some time into tying it all > > together into something which we can ask people to run before submitting > > with a straight face. > > > > > >> *System testing*: both OSSTest and XenRT are essentially system test > > [...] > > > > I think most people use "system testing" to mean testing of the > > integrated whole, as opposed to e.g. unit testing. The current > > "automated test" which we have covers some aspects of both whole system > > and unit testing. > > > > Anyway, terminology aside, the existing osstest stuff is *extremely* > > valuable IMHO, and the system testing has been very useful over the > > majority of the lifetime of the xen project, at least as long as I've > > been involved. The main limitation is the amount of resources dedicated > > to it, in terms of hardware (and its location within citrix > > infrastructure doesn't help here) and test coverage. > > > > Even with its current set of tests and limited hardware it already tests > > far more than we could ever realistically ask people to do locally > > before submitting and it catches real issues on real hardware. > > > > Any local test stuff should obviously be integrated into the system > > tests as a step as well. > > > > I notice that your description of system test omits the targeted local > > testing which we expect contributors to do before submitting a patch -- > > by targeted I mean you are changing $FOO therefore you should be trying > > $FOO! And if you think you might have an impact on $BAR you should be > > testing that too. I just mention it because your description seemed to > > imply (inadvertently I expect) that there was no testing at all between > > writing the code and the system tests running, which is not quite > > accurate. > > > > IMHO both local and system test are valuable. I think the local testing > > situation can be improved by people working within the community to do > > the work (in particular building out the infrastructure), whereas the > > system testing side of things would benefit greatly from any resourcing > > which the AB can provide in terms of hardware, hosting and sysadmin time > > etc. There is no doubt in my mind that this would be beneficial to the > > community in both the short and long term. > > > > It might also be worth considering spending some money kickstarting the > > actual tests (i.e. fleshing out the suites) in both cases, but I think > > ultimately I think this needs to be driven by community member (AB or > > otherwise) who care about particular functionality making sure the tests > > exist, probably by writing them. So in terms of budget I think that > > would be secondary to sorting out the hosting etc > > > >> *Test on demand:* this would be a mixture between local testing and > > [...] > > > > I think it would be nice long term goal to aim for this but short term > > the other two types of testing are more important. > > > >> IMHO, this would be a nice mid to long-term goal, > >> assuming it could be made to work with the funds we have. > > Heh, I should read right to the end ;-) > > > >> == OSSTest => >> > >> What runs now and thus easiest to get started on > >> > >> More Info > >> *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/ > >> *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/ > >> *http://www.youtube.com/watch?v=JxTFZIwZzJ8 > >> > >> Problems: > >> * Runs on Citrix premises (thus general access is an issue) > >> * Ian Jackson is acting as sys-admin in his spare time. But, the > >> Advisory Board could provide resource to fix this > >> * Basic test coverage > >> * Not a lot of documentation right now (which is a bit of a barrier to > >> adoption) > >> > >> Risks > >> * Not well understood (maybe you guys can fill the gaps) > > This is slowly changing, Wei, Roger and myself have all done development > > with osstest and contributed (or are in the process of doing so) new > > bits of testing. I think Dario and Anthony have played with it too. > > There is certainly more which could be done here in terms of > > documentation. I at least was planning to make this part of my focus on > > future documentation and/or test days. > > > > I think all of the above applies equally to XenRT, either system is > > going to have a learning curve and is going to need documentation for > > the community etc. > > > >> == XenRT => >> > >> Used by Citrix for XenServer testing. Tarballs have been made available > >> by Citrix under a BSD license. But the code has not been put into live > >> repos: my understanding is that Citrix would do this, if the Xen > >> community believes this is valuable. > >> > >> More Info > >> *http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT > >> *http://www.youtube.com/watch?v=s11_Iw7AI_U > >> > >> Problems: > >> * No publicly accessible demo instance (this is being worked on – to be > >> hosted on a small test bed at http://osuosl.org/ – work sponsored by Citrix) > >> * Currently does not yet support “xl” (a “xl” connector is being worked > >> on – sponsored by Citrix) > >> * Code not in yet public repo > >> > >> Potentially Interesting Properties: > >> * Very large test coverage (including performance, security and other > >> tests). Most of them should work once an “xl” connector is in place > > I think that's rather optimistic. I would expect that a reasonable > > proportion of the interesting tests will require features of xapi to > > work, e.g. pools of hosts, storage management, networking etc and/or > > require some amount of reworking to function with xl. > > > >> * Been in production at scale for a long time: thus well understood > >> * XenRT has a lot of provisioning functionality and supports a > >> distributed architecture: aka the ability to manage machines in > >> different locations (data centres). The detail is abstracted away from > >> users. This creates some interesting possibilities. For example: > >> ** Hardware Vendors on the Advisory Board could provide hardware to the > >> community on their site (assuming that these can be hosted outside a > >> firewall). Some HW vendors on the AB indicated that this would indeed be > >> doable. > >> ** This would open up the opportunity to make available cutting edge or > >> “unusual” HW for testing to the community. > >> ** It would also mean that machines that would be expensive to ship and > >> host by the project, could be hosted on premise by AB vendors > >> * XenRT has the capability to “inject” some test code on the fly (i.e. > >> the test code is attached to a job that is submitted). > >> * I checked this with the XenRT devs and the *Test on demand* approach > >> should be relatively easy to implement, but does not exist. > >> > >> I do not know what of the above would apply to OSSTest. > > I think it is all equally doable for either. > > > >> Risks > >> * Complexity > >> * The cost of supporting such a system may be too high > >> * Not in use by the community today > >> * Not clear whether a *local test* version of XenRT is feasible > >> > >> == Support and Ownership => >> > >> Whatever solution we go for, needs to be properly funded and looked > >> after. > > From the remainder of the paragraph I think you are talking specifically > > about hiring a test person here I think? > > > > I think this is essential, the current testing is done on a shoe string > > and that is one of its main limiting factors. > > > >> This is understood and the intention would be for the Xen Project > >> (aka Advisory Board) to fund a Linux Foundation employee to do this on > >> behalf of the Xen Project: this is a bit like Greg KH and others being > >> LF employees working on the kernel. Some vendors on the Advisory Board > >> indicated that providing Colo/hosting space and HW would be possible in > >> principle, which could help keeping the cost manageable. > > We should certainly be taking them up on those offers IMHO. > > > >> == Access => >> > >> Any central system, has of course the issue of access control and > >> managing users. This is obviously a barrier to entry (if we do not have > >> also a local test mechanism). Am wondering how other FOSS communities > >> handle this. This should certainly be the job of the Test Framework > >> owner (see above). > > At a minimum it ought to be possible to allow access to any employee of > > a project member, since we have the opportunity through the membership > > process to put whatever paperwork and agreements (acceptable use etc) in > > place. > > > > Unfettered access for anyone who rocks up and asks is a bit trickier. > > I'm quite happy to let that be the framework owner's problem ;-) > > > > Ian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel