Hi xen-devel, I''m pleased to announce the availability of the Xen ocaml tools repository. This contains for now, a fully rewritten version of xenstored, which should be better than the C version available in the xen repository. = License This is part of an effort to try to opensource some Citrix internals things that the open-source community could use and/or find useful. As such all the code in this repository are licensed under the LGPLv2 with ocaml static exception as common in the ocaml world. Please see the LICENSE file. = Improvements (compared to C xenstored) * scalability: transactions have a much better behavior in this version leading to significant improvements during aggressive use of xenstored and preventing possible guest xenstored DoS. * no disk database: everything is cheaply stored in memory. * small code base in a functional language. = Getting the source as easy as: hg clone http://xenbits.xensource.com/ext/xen-ocaml-tools.hg = Building You need to have a built xen, and a working ocaml compiler (native and bytecode). if you don''t have xen installed in /, you need to set your env XEN_DIST_ROOT to the xen dist path (on bourne shell export XEN_DIST_ROOT=/path/to/my/built/xen/dist/) once you''re there, typing make will build the binaries. = Installing ocaml xenstored is a drop in replacement of C xenstored, replacing /usr/sbin/xenstored by the ocaml one will just works. You also need to copy xenstored.conf in the xenstored directory on the destination at /etc/xen/. Once you did that, you need to restart your host (just like if you would want to restart C xenstored). = Any problems well, let us know, Thomas (CCed) and I, we''ll be happy to help you solve problems that may arise. Cheers, -- Vincent Hanquez _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''m really excited to see somebody else working on an OCaml XenStore! I was wondering if you could tell me what the difference are between this implementation and the one I recently released to the community? I think it would be great if we could work together to merge our work so we can get the best of both worlds. Patrick Vincent Hanquez wrote:> Hi xen-devel, > > I''m pleased to announce the availability of the Xen ocaml tools > repository. This contains for now, a fully rewritten version of xenstored, > which should be better than the C version available in the xen repository. > > = License > > This is part of an effort to try to opensource some Citrix internals > things that the open-source community could use and/or find useful. As such > all the code in this repository are licensed under the LGPLv2 with ocaml > static exception as common in the ocaml world. Please see the LICENSE file. > > = Improvements (compared to C xenstored) > > * scalability: transactions have a much better behavior in this version > leading to significant improvements during aggressive use of xenstored > and preventing possible guest xenstored DoS. > * no disk database: everything is cheaply stored in memory. > * small code base in a functional language. > > = Getting the source > > as easy as: hg clone http://xenbits.xensource.com/ext/xen-ocaml-tools.hg > > = Building > > You need to have a built xen, and a working ocaml compiler (native and > bytecode). if you don''t have xen installed in /, you need to set > your env XEN_DIST_ROOT to the xen dist path (on bourne shell > export XEN_DIST_ROOT=/path/to/my/built/xen/dist/) > > once you''re there, typing make will build the binaries. > > = Installing > > ocaml xenstored is a drop in replacement of C xenstored, replacing > /usr/sbin/xenstored by the ocaml one will just works. > You also need to copy xenstored.conf in the xenstored directory on the > destination at /etc/xen/. > > Once you did that, you need to restart your host > (just like if you would want to restart C xenstored). > > = Any problems > > well, let us know, Thomas (CCed) and I, we''ll be happy to help you solve > problems that may arise. > > Cheers,_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Patrick Colp wrote:> I''m really excited to see somebody else working on an OCaml XenStore! I > was wondering if you could tell me what the difference are between this > implementation and the one I recently released to the community? >this is a bit hard to tell without testing your version. but i think the main difference is the way we handle transactions, which should provide a stable average time to commit transactions when having lots of xenstore traffic from guests.> I think it would be great if we could work together to merge our work so > we can get the best of both worlds. >yes, it would be nice to not have 2 needlessly "competing" versions. hopefully we can make that happens at some point. -- Vincent _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vincent Hanquez wrote:> Patrick Colp wrote: >> I''m really excited to see somebody else working on an OCaml XenStore! >> I was wondering if you could tell me what the difference are between >> this implementation and the one I recently released to the community? >> > this is a bit hard to tell without testing your version. > > but i think the main difference is the way we handle transactions, which > should provide a stable average time to commit transactions when having > lots of xenstore traffic from guests.I think you''re thinking of my initial release last year. The version I released a few months ago also has an in-memory store and greatly improved transactions. It was motivated by the need to survive things like DoS attacks. I wrote a little attack program (in OCaml) which runs from any DomU and brought the original xenstored to its knees. With the attack going, it''s impossible to bring a new domain up -- it just hangs forever attempting to bring it up. Basically, the attack just hammers xenstored with micro-transactions. With the original transaction system, which allows the first committing transaction in a generation to win, long transactions could never complete. I implemented transactions that would enable all concurrent but non-conflicting transactions to commit. This made my version of xenstored resilient to the attack. I played around with this with your version too, but found that, while it would not hang forever while attempting to load a domain, it would instead die after a few seconds with the following error: Error: (2, ''No such file or directory'') I tried with with the eagain mode thing (random dropping of 1/3 of all transactions) both enabled and disabled, but it had the same effect (except that with the mode enabled, 1/3 of all transactions would fail regardless of if they should or not). I''ve been reading over your code and noticed that you seem to have a mini-implementation of libxc. I was wondering why you chose to do this over using the pre-existing libxenctrl? Does this make the final executable smaller? Patrick _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''ve cleaned up the attack code (and added copy right info), Makefile, and README a bit for the attack. Please find the revised version attached. Patrick Patrick Colp wrote:> Vincent Hanquez wrote: >> Patrick Colp wrote: >>> I''m really excited to see somebody else working on an OCaml XenStore! >>> I was wondering if you could tell me what the difference are between >>> this implementation and the one I recently released to the community? >>> >> this is a bit hard to tell without testing your version. >> >> but i think the main difference is the way we handle transactions, >> which should provide a stable average time to commit transactions when >> having lots of xenstore traffic from guests. > > I think you''re thinking of my initial release last year. The version I > released a few months ago also has an in-memory store and greatly > improved transactions. It was motivated by the need to survive things > like DoS attacks. > > I wrote a little attack program (in OCaml) which runs from any DomU and > brought the original xenstored to its knees. With the attack going, it''s > impossible to bring a new domain up -- it just hangs forever attempting > to bring it up. Basically, the attack just hammers xenstored with > micro-transactions. With the original transaction system, which allows > the first committing transaction in a generation to win, long > transactions could never complete. I implemented transactions that would > enable all concurrent but non-conflicting transactions to commit. This > made my version of xenstored resilient to the attack. > > I played around with this with your version too, but found that, while > it would not hang forever while attempting to load a domain, it would > instead die after a few seconds with the following error: > > Error: (2, ''No such file or directory'') > > I tried with with the eagain mode thing (random dropping of 1/3 of all > transactions) both enabled and disabled, but it had the same effect > (except that with the mode enabled, 1/3 of all transactions would fail > regardless of if they should or not). > > I''ve been reading over your code and noticed that you seem to have a > mini-implementation of libxc. I was wondering why you chose to do this > over using the pre-existing libxenctrl? Does this make the final > executable smaller? > > > Patrick > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Patrick Colp wrote:> > I think you''re thinking of my initial release last year. The version I released > a few months ago also has an in-memory store and greatly improved transactions. > It was motivated by the need to survive things like DoS attacks. >Is that possible to find your version of xenstored in a tarball somewhere ? attack.tar.gz seems to contains lots of things related to xenstored, but yet seems to missing watches and permissions.> I wrote a little attack program (in OCaml) which runs from any DomU and brought > the original xenstored to its knees. With the attack going, it''s impossible to > bring a new domain up -- it just hangs forever attempting to bring it up. > Basically, the attack just hammers xenstored with micro-transactions. With the > original transaction system, which allows the first committing transaction in a > generation to win, long transactions could never complete. I implemented > transactions that would enable all concurrent but non-conflicting transactions > to commit. This made my version of xenstored resilient to the attack. > > I played around with this with your version too, but found that, while it would > not hang forever while attempting to load a domain, it would instead die after a > few seconds with the following error: > > Error: (2, ''No such file or directory'') >i haven''t really had time to look yet (i''ve been swamped with others things), but will try to run your program. but what is dying in the scenario you described ? ocaml xenstored or the attack program ?> I tried with with the eagain mode thing (random dropping of 1/3 of all > transactions) both enabled and disabled, but it had the same effect (except that > with the mode enabled, 1/3 of all transactions would fail regardless of if they > should or not). > > I''ve been reading over your code and noticed that you seem to have a > mini-implementation of libxc. I was wondering why you chose to do this over > using the pre-existing libxenctrl? Does this make the final executable smaller? >libxenctrl is using the GPL license, whereas we chose the LGPL license with ocaml static exception everywhere. -- Vincent _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vincent Hanquez wrote:> Patrick Colp wrote: >> >> I think you''re thinking of my initial release last year. The version I >> released a few months ago also has an in-memory store and greatly >> improved transactions. It was motivated by the need to survive things >> like DoS attacks. >> > Is that possible to find your version of xenstored in a tarball somewhere ? > attack.tar.gz seems to contains lots of things related to xenstored, but > yet seems to missing watches and permissions.The attack code utilises some of the code from the version of XenStore I wrote. I released a patch for it to the list as well as a link to my website where the code available as a gzip and bzip2: website: http://cs.ubc.ca/~pjcolp/ bzip2 direct link: http://cs.ubc.ca/~pjcolp/xenstore-ocaml.tar.bz2 gzip direct link: http://cs.ubc.ca/~pjcolp/xenstore-ocaml.tar.gz I''m attaching the gzip version to this e-mail as well. The code is designed to be compiled against Xen as a replacement to the C version. I put it in the tools/xenstore directory in the xen-unstable tree to compile it.>> I wrote a little attack program (in OCaml) which runs from any DomU >> and brought the original xenstored to its knees. With the attack >> going, it''s impossible to bring a new domain up -- it just hangs >> forever attempting to bring it up. Basically, the attack just hammers >> xenstored with micro-transactions. With the original transaction >> system, which allows the first committing transaction in a generation >> to win, long transactions could never complete. I implemented >> transactions that would enable all concurrent but non-conflicting >> transactions to commit. This made my version of xenstored resilient to >> the attack. >> >> I played around with this with your version too, but found that, while >> it would not hang forever while attempting to load a domain, it would >> instead die after a few seconds with the following error: >> >> Error: (2, ''No such file or directory'') >> > i haven''t really had time to look yet (i''ve been swamped with others > things), but will try to run your program. > but what is dying in the scenario you described ? ocaml xenstored or the > attack program ?When the attack is run, neither the attack nor xenstored die, but while the attack is running it is impossible to start a domain. In the C version, it would hang forever attempting to start a domain. When I ran it against your OCaml version, it wouldn''t hang but instead after a few seconds the domain start would quit with the error: Error: (2, ''No such file or directory'') If you''re interested, I think it would be great to trying to merge the two XenStores together to get the best of both worlds. Are you going to be at Xen Summit? If so, would you be interested in chatting about this in person? Patrick _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Feb 17, 2009 at 12:01 AM, Vincent Hanquez <vincent.hanquez@eu.citrix.com> wrote:> Patrick Colp wrote: >> >> I think you''re thinking of my initial release last year. The version I >> released a few months ago also has an in-memory store and greatly improved >> transactions. It was motivated by the need to survive things like DoS >> attacks. >> > > Is that possible to find your version of xenstored in a tarball somewhere ? > attack.tar.gz seems to contains lots of things related to xenstored, but yet > seems to missing watches and permissions. >> >> I wrote a little attack program (in OCaml) which runs from any DomU and >> brought the original xenstored to its knees. With the attack going, it''s >> impossible to bring a new domain up -- it just hangs forever attempting to >> bring it up. Basically, the attack just hammers xenstored with >> micro-transactions. With the original transaction system, which allows the >> first committing transaction in a generation to win, long transactions could >> never complete. I implemented transactions that would enable all concurrent >> but non-conflicting transactions to commit. This made my version of >> xenstored resilient to the attack. >> >> I played around with this with your version too, but found that, while it >> would not hang forever while attempting to load a domain, it would instead >> die after a few seconds with the following error: >> >> Error: (2, ''No such file or directory'') >> > > i haven''t really had time to look yet (i''ve been swamped with others > things), but will try to run your program. > but what is dying in the scenario you described ? ocaml xenstored or the > attack program ? >> >> I tried with with the eagain mode thing (random dropping of 1/3 of all >> transactions) both enabled and disabled, but it had the same effect (except >> that with the mode enabled, 1/3 of all transactions would fail regardless of >> if they should or not). >> >> I''ve been reading over your code and noticed that you seem to have a >> mini-implementation of libxc. I was wondering why you chose to do this over >> using the pre-existing libxenctrl? Does this make the final executable >> smaller? >> > > libxenctrl is using the GPL license, whereas we chose the LGPL license with > ocaml static exception everywhere.This part is interesting! I have some doubts here: for ex, now I want to reimplement some parts of libxenstore in BSD or LGPL. But I cannot avoid to include "/usr/include/{xs,xenctrl|.h", because they have some must-have data definition. But it seems all these headers are in GPL, so my code violates the copyright! Or not?? Thanks, Jun _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 17/02/2009 01:19, "Jun Koi" <junkoi2004@gmail.com> wrote:>> libxenctrl is using the GPL license, whereas we chose the LGPL license with >> ocaml static exception everywhere. > > This part is interesting! > > I have some doubts here: for ex, now I want to reimplement some parts > of libxenstore in BSD or LGPL. But I cannot avoid to include > "/usr/include/{xs,xenctrl|.h", because they have some must-have data > definition. But it seems all these headers are in GPL, so my code > violates the copyright! Or not??I think as long as the boilerplate and formatting is your own, the actual meat of the struct definitions you could take and relicense without really any issue. Since as you say there is really no other way those definitions could be written. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel