Where is the best place to read about tmem hypercall abi? Should I start with the Linux implementation? Thanks James
On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote:> Where is the best place to read about tmem hypercall abi? Should I start with the Linux implementation? >I''m not really answering your question here.. but if you didn''t see these links, take a look, they might help: http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html -- Pasi
> On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote: > > Where is the best place to read about tmem hypercall abi? Should I start > > with the Linux implementation? > > > > I''m not really answering your question here.. but if you didn''t see these links, > take a look, they might help: > http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html >That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can''t find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the interface is fairly thin though, and I have a few Windows related problems to solve first before I can even think about making use of tmem. Thanks James
On Tue, May 28, 2013 at 12:26:15AM +0000, James Harper wrote:> > On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote: > > > Where is the best place to read about tmem hypercall abi? Should I start > > > with the Linux implementation? > > > > > > > I''m not really answering your question here.. but if you didn''t see these links, > > take a look, they might help: > > http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html > > > > That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can''t find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the interface is fairly thin though, and I have a few Windows related problems to solve first before I can even think about making use of tmem.Let''s CC Dan here. He has a stash of documents on that. But looking at the include/public/tmem.h it is very slim on comments. The ones that are of interest to you (I think) are: TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE.> > Thanks > > James > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
> From: Konrad Rzeszutek Wilk > Subject: Re: [Xen-devel] tmem docs > > On Tue, May 28, 2013 at 12:26:15AM +0000, James Harper wrote: > > > On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote: > > > > Where is the best place to read about tmem hypercall abi? Should I start > > > > with the Linux implementation? > > > > > > I''m not really answering your question here.. but if you didn''t see these links, > > > take a look, they might help: > > > http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html > > > > That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can''t > find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the > interface is fairly thin though, and I have a few Windows related problems to solve first before I can > even think about making use of tmem. > > Let''s CC Dan here. He has a stash of documents on that. > > But looking at the include/public/tmem.h it is very slim on comments. The ones > that are of interest to you (I think) are: > > TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE. > > > Thanks > > JamesHi James -- I''ll be working for a few more weeks and would be happy to answer any questions I can. However, I am no longer on the xen-devel email list so you will need to cc me directly. I''m gathering you are trying to connect a Windows kernel to the Xen tmem ABI? There have been various ideas proposed about connecting up Linux _userland_ apps to tmem and it may be possible to do that for Windows as well. However, the existing ABI is probably best suited for a guest kernel, and I''ve never had access to Windows kernel source so I''m not in a position to judge how easy or how hard it would be to modify Windows kernel source to adapt it. KY Srinivasan is currently working on cross-guest memory capacity optimization for Linux on HyperV and has expressed some interest in tmem for Windows so you may want to contact him to see if he''s given it any thought. (kys@microsoft.com) KY was an early Xen contributor also. The tmem ABI spec is here: https://oss.oracle.com/projects/tmem/dist/documentation/api/tmemspec-v001.pdf As you can see by the date, that spec version is very old (2008) but, on quick scan, it is still 99% accurate. The main difference I see is that (page 2, line 18) object identifiers are now 192-bit instead of 64-bit. See reason for that change here: http://lists.xen.org/archives/html/xen-devel/2010-09/msg00069.html> But looking at the include/public/tmem.h it is very slim on comments. The ones > that are of interest to you (I think) are: > TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE.Also, TMEM_FLUSH_PAGE and TMEM_FLUSH_OBJECT. Note that TMEM_READ, TMEM_WRITE, TMEM_XCHG, and TMEM_NEW_PAGE were never used by Linux. Also TMEM_CONTROL is only described by the implementation... it is only used by tools in dom0 so you may not care about it for Windows. And you can ignore shared+persistent pools (never implemented) and, unless there is some cool Windows cluster-filesystem, you can ignore shared+ephemeral pools too (only implemented on linux for ocfs2). The core of the linux implementation of the ABI can be found in http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/xen/tmem.c Look for the functions with name starting "xen_tmem_" This is where the Linux concepts "cleancache" and "frontswap" get translated into the Xen tmem ABI.>> interface is fairly thin thoughYes, everything funnels through a single hypercall.>> I have a few Windows related problems to solve first before I canLet me know if/how I can help! The hardest part is probably proper placement of flush calls into the OS kernel memory management code to ensure coherency. Dan P.S....> > That all tells me about how great tmem isSorry about that... I had to spend far far too much time selling the value of tmem at the cost of insufficient time improving it. :-( And I suck at sales :-(
On Tue, May 28, 2013 at 09:20:28AM -0700, Dan Magenheimer wrote:> > From: Konrad Rzeszutek Wilk > > Subject: Re: [Xen-devel] tmem docs > > > > On Tue, May 28, 2013 at 12:26:15AM +0000, James Harper wrote: > > > > On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote: > > > > > Where is the best place to read about tmem hypercall abi? Should I start > > > > > with the Linux implementation? > > > > > > > > I''m not really answering your question here.. but if you didn''t see these links, > > > > take a look, they might help: > > > > http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html > > > > > > That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can''t > > find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the > > interface is fairly thin though, and I have a few Windows related problems to solve first before I can > > even think about making use of tmem. > > > > Let''s CC Dan here. He has a stash of documents on that. > > > > But looking at the include/public/tmem.h it is very slim on comments. The ones > > that are of interest to you (I think) are: > > > > TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE. > > > > > Thanks > > > James > > Hi James -- > > I''ll be working for a few more weeks and would be happy > to answer any questions I can. However, I am no longer on the > xen-devel email list so you will need to cc me directly. > > I''m gathering you are trying to connect a Windows kernel to the > Xen tmem ABI? There have been various ideas proposed about > connecting up Linux _userland_ apps to tmem and it may be possible > to do that for Windows as well. However, the existing ABI > is probably best suited for a guest kernel, and I''ve never > had access to Windows kernel source so I''m not in a position > to judge how easy or how hard it would be to modify Windows > kernel source to adapt it. KY Srinivasan is currently working > on cross-guest memory capacity optimization for Linux on HyperV > and has expressed some interest in tmem for Windows so you > may want to contact him to see if he''s given it any thought. > (kys@microsoft.com) KY was an early Xen contributor also. > > The tmem ABI spec is here: > > https://oss.oracle.com/projects/tmem/dist/documentation/api/tmemspec-v001.pdfAwesome.> > As you can see by the date, that spec version is very old (2008) > but, on quick scan, it is still 99% accurate. The main difference > I see is that (page 2, line 18) object identifiers are now > 192-bit instead of 64-bit. See reason for that change here: > http://lists.xen.org/archives/html/xen-devel/2010-09/msg00069.html >Is there a source for the spec so I can modify it (And also put the spec in the Xen codebase).