Xen devel - Mar 2012 - Transcendent Memory ("tmem") -capable kernel now publicly released

Transcendent Memory ("tmem") [1] is a new approach to optimizing RAM
utilization
in a virtualized (and, in some cases, a physical) environment

Support for tmem has been in the Xen hypervisor since the 4.0 release, but
the tmem protocols require cooperation between the hypervisor and guest
OS kernel.  While the guest-side kernel changes are relatively simple
and non-intrusive, getting any new technology into any operating system
can be, shall we say, challenging... :-)  BUT...

Last week, Oracle released "Oracle Unbreakable Enterprise Kernel Release
2",
the first publicly-available fully-supported[2] Linux kernel implementing all
Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral tmem
pools), "frontswap" (persistent tmem pools), and in-kernel support for
"self-ballooning".

So now, finally, it should be easy to give tmem a try!

Brief How-To:
1) Install [3] the UEK2 kernel (upgrade or co-existing) in your existing
   RHEL5 or RHEL6 (or Oracle Linux 5 or 6) PV or HVM guest(s).
2) Add the "tmem" kernel boot parameter in your guest''s
grub.conf.
3) Add the "tmem" Xen boot parameter on the xen.gz line in
dom0''s grub.conf.
   (Optionally add "tmem_compress" and "tmem_dedup"
depending on your
    environment and testing plans.)
4) Reboot Xen and launch your guest(s).  Check to ensure the guests are
    running UEK2.
5) Start a workload on your guest(s).
6) Use xentop to watch the fun!  (Don''t forget to press "t"
in xentop for
   more detail.)

Thanks,
Dan Magenheimer

[1] For more information on tmem:

    Technical overview: http://oss.oracle.com/projects/tmem 
    Xen Summit 2010 (including performance analysis): 
http://www.slideshare.net/xen_com_mgr/transcendent-memoryupdate-xensummit2010final-3947783
    Tmem without virtualization: http://lwn.net/Articles/454795/ and
http://www.linux-kvm.org/wiki/images/d/d7/TmemNotVirt-Linuxcon2011-Final.pdf

[2] At this time, the tmem capability in UEK2 is officially a
    "Technology Preview" so is not recommended for production use.

[3] For more information on UEK2 including installation:

    UEK2 install/getting started:
http://www.oracle.com/technetwork/articles/servers-storage-admin/uek-rel2-getting-started-1555632.html

    UEK2 Release notes:
http://oss.oracle.com/ol6/docs/RELEASE-NOTES-UEK2-en.html

    Press release: http://www.oracle.com/us/corporate/press/1555025

Transcendent Memory ("tmem") [1] is a new approach to optimizing RAM
utilization
in a virtualized (and, in some cases, a physical) environment

Support for tmem has been in the Xen hypervisor since the 4.0 release, but
the tmem protocols require cooperation between the hypervisor and guest
OS kernel.  While the guest-side kernel changes are relatively simple
and non-intrusive, getting any new technology into any operating system
can be, shall we say, challenging... :-)  BUT...

Last week, Oracle released "Oracle Unbreakable Enterprise Kernel Release
2",
the first publicly-available fully-supported[2] Linux kernel implementing all
Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral tmem
pools), "frontswap" (persistent tmem pools), and in-kernel support for
"self-ballooning".

So now, finally, it should be easy to give tmem a try!

Brief How-To:
1) Install [3] the UEK2 kernel (upgrade or co-existing) in your existing
   RHEL5 or RHEL6 (or Oracle Linux 5 or 6) PV or HVM guest(s).
2) Add the "tmem" kernel boot parameter in your guest''s
grub.conf.
3) Add the "tmem" Xen boot parameter on the xen.gz line in
dom0''s grub.conf.
   (Optionally add "tmem_compress" and "tmem_dedup"
depending on your
    environment and testing plans.)
4) Reboot Xen and launch your guest(s).  Check to ensure the guests are
    running UEK2.
5) Start a workload on your guest(s).
6) Use xentop to watch the fun!  (Don''t forget to press "t"
in xentop for
   more detail.)

Thanks,
Dan Magenheimer

[1] For more information on tmem:

    Technical overview: http://oss.oracle.com/projects/tmem 
    Xen Summit 2010 (including performance analysis): 
http://www.slideshare.net/xen_com_mgr/transcendent-memoryupdate-xensummit2010final-3947783
    Tmem without virtualization: http://lwn.net/Articles/454795/ and
http://www.linux-kvm.org/wiki/images/d/d7/TmemNotVirt-Linuxcon2011-Final.pdf

[2] At this time, the tmem capability in UEK2 is officially a
    "Technology Preview" so is not recommended for production use.

[3] For more information on UEK2 including installation:

    UEK2 install/getting started:
http://www.oracle.com/technetwork/articles/servers-storage-admin/uek-rel2-getting-started-1555632.html

    UEK2 Release notes:
http://oss.oracle.com/ol6/docs/RELEASE-NOTES-UEK2-en.html

    Press release: http://www.oracle.com/us/corporate/press/1555025

On 23 March 2012 04:22, Dan Magenheimer <dan.magenheimer@oracle.com>
wrote:
> Transcendent Memory ("tmem") [1] is a new approach to optimizing
RAM utilization
> in a virtualized (and, in some cases, a physical) environment
>
> Support for tmem has been in the Xen hypervisor since the 4.0 release, but
> the tmem protocols require cooperation between the hypervisor and guest
> OS kernel.  While the guest-side kernel changes are relatively simple
> and non-intrusive, getting any new technology into any operating system
> can be, shall we say, challenging... :-)  BUT...
>
> Last week, Oracle released "Oracle Unbreakable Enterprise Kernel
Release 2",
> the first publicly-available fully-supported[2] Linux kernel implementing
all
> Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral tmem
> pools), "frontswap" (persistent tmem pools), and in-kernel
support for
> "self-ballooning".
>
> So now, finally, it should be easy to give tmem a try!
>
> Brief How-To:
> 1) Install [3] the UEK2 kernel (upgrade or co-existing) in your existing
>   RHEL5 or RHEL6 (or Oracle Linux 5 or 6) PV or HVM guest(s).
> 2) Add the "tmem" kernel boot parameter in your guest''s
grub.conf.
> 3) Add the "tmem" Xen boot parameter on the xen.gz line in
dom0''s grub.conf.
>   (Optionally add "tmem_compress" and "tmem_dedup"
depending on your
>    environment and testing plans.)
> 4) Reboot Xen and launch your guest(s).  Check to ensure the guests are
>    running UEK2.
> 5) Start a workload on your guest(s).
> 6) Use xentop to watch the fun!  (Don''t forget to press
"t" in xentop for
>   more detail.)
>
> Thanks,
> Dan Magenheimer
>
> [1] For more information on tmem:
>
>    Technical overview: http://oss.oracle.com/projects/tmem
>    Xen Summit 2010 (including performance analysis):
>
http://www.slideshare.net/xen_com_mgr/transcendent-memoryupdate-xensummit2010final-3947783
>    Tmem without virtualization: http://lwn.net/Articles/454795/ and
http://www.linux-kvm.org/wiki/images/d/d7/TmemNotVirt-Linuxcon2011-Final.pdf
>
> [2] At this time, the tmem capability in UEK2 is officially a
>    "Technology Preview" so is not recommended for production use.
>
> [3] For more information on UEK2 including installation:
>
>    UEK2 install/getting started:
http://www.oracle.com/technetwork/articles/servers-storage-admin/uek-rel2-getting-started-1555632.html
>
>    UEK2 Release notes:
http://oss.oracle.com/ol6/docs/RELEASE-NOTES-UEK2-en.html
>
>    Press release: http://www.oracle.com/us/corporate/press/1555025
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
Hi Dan.

Is UEK2 configured with dom0 support by default or does one have to
download the sources and recompile with appropriate options?

Joseph.

-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846

On 23 March 2012 04:22, Dan Magenheimer <dan.magenheimer@oracle.com>
wrote:
> Transcendent Memory ("tmem") [1] is a new approach to optimizing
RAM utilization
> in a virtualized (and, in some cases, a physical) environment
>
> Support for tmem has been in the Xen hypervisor since the 4.0 release, but
> the tmem protocols require cooperation between the hypervisor and guest
> OS kernel.  While the guest-side kernel changes are relatively simple
> and non-intrusive, getting any new technology into any operating system
> can be, shall we say, challenging... :-)  BUT...
>
> Last week, Oracle released "Oracle Unbreakable Enterprise Kernel
Release 2",
> the first publicly-available fully-supported[2] Linux kernel implementing
all
> Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral tmem
> pools), "frontswap" (persistent tmem pools), and in-kernel
support for
> "self-ballooning".
>
> So now, finally, it should be easy to give tmem a try!
>
> Brief How-To:
> 1) Install [3] the UEK2 kernel (upgrade or co-existing) in your existing
>   RHEL5 or RHEL6 (or Oracle Linux 5 or 6) PV or HVM guest(s).
> 2) Add the "tmem" kernel boot parameter in your guest''s
grub.conf.
> 3) Add the "tmem" Xen boot parameter on the xen.gz line in
dom0''s grub.conf.
>   (Optionally add "tmem_compress" and "tmem_dedup"
depending on your
>    environment and testing plans.)
> 4) Reboot Xen and launch your guest(s).  Check to ensure the guests are
>    running UEK2.
> 5) Start a workload on your guest(s).
> 6) Use xentop to watch the fun!  (Don''t forget to press
"t" in xentop for
>   more detail.)
>
> Thanks,
> Dan Magenheimer
>
> [1] For more information on tmem:
>
>    Technical overview: http://oss.oracle.com/projects/tmem
>    Xen Summit 2010 (including performance analysis):
>
http://www.slideshare.net/xen_com_mgr/transcendent-memoryupdate-xensummit2010final-3947783
>    Tmem without virtualization: http://lwn.net/Articles/454795/ and
http://www.linux-kvm.org/wiki/images/d/d7/TmemNotVirt-Linuxcon2011-Final.pdf
>
> [2] At this time, the tmem capability in UEK2 is officially a
>    "Technology Preview" so is not recommended for production use.
>
> [3] For more information on UEK2 including installation:
>
>    UEK2 install/getting started:
http://www.oracle.com/technetwork/articles/servers-storage-admin/uek-rel2-getting-started-1555632.html
>
>    UEK2 Release notes:
http://oss.oracle.com/ol6/docs/RELEASE-NOTES-UEK2-en.html
>
>    Press release: http://www.oracle.com/us/corporate/press/1555025
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
Hi Dan.

Is UEK2 configured with dom0 support by default or does one have to
download the sources and recompile with appropriate options?

Joseph.

-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846

> From: Joseph Glanville [mailto:joseph.glanville@orionvm.com.au]
> Subject: Re: [Xen-devel] [Xen-users] Transcendent Memory ("tmem")
-capable kernel now publicly
> released
> 
> On 23 March 2012 04:22, Dan Magenheimer <dan.magenheimer@oracle.com>
wrote:
> > Transcendent Memory ("tmem") [1] is a new approach to
optimizing RAM utilization
> > in a virtualized (and, in some cases, a physical) environment
> >
> > Support for tmem has been in the Xen hypervisor since the 4.0 release,
but
> > the tmem protocols require cooperation between the hypervisor and
guest
> > OS kernel.  While the guest-side kernel changes are relatively simple
> > and non-intrusive, getting any new technology into any operating
system
> > can be, shall we say, challenging... :-)  BUT...
> >
> > Last week, Oracle released "Oracle Unbreakable Enterprise Kernel
Release 2",
> > the first publicly-available fully-supported[2] Linux kernel
implementing all
> > Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral tmem
> > pools), "frontswap" (persistent tmem pools), and in-kernel
support for
> > "self-ballooning".
> >
> > So now, finally, it should be easy to give tmem a try!
> >
> > Brief How-To:
> 
> Hi Dan.
> 
> Is UEK2 configured with dom0 support by default or does one have to
> download the sources and recompile with appropriate options?
> 
> Joseph.
Hi Joseph --

Since UEK2 is based on Linux 3.0, I think it has all the support
necessary to be used as a dom0.  However I don''t think all the
latest post-3.0 bug fixes and performance fixes for dom0 support have
been back-ported (yet).  Let me check and get back to you (or
see if another UEK2 expert will reply on-list).

In any case, the brief tmem how-to should probably have stated that
the tmem feature is primarily for use in the hypervisor and guests,
NOT dom0.  Turning it on in dom0 may have some value for Xen
systems that primarily use "file:" for disk access (i.e. are
trading off potential data loss on a system crash for higher
performance), but we don''t really test that configuration.

Dan

> From: Joseph Glanville [mailto:joseph.glanville@orionvm.com.au]
> Subject: Re: [Xen-devel] [Xen-users] Transcendent Memory ("tmem")
-capable kernel now publicly
> released
> 
> On 23 March 2012 04:22, Dan Magenheimer <dan.magenheimer@oracle.com>
wrote:
> > Transcendent Memory ("tmem") [1] is a new approach to
optimizing RAM utilization
> > in a virtualized (and, in some cases, a physical) environment
> >
> > Support for tmem has been in the Xen hypervisor since the 4.0 release,
but
> > the tmem protocols require cooperation between the hypervisor and
guest
> > OS kernel.  While the guest-side kernel changes are relatively simple
> > and non-intrusive, getting any new technology into any operating
system
> > can be, shall we say, challenging... :-)  BUT...
> >
> > Last week, Oracle released "Oracle Unbreakable Enterprise Kernel
Release 2",
> > the first publicly-available fully-supported[2] Linux kernel
implementing all
> > Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral tmem
> > pools), "frontswap" (persistent tmem pools), and in-kernel
support for
> > "self-ballooning".
> >
> > So now, finally, it should be easy to give tmem a try!
> >
> > Brief How-To:
> 
> Hi Dan.
> 
> Is UEK2 configured with dom0 support by default or does one have to
> download the sources and recompile with appropriate options?
> 
> Joseph.
Hi Joseph --

Since UEK2 is based on Linux 3.0, I think it has all the support
necessary to be used as a dom0.  However I don''t think all the
latest post-3.0 bug fixes and performance fixes for dom0 support have
been back-ported (yet).  Let me check and get back to you (or
see if another UEK2 expert will reply on-list).

In any case, the brief tmem how-to should probably have stated that
the tmem feature is primarily for use in the hypervisor and guests,
NOT dom0.  Turning it on in dom0 may have some value for Xen
systems that primarily use "file:" for disk access (i.e. are
trading off potential data loss on a system crash for higher
performance), but we don''t really test that configuration.

Dan

Dan,

2012/3/22 Dan Magenheimer
<dan.magenheimer@oracle.com>:
> Transcendent Memory ("tmem") [1] is a new approach to optimizing
RAM utilization
> in a virtualized (and, in some cases, a physical) environment
> OS kernel.  While the guest-side kernel changes are relatively simple
> and non-intrusive, getting any new technology into any operating system
> can be, shall we say, challenging... :-)  BUT...
thanks for the non-stoppable work on this!
I don''t even remember if it''s been 3 or 4 years, you have been
working
to make this finally happen.
It''s awesome to see this will now finally come to all Xen''
users benefit.

I hope many people will be using it in the future.

Everyone say
Thanks Dan!

-- 
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

Dan,

2012/3/22 Dan Magenheimer
<dan.magenheimer@oracle.com>:
> Transcendent Memory ("tmem") [1] is a new approach to optimizing
RAM utilization
> in a virtualized (and, in some cases, a physical) environment
> OS kernel.  While the guest-side kernel changes are relatively simple
> and non-intrusive, getting any new technology into any operating system
> can be, shall we say, challenging... :-)  BUT...
thanks for the non-stoppable work on this!
I don''t even remember if it''s been 3 or 4 years, you have been
working
to make this finally happen.
It''s awesome to see this will now finally come to all Xen''
users benefit.

I hope many people will be using it in the future.

Everyone say
Thanks Dan!

-- 
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

On 25 March 2012 07:46, Joseph Glanville
<joseph.glanville@orionvm.com.au>wrote:
> On 23 March 2012 04:22, Dan Magenheimer <dan.magenheimer@oracle.com>
> wrote:
> > Transcendent Memory ("tmem") [1] is a new approach to
optimizing RAM
> utilization
> > in a virtualized (and, in some cases, a physical) environment
> >
> > Support for tmem has been in the Xen hypervisor since the 4.0 release,
> but
> > the tmem protocols require cooperation between the hypervisor and
guest
> > OS kernel.  While the guest-side kernel changes are relatively simple
> > and non-intrusive, getting any new technology into any operating
system
> > can be, shall we say, challenging... :-)  BUT...
> >
> > Last week, Oracle released "Oracle Unbreakable Enterprise Kernel
Release
> 2",
> > the first publicly-available fully-supported[2] Linux kernel
> implementing all
> > Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral
> tmem
> > pools), "frontswap" (persistent tmem pools), and in-kernel
support for
> > "self-ballooning".
> >
> > So now, finally, it should be easy to give tmem a try!
>
Dan,

Thanks so much. This is great news. I just wanted to know what kind of
license does it have. Is it open source..or..? --Thank  you


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

Hi Eva --

 

(sorry to list for top-post... when responding to html messages with my mailer,
overriding this is very painful)

 

Every piece of code in tmem is entirely open source and GPL.  I''m not a
licensing expert but I think it is all GPLv2.

 

Oracle Unbreakable Enterprise Kernel Release 2 ("UEK2") runs on and is
the default kernel on Oracle Linux 5 ("OL5") and Oracle Linux 6
("OL6").  UEK2 source is published in a public git tree (unlike RH
kernels) so is freely available if you wish to examine the source for git
commits (i.e. for frontswap patches that tmem uses that are not yet upstream). 
OL5 and OL6 are clones of the RHEL equivalents (and the original RHEL kernels
are included, but are not the default booted kernels).  OL5 and OL6 are freely
available, as are all errata and updates.  See
http://blogs.oracle.com/wim/entry/setting_up_oracle_6  for more info.

 

Hi Psusi (Phillip?) -

 

Just saw your tmem post... sorry I don''t keep up with xen-users.   If
you are looking at implementing Xen tmem support on/for a future Ubuntu release,
let me know if I can help and see:
http://oss.oracle.com/git/?p=linux-2.6-unbreakable.git;a=summary

 

Thanks,

Dan

 

From: eva [mailto:evammg@gmail.com] 
Sent: Monday, April 16, 2012 4:56 AM
To: xen-users@lists.xen.org
Cc: Dan Magenheimer
Subject: Re: [Xen-users] Transcendent Memory ("tmem") -capable kernel
now publicly released

 

 

On 25 March 2012 07:46, Joseph Glanville <HYPERLINK
"mailto:joseph.glanville@orionvm.com.au"joseph.glanville@orionvm.com.au>
wrote:

On 23 March 2012 04:22, Dan Magenheimer <HYPERLINK
"mailto:dan.magenheimer@oracle.com"dan.magenheimer@oracle.com>
wrote:
> Transcendent Memory ("tmem") [1] is a new approach to optimizing
RAM utilization
> in a virtualized (and, in some cases, a physical) environment
>
> Support for tmem has been in the Xen hypervisor since the 4.0 release, but
> the tmem protocols require cooperation between the hypervisor and guest
> OS kernel.  While the guest-side kernel changes are relatively simple
> and non-intrusive, getting any new technology into any operating system
> can be, shall we say, challenging... :-)  BUT...
>
> Last week, Oracle released "Oracle Unbreakable Enterprise Kernel
Release 2",
> the first publicly-available fully-supported[2] Linux kernel implementing
all
> Xen-guest-side capabilities for tmem, including "cleancache"
(ephemeral tmem
> pools), "frontswap" (persistent tmem pools), and in-kernel
support for
> "self-ballooning".
>
> So now, finally, it should be easy to give tmem a try!
 

Dan,

 

Thanks so much. This is great news. I just wanted to know what kind of license
does it have. Is it open source..or..? --Thank  you


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 4/16/2012 12:40 PM, Dan Magenheimer wrote:
> Hi Psusi (Phillip?) - Just saw your tmem post... sorry I don''t
keep
> up with xen-users.   If you are looking at implementing Xen tmem
> support on/for a future Ubuntu release, let me know if I can help and
> see:
> http://oss.oracle.com/git/?p=linux-2.6-unbreakable.git;a=summary
The last time I played with it I think I managed to get it working after 
adding the "tmem" command line arguments to xen and the guest kernel. 
What I have not been able to do is figure out how to monitor its 
usage/effectiveness.  The output of xm tmem was completely 
indecipherable.  Could you shed some light on that?

Also I was wondering about how pages move back and forth between 
cleancache and pagecache.  Are they copied or moved back and forth?  In 
other words, is the data placed in cleancache only once the local 
pagecache discards it, or as soon as it is read from disk, and when it 
is later requested, is it copied back from cleancache to the local 
pagecache, thus resulting in the data being duplicated in memory, once 
for the guest, and once in tmem?

Hi Phillip,

I remember we exchanged an e-mail about tmem the other day. I have
investigated a little bit about tmem. First of all, the current linux
branch has tmem + cleancache upstream, but it does not contain frontswap.
Basically, file systems (ext3, ext4) decide whether it puts memory to page
cache, so it puts pages to tmem when it wants to remove pages from the
memory. This does not change the amount of memory used in the system. But
tmem support many other options about how to manage these pages such as
zcache (one kernel) and ramster (multiple kernels).

For the log, refer to the following parsing I did. It might help!

-----------------------------------------------------------------------------------

G=Tt:204155,Te:100904,Cf:0,Af:0,Pf:0,Ta:6557,Lm:0,Et:0,Ea:1,Rt:0,Ra:0,Rx:0,Fp:0,Ec:1240,Em:7597,Oc:179,Om:571,Nc:103,Nm:448,Pc:1240,Pm:7597,Fc:1240,Fm:7597,Sc:0,Sm:0,Ep:17542,Gd:0,Zt:0,Gz:0
C=CI:11,ww:0,ca:0,co:0,fr:0,Tc:9362200,Ge:445,Pp:0,Gp:0,Ec:1240,Em:1685,cp:0,cb:0,cn:0,cm:0

G= (global)
Tt:204155, (total_tmem_ops)
Te:100904, (errored_tmem_ops)
Cf:0, (failed_copies)
Af:0, (alloc_failed)
Pf:0, (alloc_page_failed)
Ta:6557, (tmh_avail_pages())
Lm:0, (low_on_memory)
Et:0, (evicted_pgs)
Ea:1, (evict_attempts)
Rt:0, (relinq_pgs)
Ra:0, (relinq_attempts)
Rx:0, (max_evicts_per_relinq)
Fp:0, (total_flush_pool)
Ec:1240, (glogal_eph_count)
Em:7597, (global_eph_count_max)
Oc:179, (global_obj_count)
Om:571, (global_obj_count_max)
Nc:103, (global_rtree_node_count)
Nm:448, (global_rtree_node_count_max)
Pc:1240, (global_pgp_count)
Pm:7597, (global_pgp_count_max)
Fc:1240, (global_page_count)
Fm:7597, (global_page_count_max)Sc:0, (global_pcd_count)
Sm:0, (global_pcd_count_max)
Ep:17542, (tot_good_eph_puts)
Gd:0, (deduped_puts)
Zt:0, (pcd_tot_tze_size)
Gz:0 (pcd_tot_csize)

-----------------------------------------------------------------------------------

S=SI:0,PT:ES,U0:f34b3f0627e0f696,U1:ff55f9cce0264db8,SC:9,SC:10,Pc:0,Pm:0,Oc:0,Om:0,Nc:0,Nm:0,ps:0,pt:0,pd:0,pr:0,px:0,gs:0,gt:0,fs:0,ft:32,os:0,ot:96

S= (shared)
SI:0, (pool_id)
PT:ES, (Ephemeral Shared)
U0:f34b3f0627e0f696,U1:ff55f9cce0264db8, (UUID)
SC:9, (cli_id)
SC:10, (cli_id)
Pc:0, (pgp_count)
Pm:0, (pgp_count_max)
Oc:0, (obj_count)
Om:0, (obj_count_max)
Nc:0, (objnode_count)
Nm:0, (objnode_count_max)
ps:0, (good_puts)
pt:0, (puts)
pd:0, (dup_puts_flushed)
pr:0, (dup_puts_replaced)
px:0, (no_mem_puts)
gs:0, (found_gets)
gt:0, (gets)
fs:0, (flushs_found)
ft:32, (flushs)
os:0, (flush_objs_found)
ot:96 (flush_objs)

-----------------------------------------------------------------------------------

C=CI:11,ww:0,ca:0,co:0,fr:0,Tc:9362200,Ge:445,Pp:0,Gp:0,Ec:1240,Em:1685,cp:0,cb:0,cn:0,cm:0

C= (client)
CI:11, (cli_id)
ww:0, (weight)
ca:0, (cap)
co:0, (compress)
fr:0, (frozen)
Tc:9362200, (total_cycles)
Ge:445, (succ_eph_gets)


Pp:0, (succ_pers_puts)
Gp:0, (succ_pers_gets)
Ec:1240, (eph_count)
Em:1685, (eph_count_max)
cp:0, (compressed_pages)
cb:0, (compressed_sum_size)
cn:0, (compress_poor)
cm:0 (compress_nomem)

-----------------------------------------------------------------------------------

P=CI:11,PI:0,PT:EP,U0:0,U1:0,Pc:1240,Pm:1685,Oc:179,Om:194,Nc:103,Nm:128,ps:1685,pt:1685,pd:0,pr:0,px:0,gs:445,gt:2817,fs:0,ft:10,os:0,ot:33

P= (pools)
CI:11, (cli_id)
PI:0, (pool_id)
PT:E (ephemeral) P (private), (reference: P (persistent) S(shared)
U0:0, (UUID)
U1:0, (UUID)
Pc:1240, (pgp_count)
Pm:1685, (pgp_count_max)
Oc:179, (obj_count)
Om:194, (obj_count_max)
Nc:103, (objnode_count)
Nm:128, (objnode_count_max)
ps:1685, (good_puts)
pt:1685, (puts)
pd:0, (dup_puts_flushed)
pr:0, (dup_puts_replaced)
px:0, (no_mem_puts)
gs:445, (found_gets)
gt:2817, (gets)
fs:0, (flushs_found)
ft:10, (flushs)
os:0, (flush_objs_found)
ot:33 (flush_objs)

-----------------------------------------------------------------------------------

T=Gn:5443,Gt:21414124,Gx:55912,Gm:1980,Pn:17542,Pt:72415084,Px:238988,Pm:2532,gn:112911,gt:29206664,gx:13368,gm:120,pn:0,pt:0,px:0,pm:2147483647,Fn:43718,Ft:10429096,Fx:5120,Fm:180,On:12262,Ot:12024416,Ox:950220,Om:180,Cn:22985,Ct:43492480,Cx:129064,Cm:964,cn:0,ct:0,cx:0,cm:2147483647,dn:0,dt:0,dx:0,dm:2147483647

T= (global_perf)
Gn:5443, (succ_get_count)
Gt:21414124, (succ_get_sum_cycles)
Gx:55912, (succ_get_max_cycles)
Gm:1980, (succ_get_min_cycles)
Pn:17542, (succ_put_count)
Pt:72415084, (succ_put_sum_cycles)
Px:238988, (succ_put_max_cycles)
Pm:2532, (succ_put_min_cycles)
gn:112911, (non_succ_get_count)
gt:29206664, (non_succ_get_sum_cycles)
gx:13368, (non_succ_get_max_cycles)
gm:120, (non_succ_get_min_cycles)
pn:0, (non_succ_put_count)
pt:0, (non_succ_put_sum_cycles)
px:0, (non_succ_put_max_cycles)
pm:2147483647, (non_succ_put_min_cycles)
Fn:43718, (flush_count)
Ft:10429096, (flush_sum_cycles)
Fx:5120, (flush_max_cycles)
Fm:180, (flush_min_cycles)
On:12262, (flush_obj_count)
Ot:12024416, (flush_obj_sum_cycles)
Ox:950220, (flush_obj_max_cycles)
Om:180, (flush_obj_min_cycles)
Cn:22985, (pg_copy_count)
Ct:43492480, (pg_copy_sum_cycles)
Cx:129064, (pg_copy_max_cycles)
Cm:964, (pg_copy_min_cycles)
cn:0, (compress_count)
ct:0, (compress_sum_cycles)
cx:0, (compress_max_cycles)
cm:2147483647, (compress_min_cycles)
dn:0, (decmopress_count)
dt:0, (decompress_sum_cycles)
dx:0, (decompress_max_cycles)
dm:2147483647 (decompress_min_cycles)

-----------------------------------------------------------------------------------

Good luck!

Jinho

On Mon, Apr 16, 2012 at 2:41 PM, Phillip Susi <psusi@ubuntu.com> wrote:
> On 4/16/2012 12:40 PM, Dan Magenheimer wrote:
>
>> Hi Psusi (Phillip?) - Just saw your tmem post... sorry I don''t
keep
>>
>> up with xen-users.   If you are looking at implementing Xen tmem
>> support on/for a future Ubuntu release, let me know if I can help and
>> see:
>>
http://oss.oracle.com/git/?p=**linux-2.6-unbreakable.git;a=**summary<http://oss.oracle.com/git/?p=linux-2.6-unbreakable.git;a=summary>
>>
>
> The last time I played with it I think I managed to get it working after
> adding the "tmem" command line arguments to xen and the guest
kernel. What
> I have not been able to do is figure out how to monitor its
> usage/effectiveness.  The output of xm tmem was completely indecipherable.
>  Could you shed some light on that?
>
> Also I was wondering about how pages move back and forth between
> cleancache and pagecache.  Are they copied or moved back and forth?  In
> other words, is the data placed in cleancache only once the local pagecache
> discards it, or as soon as it is read from disk, and when it is later
> requested, is it copied back from cleancache to the local pagecache, thus
> resulting in the data being duplicated in memory, once for the guest, and
> once in tmem?
>
>
>
> ______________________________**_________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
>


-- 
Jinho Hwang
PhD Student
Department of Computer Science
The George Washington University
Washington, DC 20052
hwang.jinho@gmail.com (email)
276.336.0971 (Cell)
202.994.4875 (fax)
070.8285.6546 (myLg070)


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

> From: Phillip Susi [mailto:psusi@ubuntu.com]
> Sent: Monday, April 16, 2012 12:41 PM
> To: Dan Magenheimer
> Cc: eva; xen-users@lists.xen.org
> Subject: Re: [Xen-users] Transcendent Memory ("tmem") -capable
kernel now publicly released
> 
> On 4/16/2012 12:40 PM, Dan Magenheimer wrote:
> > Hi Psusi (Phillip?) - Just saw your tmem post... sorry I
don''t keep
> > up with xen-users.   If you are looking at implementing Xen tmem
> > support on/for a future Ubuntu release, let me know if I can help and
> > see:
> > http://oss.oracle.com/git/?p=linux-2.6-unbreakable.git;a=summary
> 
> The last time I played with it I think I managed to get it working after
> adding the "tmem" command line arguments to xen and the guest
kernel.
> What I have not been able to do is figure out how to monitor its
> usage/effectiveness.  The output of xm tmem was completely
> indecipherable.  Could you shed some light on that?
Hi Phillip --

That weird ASCII text is intended to allow the hypervisor to
communicate with userland in a fully-backwards-and-forwards-compatible
way.  It must be parsed by a userland program, i.e.:

# xm tmem-list --long --all | /usr/sbin/xen-tmem-list-parse

I usually surround the above with ''watch -d "<above
commands>"''
in a big window to watch things change.  Dunno if ubuntu
has the watch command though.

(If the parsing program isn''t at that location, it may not be
built or maybe is installed elsewhere on a Debian-ish system.
See the source for it in xen source in xen/tools/misc.)

The parsed output has a huge amount of detail so is still mostly
undecipherable to mortals.  The key data can also be watched in
xentop (aka "xm top")... press "t" for tmem data (which will
show up only if any of the key values are non-zero).  Also
in xentop, you can watch selfballooning working.  (Note
"xm list" has had a bug forever so doesn''t show "current
memory"
just the memory the domain was launched with.)
 
> Also I was wondering about how pages move back and forth between
> cleancache and pagecache.  Are they copied or moved back and forth?  In
> other words, is the data placed in cleancache only once the local
> pagecache discards it, or as soon as it is read from disk, and when it
> is later requested, is it copied back from cleancache to the local
> pagecache, thus resulting in the data being duplicated in memory, once
> for the guest, and once in tmem?
For ephemeral pools (i.e. via cleancache), "gets" are destructive,
so the page of data is moved (removed from tmem) on a successful get.
So no duplication.  For persistent pools (i.e. via frontswap),
"gets" are non-destructive, so the page of data is copied from tmem
on a successful get, which mimics a swap device, thus the name
"persistent"; a "flush" is required to destroy persistent
pages of data.

See http://oss.oracle.com/projects/tmem/dist/documentation/api/tmemspec-v001.pdf
(which is old, but still 99% correct) and you may also be interested
in this: http://lwn.net/Articles/454795/ )

Dan

P.S. Although Xen tmem will work without frontswap, I don''t recommend
it
(esp selfballooning) without frontswap.  Selfballooning can be aggressive
and may sometimes cause swapping, which is absorbed by frontswap
instead of going to a swap disk.  That''s why the Oracle UEK2 kernel
includes frontswap even though it isn''t upstream yet, see
http://lwn.net/Articles/465317/ (we''ll be trying upstream again soon).

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/16/2012 04:11 PM, Dan Magenheimer wrote:
> That weird ASCII text is intended to allow the hypervisor to
> communicate with userland in a fully-backwards-and-forwards-compatible
> way.  It must be parsed by a userland program, i.e.:
> 
> # xm tmem-list --long --all | /usr/sbin/xen-tmem-list-parse
> 
> I usually surround the above with ''watch -d "<above
commands>"''
> in a big window to watch things change.  Dunno if ubuntu
> has the watch command though.
> 
> (If the parsing program isn''t at that location, it may not be
> built or maybe is installed elsewhere on a Debian-ish system.
> See the source for it in xen source in xen/tools/misc.)
I''m curious as to why the parser is a separate utility instead of just
being the normal output of the management tool, with an optional
--machine-parsable switch to fall back to the current behavior.
> The parsed output has a huge amount of detail so is still mostly
> undecipherable to mortals.  The key data can also be watched in
> xentop (aka "xm top")... press "t" for tmem data (which
will
> show up only if any of the key values are non-zero).  Also
> in xentop, you can watch selfballooning working.  (Note
> "xm list" has had a bug forever so doesn''t show
"current memory"
> just the memory the domain was launched with.)
Now THAT sounds like what I have been looking for.  I''ll have to check
it out.
> For ephemeral pools (i.e. via cleancache), "gets" are
destructive,
> so the page of data is moved (removed from tmem) on a successful get.
> So no duplication.  For persistent pools (i.e. via frontswap),
> "gets" are non-destructive, so the page of data is copied from
tmem
> on a successful get, which mimics a swap device, thus the name
> "persistent"; a "flush" is required to destroy
persistent pages of data.
Hrm... I thought one of the advantages of tmem was that when multiple domains
happen to be caching the same data, the page only needs to be kept in ram once. 
If get moves the page out of tmem and into the domain, that would seem to
preclude that.  Also I understand that tmem can compress the data, in which case
it doesn''t seem possible to do a move as the data must first be
decompressed.
> P.S. Although Xen tmem will work without frontswap, I don''t
recommend it
> (esp selfballooning) without frontswap.  Selfballooning can be aggressive
> and may sometimes cause swapping, which is absorbed by frontswap
> instead of going to a swap disk.  That''s why the Oracle UEK2
kernel
> includes frontswap even though it isn''t upstream yet, see
> http://lwn.net/Articles/465317/ (we''ll be trying upstream again
soon).
I''ll have to try and find the patch series somewhere and merge it into
my local Ubuntu git tree.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPjNu5AAoJEJrBOlT6nu75pZsH/1XSdxQyLS0b7xhZIi4VkQRB
36dKAgYaxQsyF+Y3FAJQjWP9Z4XS4YyycbnCtP02ZwJOwmAsSuBA6XHrdnLxMOKQ
tk5VoAomSWmnD+A3wAIat/IyvNioGrh1PfltmcAk/iifGCWAquHZHuKpd0IVMllG
cDjhRKJgZ5M24w1BRp276BW+UX1mj+InQJOLI7QQwv6eLvrbaVIHlpmeG0/E8BlY
5y1NXZAHKfKaqY3X/3Iq+gyM6Baqa5herGiAPNfo4KlQMtOj3t5Mdf0+++sXIDC/
J7cH6Ux/GFE0QImQ19MZ+O++8gAQYgL+VOLccmWmYhvtHPQ7agMyrzgLFHU4xw0=wrIj
-----END PGP SIGNATURE-----

> From: Phillip Susi [mailto:psusi@ubuntu.com]
> Subject: Re: [Xen-users] Transcendent Memory ("tmem") -capable
kernel now publicly released
Hi Phillip!  Thanks for the great questions!
> On 04/16/2012 04:11 PM, Dan Magenheimer wrote:
> > That weird ASCII text is intended to allow the hypervisor to
> > communicate with userland in a fully-backwards-and-forwards-compatible
> > way.  It must be parsed by a userland program, i.e.:
> >
> > # xm tmem-list --long --all | /usr/sbin/xen-tmem-list-parse
> >
> > I usually surround the above with ''watch -d "<above
commands>"''
> > in a big window to watch things change.  Dunno if ubuntu
> > has the watch command though.
> >
> > (If the parsing program isn''t at that location, it may not be
> > built or maybe is installed elsewhere on a Debian-ish system.
> > See the source for it in xen source in xen/tools/misc.)
> 
> I''m curious as to why the parser is a separate utility instead of
just being the normal output of the
> management tool, with an optional --machine-parsable switch to fall back to
the current behavior.
First, as you''ve seen, most of the tmem-list info is really more like
debugfs stuff from the kernel.  Very valuable to developers and
maybe interesting to performance analysts or students studying 
tmem but not useful to normal users.  The info was intended
to grow (and/or shrink) as more (less) data was determined
to be useful to its intended audience.

Second, tmem predates "xl" and all the great tools-layering work
done by Ian and Ian and Stefano and others.  Maybe some of the
key data could easily be added to and reported via xl nowadays
but, given the intended audience, I''m not sure it''s worth
reinventing
the wheel... though patches welcome!
> > The parsed output has a huge amount of detail so is still mostly
> > undecipherable to mortals.  The key data can also be watched in
> > xentop (aka "xm top")... press "t" for tmem data
(which will
> > show up only if any of the key values are non-zero).  Also
> > in xentop, you can watch selfballooning working.  (Note
> > "xm list" has had a bug forever so doesn''t show
"current memory"
> > just the memory the domain was launched with.)
> 
> Now THAT sounds like what I have been looking for.  I''ll have to
check it out.
Great! ;-)
> > For ephemeral pools (i.e. via cleancache), "gets" are
destructive,
> > so the page of data is moved (removed from tmem) on a successful get.
> > So no duplication.  For persistent pools (i.e. via frontswap),
> > "gets" are non-destructive, so the page of data is copied
from tmem
> > on a successful get, which mimics a swap device, thus the name
> > "persistent"; a "flush" is required to destroy
persistent pages of data.
> 
> Hrm... I thought one of the advantages of tmem was that when multiple
domains happen to be caching the
> same data, the page only needs to be kept in ram once.  If get moves the
page out of tmem and into the
> domain, that would seem to preclude that.  Also I understand that tmem can
compress the data, in which
> case it doesn''t seem possible to do a move as the data must first
be decompressed.
I suspect we''re quibbling about the precise meaning of "copy"
and "move".
The guest only hands a pageframe.  The hypervisor is responsible
for the data movement and/or removing the hypervisor''s copy of
the data.  There''s no fancy virtual-address-translation tricks.

By default, tmem only "shares" data across domains if all of these
domains are mounting the same clustered filesystem which is only
implemented today for ocfs2.  With tmem_dedup (as an additional Xen
boot option), pages "put" into tmem are deduplicated so there is
only one copy kept, reference counted, for identical pages
whether from the same domain or different domains.  With
tmem_compress as an additional Xen boot option, the hypervisor
de/compresses the data as part of the get/put operations.
And, yes, tmem_dedup and tmem_compress can be used together
to minimize total memory use.

Tmem_compress and tmem_dedup cost a lot more cycles but are usually
well worth it.  They aren''t turned on by default only because
they''ve
gotten much less testing.
> > P.S. Although Xen tmem will work without frontswap, I don''t
recommend it
> > (esp selfballooning) without frontswap.  Selfballooning can be
aggressive
> > and may sometimes cause swapping, which is absorbed by frontswap
> > instead of going to a swap disk.  That''s why the Oracle UEK2
kernel
> > includes frontswap even though it isn''t upstream yet, see
> > http://lwn.net/Articles/465317/ (we''ll be trying upstream
again soon).
> 
> I''ll have to try and find the patch series somewhere and merge it
into my local Ubuntu git tree.
Try http://oss.oracle.com/git/djm/tmem.git/?p=djm/tmem.git;a=summary 
with the frontswap-v14 or frontswap-v11 branch, depending on
your kernel version.  All the differences between v11 and v14
are either doc-only or irrelevant to Xen.  (v13 is also in
linux-next)

Dan

Hi  everyone
This feature is very cool.When I execute "make linux kernel" on my
VM(vcpus=4,memory=256M to simulate memory pressure),it take almost 30 hours,
and tmem reduce this time to 2 hours~~~

I am  go on testing it~~~



Gavin



--
View this message in context:
http://xen.1045712.n5.nabble.com/Transcendent-Memory-tmem-capable-kernel-now-publicly-released-tp5587302p5710502.html
Sent from the Xen - User mailing list archive at Nabble.com.

Hi,

how are you implementing it?

I''m currently using it on test machines with 
    xen bootargs tmem tmem_dedup tmem_compress
and
    dom0 kernel bootargs tmem

the domUs also have recent kernels with tmem bootargs as well as the
zcache module loaded.

I noticed very high memory latency when overcommiting memory.

cheers,

Stephan




_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

hi  Dan.
        I  found a question that I can‘t understand when I test multi
tmem-aware VMs.
        I  run two VMs and use option "tmem_dedup" in dom0 to test
dedup
between 2 clients.All the 2 VMs (vcpu=4,memory=256M) run "make Linux
kernel"
operation. VM1 run first and after 3 minutes VM2 run. And VM1 end
workload(make Linux kernel)is 10 minutes earlier than VM2. The whole
workload last almost 3 hours . 

I found Tc(total_cycles) VM2 is almost double to VM1.
VM1's Tc(pre=468057722,post=769901743325),  so total_cycles is
769,433,685,603.
VM2's Tc(pre=551611474,post=1428367229662),so total_cycles is
1,427,815,618,188.
so VM2's all tmem-ops is double to VM1，include "put" and
"get".
Are these statistics all right ??


Thanks a lot


Gavin



--
View this message in context:
http://xen.1045712.n5.nabble.com/Transcendent-Memory-tmem-capable-kernel-now-publicly-released-tp5587302p5710585.html
Sent from the Xen - User mailing list archive at Nabble.com.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users