tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
2006-Nov-02 01:56 UTC
[ANN] memcache-client extensions plugin
From: http://blog.zvents.com/2006/11/1/rails-plugin-memcacheclient-extensions =MemCacheClient Extensions == About The memcache-client_extensions plugins adds three new commands to the memcache client API: 1. get_multi : retrieve more than 1 key in parallel 2. stats : retrieve server performance and utilization statistics 3. flush_all : empty all information stored in memcached == Installation 1. This plugin requires that the memcache-client gem is installed. # gem install memcache-client 2. Install the plugin or the gem $ script/plugin install svn://rubyforge.org/var/svn/zventstools/projects/memcache-client_extensions - OR - # gem install memcache-client_extensions == get_multi Retrieve multiple values from memcached in parallel, if possible. The memcached protocol supports the ability to retrieve multiple keys in a single request. Pass in an array of keys to this method and it will: a. map the key to the appropriate memcached server b. send a single request to each server that has one or more key values Returns a hash of values.>> CACHE["a"] = 1=> 1>> CACHE["b"] = 2=> 2>> CACHE.get_multi(["a","b"])=> {"a"=>1, "b"=>2} Here''s a benchmark showing the speedup: CACHE["a"] = 1 CACHE["b"] = 2 CACHE["c"] = 3 CACHE["d"] = 4 keys = ["a","b","c","d","e"] Benchmark.bm do |x| x.report { for i in 1..1000; keys.each{|k| CACHE.get(k);} end } x.report { for i in 1..1000; CACHE.get_multi(keys); end } end returns: user system total real 0.180000 0.130000 0.310000 ( 0.459418) 0.200000 0.030000 0.230000 ( 0.269632) There''s a fair amount of non-DRY between get_multi and get (and threadsafe_cache_get/multi_threadsafe_cache_get and cache_get/multi_cache_get for that matter) but I think it''s worth it since the extra overhead to handle multiple return values is unneeded for a single-key get (which is by far the most common case). == stats The stats method returns statistics for each memcached server. An explanation of the statistics can be found in the memcached docs: http://cvs.danga.com/browse.cgi/wcmtools/memcached/doc/protocol.txt?rev=HEAD&content-type=text/plain Example:>> CACHE.stats=> {"localhost:11211"=>{"pid"=>"20188", "bytes"=>"4718", "connection_structures"=>"4", "time"=>"1162278121", "pointer_size"=>"32", "limit_maxbytes"=>"67108864", "version"=>"1.2.0", "cmd_get"=>"14532", "cmd_set"=>"32", "bytes_written"=>"432583", "uptime"=>"1557", "curr_items"=>"4", "curr_connections"=>"3", "total_connections"=>"19", "get_misses"=>"0", "rusage_user"=>"0.119981", "rusage_system"=>"0.313952", "total_items"=>"32", "get_hits"=>"14532", "bytes_read"=>"190619"}} == flush_all The flush_all method empties all cache namespaces on all memcached servers. This method is very useful for testing your code with memcached since you normally want to reset the cache to a known (empty) state at the beginning of each test. == Bugs, Code and Contributing There''s a RubyForge project set up at: http://rubyforge.org/projects/zventstools/ Anonymous SVN access: $ svn checkout svn://rubyforge.org/var/svn/zventstools Author: Tyler Kovacs (tyler dot kovacs at gmail dot com) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~---
Cool :-) Slightly OT here, but is there a way to access all values from a namespace in memcached? Like "get all my books" or something. Seems like a good extension to your get_multi function. But perhaps the memcached protocol doesn''t allow for this. That was my finding the last time I read through it. Somebody should fix this though: clearing/adding-to/retrieving a namespace is a real good feature to add. Vish On 11/2/06, tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > > From: > http://blog.zvents.com/2006/11/1/rails-plugin-memcacheclient-extensions > > =MemCacheClient Extensions > > == About > > The memcache-client_extensions plugins adds three new commands to the > memcache client API: > > 1. get_multi : retrieve more than 1 key in parallel > 2. stats : retrieve server performance and utilization statistics > 3. flush_all : empty all information stored in memcached > > == Installation > > 1. This plugin requires that the memcache-client gem is installed. > # gem install memcache-client > > 2. Install the plugin or the gem > $ script/plugin install > > svn://rubyforge.org/var/svn/zventstools/projects/memcache-client_extensions > - OR - > # gem install memcache-client_extensions > > == get_multi > > Retrieve multiple values from memcached in parallel, if possible. The > memcached protocol supports the ability to retrieve multiple keys in a > single request. Pass in an array of keys to this method and it will: > a. map the key to the appropriate memcached server > b. send a single request to each server that has one or more key > values > > Returns a hash of values. > > >> CACHE["a"] = 1 > => 1 > >> CACHE["b"] = 2 > => 2 > >> CACHE.get_multi(["a","b"]) > => {"a"=>1, "b"=>2} > > Here''s a benchmark showing the speedup: > > CACHE["a"] = 1 > CACHE["b"] = 2 > CACHE["c"] = 3 > CACHE["d"] = 4 > keys = ["a","b","c","d","e"] > Benchmark.bm do |x| > x.report { for i in 1..1000; keys.each{|k| CACHE.get(k);} end } > x.report { for i in 1..1000; CACHE.get_multi(keys); end } > end > > returns: > user system total real > 0.180000 0.130000 0.310000 ( 0.459418) > 0.200000 0.030000 0.230000 ( 0.269632) > > There''s a fair amount of non-DRY between get_multi and get (and > threadsafe_cache_get/multi_threadsafe_cache_get and > cache_get/multi_cache_get for that matter) but I think it''s worth it > since the extra overhead to handle multiple return values is unneeded > for a single-key get (which is by far the most common case). > > == stats > > The stats method returns statistics for each memcached server. An > explanation of the statistics can be found in the memcached docs: > > http://cvs.danga.com/browse.cgi/wcmtools/memcached/doc/protocol.txt?rev=HEAD&content-type=text/plain > > Example: > >> CACHE.stats > => {"localhost:11211"=>{"pid"=>"20188", "bytes"=>"4718", > "connection_structures"=>"4", "time"=>"1162278121", > "pointer_size"=>"32", "limit_maxbytes"=>"67108864", "version"=>"1.2.0", > "cmd_get"=>"14532", "cmd_set"=>"32", "bytes_written"=>"432583", > "uptime"=>"1557", "curr_items"=>"4", "curr_connections"=>"3", > "total_connections"=>"19", "get_misses"=>"0", > "rusage_user"=>"0.119981", "rusage_system"=>"0.313952", > "total_items"=>"32", "get_hits"=>"14532", "bytes_read"=>"190619"}} > > == flush_all > > The flush_all method empties all cache namespaces on all memcached > servers. This method is very useful for testing your code with > memcached since you normally want to reset the cache to a known > (empty) state at the beginning of each test. > > == Bugs, Code and Contributing > > There''s a RubyForge project set up at: > > http://rubyforge.org/projects/zventstools/ > > Anonymous SVN access: > > $ svn checkout svn://rubyforge.org/var/svn/zventstools > > Author: Tyler Kovacs (tyler dot kovacs at gmail dot com) > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
2006-Nov-05 23:21 UTC
Re: [ANN] memcache-client extensions plugin
No, the memcached protocol doesn''t support a "get all" method. Here''s a thread from the memcached mailing that covers the issue: http://lists.danga.com/pipermail/memcached/2006-April/002112.html You could memcache your dataset in an array (in addition to memcaching each individual item for single-item retrieval), but that probably wouldn''t be a good idea for anything other than very small datasets. Vishnu Gopal wrote:> Cool :-) > > Slightly OT here, but is there a way to access all values from a namespace > in memcached? Like "get all my books" or something. Seems like a good > extension to your get_multi function. > > But perhaps the memcached protocol doesn''t allow for this. That was my > finding the last time I read through it. Somebody should fix this though: > clearing/adding-to/retrieving a namespace is a real good feature to add. > > Vish > > On 11/2/06, tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > > > From: > > http://blog.zvents.com/2006/11/1/rails-plugin-memcacheclient-extensions > > > > =MemCacheClient Extensions > > > > == About > > > > The memcache-client_extensions plugins adds three new commands to the > > memcache client API: > > > > 1. get_multi : retrieve more than 1 key in parallel > > 2. stats : retrieve server performance and utilization statistics > > 3. flush_all : empty all information stored in memcached > > > > == Installation > > > > 1. This plugin requires that the memcache-client gem is installed. > > # gem install memcache-client > > > > 2. Install the plugin or the gem > > $ script/plugin install > > > > svn://rubyforge.org/var/svn/zventstools/projects/memcache-client_extensions > > - OR - > > # gem install memcache-client_extensions > > > > == get_multi > > > > Retrieve multiple values from memcached in parallel, if possible. The > > memcached protocol supports the ability to retrieve multiple keys in a > > single request. Pass in an array of keys to this method and it will: > > a. map the key to the appropriate memcached server > > b. send a single request to each server that has one or more key > > values > > > > Returns a hash of values. > > > > >> CACHE["a"] = 1 > > => 1 > > >> CACHE["b"] = 2 > > => 2 > > >> CACHE.get_multi(["a","b"]) > > => {"a"=>1, "b"=>2} > > > > Here''s a benchmark showing the speedup: > > > > CACHE["a"] = 1 > > CACHE["b"] = 2 > > CACHE["c"] = 3 > > CACHE["d"] = 4 > > keys = ["a","b","c","d","e"] > > Benchmark.bm do |x| > > x.report { for i in 1..1000; keys.each{|k| CACHE.get(k);} end } > > x.report { for i in 1..1000; CACHE.get_multi(keys); end } > > end > > > > returns: > > user system total real > > 0.180000 0.130000 0.310000 ( 0.459418) > > 0.200000 0.030000 0.230000 ( 0.269632) > > > > There''s a fair amount of non-DRY between get_multi and get (and > > threadsafe_cache_get/multi_threadsafe_cache_get and > > cache_get/multi_cache_get for that matter) but I think it''s worth it > > since the extra overhead to handle multiple return values is unneeded > > for a single-key get (which is by far the most common case). > > > > == stats > > > > The stats method returns statistics for each memcached server. An > > explanation of the statistics can be found in the memcached docs: > > > > http://cvs.danga.com/browse.cgi/wcmtools/memcached/doc/protocol.txt?rev=HEAD&content-type=text/plain > > > > Example: > > >> CACHE.stats > > => {"localhost:11211"=>{"pid"=>"20188", "bytes"=>"4718", > > "connection_structures"=>"4", "time"=>"1162278121", > > "pointer_size"=>"32", "limit_maxbytes"=>"67108864", "version"=>"1.2.0", > > "cmd_get"=>"14532", "cmd_set"=>"32", "bytes_written"=>"432583", > > "uptime"=>"1557", "curr_items"=>"4", "curr_connections"=>"3", > > "total_connections"=>"19", "get_misses"=>"0", > > "rusage_user"=>"0.119981", "rusage_system"=>"0.313952", > > "total_items"=>"32", "get_hits"=>"14532", "bytes_read"=>"190619"}} > > > > == flush_all > > > > The flush_all method empties all cache namespaces on all memcached > > servers. This method is very useful for testing your code with > > memcached since you normally want to reset the cache to a known > > (empty) state at the beginning of each test. > > > > == Bugs, Code and Contributing > > > > There''s a RubyForge project set up at: > > > > http://rubyforge.org/projects/zventstools/ > > > > Anonymous SVN access: > > > > $ svn checkout svn://rubyforge.org/var/svn/zventstools > > > > Author: Tyler Kovacs (tyler dot kovacs at gmail dot com) > > > > > > > > > > > ------=_Part_23905_27212983.1162620794559 > Content-Type: text/html; charset=ISO-8859-1 > X-Google-AttachSize: 5571 > > Cool :-)<br><br>Slightly OT here, but is there a way to access all values from a namespace in memcached? Like "get all my books" or something. Seems like a good extension to your get_multi function. <br><br>But perhaps the memcached protocol doesn''t allow for this. That was my finding the last time I read through it. Somebody should fix this though: clearing/adding-to/retrieving a namespace is a real good feature to add. > <br><br>Vish<br><br><div><span class="gmail_quote">On 11/2/06, <b class="gmail_sendername"><a href="mailto:tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org">tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org</a></b> <<a href="mailto:tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org">tyler.kovacs-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org > </a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>From:<br><a href="http://blog.zvents.com/2006/11/1/rails-plugin-memcacheclient-extensions"> > http://blog.zvents.com/2006/11/1/rails-plugin-memcacheclient-extensions</a><br><br>=MemCacheClient Extensions<br><br>== About<br><br>The memcache-client_extensions plugins adds three new commands to the<br>memcache client API: > <br><br>1. get_multi : retrieve more than 1 key in parallel<br>2. stats : retrieve server performance and utilization statistics<br>3. flush_all : empty all information stored in memcached<br><br>== Installation<br><br> > 1. This plugin requires that the memcache-client gem is installed.<br> # gem install memcache-client<br><br>2. Install the plugin or the gem<br> $ script/plugin install<br>svn://rubyforge.org/var/svn/zventstools/projects/memcache-client_extensions > <br>- OR -<br> # gem install memcache-client_extensions<br><br>== get_multi<br><br>Retrieve multiple values from memcached in parallel, if possible. The<br>memcached protocol supports the ability to retrieve multiple keys in a > <br>single request. Pass in an array of keys to this method and it will:<br> a. map the key to the appropriate memcached server<br> b. send a single request to each server that has one or more key<br>values<br><br>Returns a hash of values. > <br><br>>> CACHE["a"] = 1<br>=> 1<br>>> CACHE["b"] = 2<br>=> 2<br>>> CACHE.get_multi(["a","b"])<br>=> {"a"=>1, "b"=>2}<br><br>Here''s a benchmark showing the speedup: > <br><br>CACHE["a"] = 1<br>CACHE["b"] = 2<br>CACHE["c"] = 3<br>CACHE["d"] = 4<br>keys = ["a","b","c","d","e"]<br><a href="http://Benchmark.bm"> > Benchmark.bm</a> do |x|<br> x.report { for i in 1..1000; keys.each{|k| CACHE.get(k);} end }<br> x.report { for i in 1..1000; CACHE.get_multi(keys); end }<br>end<br><br>returns:<br> user system total real > <br> 0.180000 0.130000 0.310000 ( 0.459418)<br> 0.200000 0.030000 0.230000 ( 0.269632)<br><br>There''s a fair amount of non-DRY between get_multi and get (and<br>threadsafe_cache_get/multi_threadsafe_cache_get and > <br>cache_get/multi_cache_get for that matter) but I think it''s worth it<br>since the extra overhead to handle multiple return values is unneeded<br>for a single-key get (which is by far the most common case).<br><br>== stats > <br><br>The stats method returns statistics for each memcached server. An<br>explanation of the statistics can be found in the memcached docs:<br><a href="http://cvs.danga.com/browse.cgi/wcmtools/memcached/doc/protocol.txt?rev=HEAD&content-type=text/plain"> > http://cvs.danga.com/browse.cgi/wcmtools/memcached/doc/protocol.txt?rev=HEAD&content-type=text/plain</a><br><br>Example:<br>>> CACHE.stats<br>=> {"localhost:11211"=>{"pid"=>"20188", "bytes"=>"4718", > <br>"connection_structures"=>"4", "time"=>"1162278121",<br>"pointer_size"=>"32", "limit_maxbytes"=>"67108864", "version"=>" > 1.2.0",<br>"cmd_get"=>"14532", "cmd_set"=>"32", "bytes_written"=>"432583",<br>"uptime"=>"1557", "curr_items"=>"4", "curr_connections"=>"3", > <br>"total_connections"=>"19", "get_misses"=>"0",<br>"rusage_user"=>"0.119981", "rusage_system"=>"0.313952",<br>"total_items"=>"32", "get_hits"=>"14532", "bytes_read"=>"190619"}} > <br><br>== flush_all<br><br>The flush_all method empties all cache namespaces on all memcached<br>servers. This method is very useful for testing your code with<br>memcached since you normally want to reset the cache to a known > <br>(empty) state at the beginning of each test.<br><br>== Bugs, Code and Contributing<br><br>There''s a RubyForge project set up at:<br><br><a href="http://rubyforge.org/projects/zventstools/">http://rubyforge.org/projects/zventstools/ > </a><br><br>Anonymous SVN access:<br><br>$ svn checkout svn://rubyforge.org/var/svn/zventstools<br><br>Author: Tyler Kovacs (tyler dot kovacs at gmail dot com)<br><br><br> > ------=_Part_23905_27212983.1162620794559----~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---