Could someone please elaborate on the technical differences and practical impact of whether choosing memory-based tables in MySQL or using memcached. I got this far on my own: It seems that MySQL uses the NDB engine for transaction-safe memory access in a cluster. the memory storage engine seems to be faster but not synchronizable by any means in a cluster. memcached seems to be ultimately fast but requires extra effort in your code. Who got experience on that? Best regards Peter
You can use pure ruby memory structure like hash or array and access them via drb server. it use lot off memory (but that cheap) and you have good performance, and pure ruby object, no sql needed (like memcache). If you have complex processing you can do that in the drb server unlike with memcached, u dont touch the database, and its quite fast for processing, but you depend on drb. if u need load balancing and fail over, you can write a pool to do that . Pete wrote:> Could someone please elaborate on the technical differences and practical > impact of whether choosing memory-based tables in MySQL or using > memcached. > > I got this far on my own: > It seems that MySQL uses the NDB engine for transaction-safe memory > access > in a cluster. the memory storage engine seems to be faster but not > synchronizable by any means in a cluster. memcached seems to be > ultimately fast > but requires extra effort in your code. > > Who got experience on that? > > Best regards > Peter > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails > >"Ce message et toutes les pièces jointes sont des informations strictement confidentielles et réservées au(x) destinataire(s). Ce courriel n''a pas de valeur contractuelle et son contenu ne constitue ni une acceptation, ni un engagement de la part de l''auteur et des sociétés du groupe Serveur et Artprice, sauf dans le cas où cela aurait été prévu avec le destinataire par un accord écrit. Le contenu de ce message et les pièces jointes ne peuvent constituer une preuve au sens de l''article 1316-1 du Code Civil. L.auteur et les sociétés du groupe Serveur et Artprice déclinent toute responsabilité au titre de ce courriel s''il a été altéré, déformé, falsifié ou indûment utilisé par des tiers ou encore s''il a causé tout dommage ou perte de toute nature. Si vous n''êtes pas le bon destinataire, merci de nous contacter et de ne pas le divulguer." "This message including any attachments are confidential and privileged material intended solely for the addressees. Its contents do not constitute a commitment by groupe Serveur sas and Artprice SA, except when provided for in a written agreement with the addressees. The contents of this message cannot constitute neither the proof nor the acceptance of any agreement as per article 1316-1 of the French civil code. Groupe Serveur sas and Artprice SA shall not be rendered liable in any manner whatsoever for the delay and/or loss in transit of this message, for corruption, alteration, falsification, misuse or fraudulent use (which may be made) of this message. If you receive this message in error, please delete it and immediately notify the sender. If the reader of this message is not the intended recipient, you are hereby notified that any unauthorized use, copying or dissemination is prohibited."
Pete wrote:> Could someone please elaborate on the technical differences and practical > impact of whether choosing memory-based tables in MySQL or using > memcached. > > I got this far on my own: > It seems that MySQL uses the NDB engine for transaction-safe memory > access > in a cluster. the memory storage engine seems to be faster but not > synchronizable by any means in a cluster. memcached seems to be > ultimately fast > but requires extra effort in your code. > > Who got experience on that?Hi, mysql memory tables aren''t likely to lose data. Memcache WILL lose data. It''s a cache, that''s what it does. memcache''s purpose is to store a copy of frequently used data for fast access. Don''t store anything that you don''t want to lose in memcache. Jason
mySQL memory tables hold the information until the connection is closed or the table is dropped. These are useful if you have a large process and want to use a stored procedure to move/update data to keep the network traffic low. memCache was created as a proxy to the database for performance reasons. Its to fill requests that would normally go to a database. So, if every request on your website requires a row from the ''members'' table it would be smart to cache that with memCache. Why not just use in memory tables and forget memCache altogether? Because when a site has so much traffic, like livejournal, and you only have 1 database server to perform the transactions you need to have a minimum number of requests to that server. Plus, memCache is optimized for exactly this purpose, there is probably less de/serializing work that has to occur than if you were to use a database. On 12/16/05, Jason Edgecombe <jedgecombe-iK5jlJkdgf9Jm/Hvfsr4+Q@public.gmane.org> wrote:> Pete wrote: > > > Could someone please elaborate on the technical differences and practical > > impact of whether choosing memory-based tables in MySQL or using > > memcached. > > > > I got this far on my own: > > It seems that MySQL uses the NDB engine for transaction-safe memory > > access > > in a cluster. the memory storage engine seems to be faster but not > > synchronizable by any means in a cluster. memcached seems to be > > ultimately fast > > but requires extra effort in your code. > > > > Who got experience on that? > > Hi, > > mysql memory tables aren''t likely to lose data. > > Memcache WILL lose data. It''s a cache, that''s what it does. memcache''s > purpose is to store a copy of frequently used data for fast access. > Don''t store anything that you don''t want to lose in memcache. > > Jason > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
On Dec 16, 2005, at 1:48 AM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote:> Could someone please elaborate on the technical differences and > practical > impact of whether choosing memory-based tables in MySQL or using > memcached. > > I got this far on my own: > It seems that MySQL uses the NDB engine for transaction-safe memory > access > in a cluster. the memory storage engine seems to be faster but not > synchronizable by any means in a cluster. memcached seems to be > ultimately > fast but requires extra effort in your code. > > Who got experience on that?I have no experience with MySQL memory tables, but. memcached doesn''t have query parsing overhead. memcached doesn''t have database adapter overhead. memcached can be run across multiple machines painlessly. memcached is easier to set up. memcached can be painlessly used with unstructured data. memcached becomes transparent with proper abstraction. -- Eric Hodel - drbrain-48TerJ1FxhPk1uMJSBkQmQ@public.gmane.org - http://segment7.net This implementation is HODEL-HASH-9600 compliant http://trackmap.robotcoop.com
On Dec 16, 2005, at 5:35 AM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote:> You can use pure ruby memory structure like hash or array and access > them via drb server. > it use lot off memory (but that cheap) and you have good performance, > and pure ruby object, no sql needed (like memcache).memcached does not use SQL, it is more like a distributed dictionary. Memcached can store pure ruby objects (via Marshal, just like DRb).> If you have complex processing you can do that in the drb server > unlike > with memcached, u dont touch the database, and its quitememcached is not an RDBMS, it is just a dictionary. It is faster than DRb and will (potentially) consume less memory because of the way it allocates space for cache items.> fast for processing, but you depend on drb. if u need load > balancing and > fail over, you can write a pool to do that .memcached does load balancing painlessly. -- Eric Hodel - drbrain-48TerJ1FxhPk1uMJSBkQmQ@public.gmane.org - http://segment7.net This implementation is HODEL-HASH-9600 compliant http://trackmap.robotcoop.com
cremes.devlist-ee4meeAH724@public.gmane.org
2005-Dec-17 03:12 UTC
Re: mysql mem-tables vs. memcached
On Dec 16, 2005, at 2:07 PM, Eric Hodel wrote:> On Dec 16, 2005, at 1:48 AM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > wrote: > >> Could someone please elaborate on the technical differences and >> practical >> impact of whether choosing memory-based tables in MySQL or using >> memcached. >> >> I got this far on my own: >> It seems that MySQL uses the NDB engine for transaction-safe >> memory access >> in a cluster. the memory storage engine seems to be faster but not >> synchronizable by any means in a cluster. memcached seems to be >> ultimately >> fast but requires extra effort in your code. >> >> Who got experience on that? > > I have no experience with MySQL memory tables, but. > > memcached doesn''t have query parsing overhead. > memcached doesn''t have database adapter overhead. > memcached can be run across multiple machines painlessly. > memcached is easier to set up. > memcached can be painlessly used with unstructured data. > memcached becomes transparent with proper abstraction.Can you point to any sample code where someone is using memcached with ruby? I took a look around all the wikis and came up dry. The best examples I could find were in the wikipedia entry for memcached; the examples were not in ruby but it was easy to parse through them. While lurking on this list I think I saw a note or two implying that Rails uses memcached internally (as a configurable option). If you can confirm or deny this then I''d bet that code would be a perfect example. Thanks for any help and pointers.
On Dec 16, 2005, at 8:43 PM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote:> Can you point to any sample code where someone is using memcached > with ruby?43 Things uses memcached extensively (but not as extensively as LiveJournal).> I took a look around all the wikis and came up dry. The > best examples I could find were in the wikipedia entry for memcached; > the examples were not in ruby but it was easy to parse through them. > > While lurking on this list I think I saw a note or two implying that > Rails uses memcached internally (as a configurable option). If you > can confirm or deny this then I''d bet that code would be a perfect > example.Here''s how to use memcache to store sessions, I put it in config/ environment.rb. You will need the Ruby-MemCache gem installed. memcache_options = { :c_threshold => 10_000, :compression => true, :debug => false, :namespace => ''your_namespace'', :readonly => false, :urlencode => false } CACHE = MemCache.new memcache_options CACHE.servers = ''localhost:11211'' session_options = { :database_manager => CGI::Session::MemCacheStore, :cache => CACHE, :session_domain => ''example.com'' } ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS.update session_options Note that there is a bug [1] but you won''t be able to tell you''re experiencing it [2]. Since you''ve got a CACHE global constant, you can use it to store things that take a long time to build: CACHE.set ''yourkey'', ''something that takes a long time to build'' then later: value = CACHE.get ''yourkey'' [1] http://dev.rubyonrails.org/ticket/3214 [2] http://dev.rubyonrails.org/ticket/3216 -- Eric Hodel - drbrain-48TerJ1FxhPk1uMJSBkQmQ@public.gmane.org - http://segment7.net This implementation is HODEL-HASH-9600 compliant http://trackmap.robotcoop.com
Eric Hodel wrote:> On Dec 16, 2005, at 8:43 PM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote: > >> Can you point to any sample code where someone is using memcached >> with ruby? > > > 43 Things uses memcached extensively (but not as extensively as > LiveJournal). > >> I took a look around all the wikis and came up dry. The >> best examples I could find were in the wikipedia entry for memcached; >> the examples were not in ruby but it was easy to parse through them. >> >> While lurking on this list I think I saw a note or two implying that >> Rails uses memcached internally (as a configurable option). If you >> can confirm or deny this then I''d bet that code would be a perfect >> example. > > > Here''s how to use memcache to store sessions, I put it in config/ > environment.rb. You will need the Ruby-MemCache gem installed. > > memcache_options = { > :c_threshold => 10_000, > :compression => true, > :debug => false, > :namespace => ''your_namespace'', > :readonly => false, > :urlencode => false > } > > CACHE = MemCache.new memcache_options > CACHE.servers = ''localhost:11211'' > > session_options = { > :database_manager => CGI::Session::MemCacheStore, > :cache => CACHE, > :session_domain => ''example.com'' > } > > ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS.update > session_options > > Note that there is a bug [1] but you won''t be able to tell you''re > experiencing it [2]. > > Since you''ve got a CACHE global constant, you can use it to store > things that take a long time to build: > > CACHE.set ''yourkey'', ''something that takes a long time to build'' > > then later: > > value = CACHE.get ''yourkey'' > > [1] http://dev.rubyonrails.org/ticket/3214 > [2] http://dev.rubyonrails.org/ticket/3216 >Has anyone subclasses ActiveRecord to provide transparent caching of records using memcache? How would someone go about doing this? Jason
On Dec 17, 2005, at 2:21 PM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote:> Eric Hodel wrote: > >> On Dec 16, 2005, at 8:43 PM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org >> wrote: >> >>> Can you point to any sample code where someone is using memcached >>> with ruby? >> >> >> 43 Things uses memcached extensively (but not as extensively as >> LiveJournal). > > Has anyone subclasses ActiveRecord to provide transparent caching of > records using memcache?Yes. 43 Things uses this.> How would someone go about doing this?I can probably package it up for release after the new year. It is not difficult to implement, we override #find and #find_by_sql and add some hacks to override the single table inheritance silliness. -- Eric Hodel - drbrain-48TerJ1FxhPk1uMJSBkQmQ@public.gmane.org - http://segment7.net This implementation is HODEL-HASH-9600 compliant http://trackmap.robotcoop.com
Eric, How do you handle cache expiry? On 12/17/05, Eric Hodel <drbrain-48TerJ1FxhPk1uMJSBkQmQ@public.gmane.org> wrote:> On Dec 17, 2005, at 2:21 PM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote: > > > Eric Hodel wrote: > > > >> On Dec 16, 2005, at 8:43 PM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > >> wrote: > >> > >>> Can you point to any sample code where someone is using memcached > >>> with ruby? > >> > >> > >> 43 Things uses memcached extensively (but not as extensively as > >> LiveJournal). > > > > Has anyone subclasses ActiveRecord to provide transparent caching of > > records using memcache? > > Yes. 43 Things uses this. > > > How would someone go about doing this? > > I can probably package it up for release after the new year. It is > not difficult to implement, we override #find and #find_by_sql and > add some hacks to override the single table inheritance silliness. > > -- > Eric Hodel - drbrain-48TerJ1FxhPk1uMJSBkQmQ@public.gmane.org - http://segment7.net > This implementation is HODEL-HASH-9600 compliant > > http://trackmap.robotcoop.com > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
Eric Hodel wrote:> On Dec 16, 2005, at 5:35 AM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote: > >> You can use pure ruby memory structure like hash or array and access >> them via drb server. >> it use lot off memory (but that cheap) and you have good performance, >> and pure ruby object, no sql needed (like memcache). > > > memcached does not use SQL, it is more like a distributed > dictionary. Memcached can store pure ruby objects (via Marshal, just > like DRb). > >> If you have complex processing you can do that in the drb server unlike >> with memcached, u dont touch the database, and its quite > > > memcached is not an RDBMS, it is just a dictionary. It is faster > than DRb and will (potentially) consume less memory because of the > way it allocates space for cache items.yes but the chache cant handle any programming logic, if u have to sort for exemple u must fetch the object, (this could be long for thousand of them), and sort localy. uless a drb server can do the trick remotely ! it depends you needs .> >> fast for processing, but you depend on drb. if u need load balancing >> and >> fail over, you can write a pool to do that . > > > memcached does load balancing painlessly. >yes and you dont even had to write code for that ! memchache is great for proxy fast storing ! "Ce message et toutes les pièces jointes sont des informations strictement confidentielles et réservées au(x) destinataire(s). Ce courriel n''a pas de valeur contractuelle et son contenu ne constitue ni une acceptation, ni un engagement de la part de l''auteur et des sociétés du groupe Serveur et Artprice, sauf dans le cas où cela aurait été prévu avec le destinataire par un accord écrit. Le contenu de ce message et les pièces jointes ne peuvent constituer une preuve au sens de l''article 1316-1 du Code Civil. L.auteur et les sociétés du groupe Serveur et Artprice déclinent toute responsabilité au titre de ce courriel s''il a été altéré, déformé, falsifié ou indûment utilisé par des tiers ou encore s''il a causé tout dommage ou perte de toute nature. Si vous n''êtes pas le bon destinataire, merci de nous contacter et de ne pas le divulguer." "This message including any attachments are confidential and privileged material intended solely for the addressees. Its contents do not constitute a commitment by groupe Serveur sas and Artprice SA, except when provided for in a written agreement with the addressees. The contents of this message cannot constitute neither the proof nor the acceptance of any agreement as per article 1316-1 of the French civil code. Groupe Serveur sas and Artprice SA shall not be rendered liable in any manner whatsoever for the delay and/or loss in transit of this message, for corruption, alteration, falsification, misuse or fraudulent use (which may be made) of this message. If you receive this message in error, please delete it and immediately notify the sender. If the reader of this message is not the intended recipient, you are hereby notified that any unauthorized use, copying or dissemination is prohibited."