All, Platform: Mandriva Linux release 2006.0 (Official) for i586 Kernel 2.6.12-27mdksmp on a 4-processor i686 / \l Background: I''ve got an install of memcached that I''m attempting to use to store data across requests. My application allows users to upload spreadsheets of any size (up to 65K rows given Excel limitations), and then I parse these spreadsheets and store their data in the database. I''m attempting to use memcached to hold an intermediate object that I use to display the data from the spreadsheet. I can successfully store an object built from spreadsheet data of 1000 rows. Problem: I cannot successfully store an object built from 10000 rows or greater in memcached. I have definitely narrowed it down to the cache put action in the memcache client code. When I print out the size of the marshaled object (from Marshal.dump) that is being put into the memcache, I get the following: 1000 rows: 33935 (successfully stored) 10000 rows: 1559254 (not stored) 20000 rows: 3098224 (not stored) Question: Granted, these are big numbers, but I would expect that memcache could take objects of this size. I''m not sure here if it''s memcache that is choking or the Ruby Marshal library. Does anyone have any insight into this? Just as a fun aside, everything works fine on my Windows box :) (using the pre-compiled memcached code). I was hoping to avoid some crappy temporary table solution, but it looks like I will be forced into that. Thanks, Wes -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Some more info.:
I''ve verified that I can Marshal.dump and Marshal.load my giant objects
without incident.
I am led to assume that the failure occurs on the sock.write command
(see below snippet from the "put" method of the memcache.rb file in
the
memcache-client gem). I think that sock.write can''t handle a string of
that size.
Can anyone tell me if I''m on the right track? I''m going to
attempt to
see if I can''t do the write in reasonably sized blocks instead of doing
it all at once.
Thanks,
Wes
==================
BEGIN code snippet
sock = server.socket
raise MemCacheError, "No connection to server" if sock.nil?
marshaled_value = Marshal.dump value
command = "set #{cache_key} 0 #{expiry}
#{marshaled_value.size}\r\n#{marshaled_value}\r\n"
begin
sock.write command
sock.gets
rescue SystemCallError, IOError => err
server.close
raise MemCacheError, err.message
end
END code snippet
--
Posted via http://www.ruby-forum.com/.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
It turns out that the maximum allowed size of an object in memcache is 1 MB. I am looking into either how to increase that maximum or break up my object into pieces so that I can effectively store it. What''s a little annoying is that the memcache library suppresses the message that comes back from the socket, which is a pretty clear message: SERVER_ERROR object too large for cache Thanks, Wes -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
SOLUTION: I modified memcache-client to break up objects larger than 1048510 bytes (just under 1MB, you can''t successfully store more than this in the memcache) into pieces internally so that they can be stored. When a get is done, the pieces are reconstituted before the Marshal.load. The delete also handles cleanup. Also, in my OP, I said that everything worked fine on Windows, but that wasn''t true. Wes -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Wes Gamble wrote in post #216702:> SOLUTION: I modified memcache-client to break up objects larger than > 1048510 bytes (just under 1MB, you can''t successfully store more than > this in the memcache) into pieces internally so that they can be stored. > When a get is done, the pieces are reconstituted before the > Marshal.load. The delete also handles cleanup. > > Also, in my OP, I said that everything worked fine on Windows, but that > wasn''t true. > > WesSorry for bringing out this post from dead. However, i thought it would be helpful to others hence this. The maximum size of data which memcache can load depends on a macro POWER_BLOCK (in slabs.c). If we modify this macro to higher values, say 2 MB of more, memcache would be able to handle data larger than 1 MB but less than equals 2MB. A better approach would be to make it a parameter to memcache hence eliminating the need to compile every time you need to increase memory. -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
Ha ha - now it is an option! But only for the last two years so I
don''t
feel stupid for doing what I did.
From the memcached man page on OS X:
" -I <size>
Override the default size of each slab page. Default is
1mb. Default is 1m, minimum is 1k, max is 128m. Adjusting this value
changes the item size limit. Beware that this also increases the number
of slabs (use -v to view), and the overal memory usage of
memcached."
Wes
--
Posted via http://www.ruby-forum.com/.
--
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.