Hi, I have been using Ferret on ruby-forum.com for a few days. While the searching works with reasonable performance, there are a lot of problems related to indexing. Sometimes the process hangs with 100% CPU usage, sometimes it leaves Lockfiles in the directory (causing other processes to fail while one ist still working), sometimes it even crashes with a segfault. I''m going to try to use only one indexing process to solve the locking problems. A process that looks periodically for posts that need to be indexed is probably the most robust solution. Andreas -- Posted via http://www.ruby-forum.com/.
Andreas S. wrote:>I''m going to try to use only one indexing process to solve the locking >problems. A process that looks periodically for posts that need to be >indexed is probably the most robust solution. > >You must''ve tried this already, just checking - did you set :auto_flush => true while creating your index? The documentation says it prevents locking when multiple processes are accessing the index. Vamsee.
Vamsee Kanakala wrote:> Andreas S. wrote: > >>I''m going to try to use only one indexing process to solve the locking >>problems. A process that looks periodically for posts that need to be >>indexed is probably the most robust solution. >> >> > You must''ve tried this already, just checking - did you set :auto_flush > => true while creating your index?Yes.> The documentation says it prevents > locking when multiple processes are accessing the index.Locking usually works correctly, but *sometimes* I get lockfiles that are never removed. What I don''t understand is why one of the proceses is still able to write to the index then, while the others aren''t. -- Posted via http://www.ruby-forum.com/.
Hi, Andreas, maybe a stupid guess: Maybe because this one process is the locking one? Regards Jan Andreas S. wrote:>Vamsee Kanakala wrote: > > >>Andreas S. wrote: >> >> >> >>>I''m going to try to use only one indexing process to solve the locking >>>problems. A process that looks periodically for posts that need to be >>>indexed is probably the most robust solution. >>> >>> >>> >>> >>You must''ve tried this already, just checking - did you set :auto_flush >>=> true while creating your index? >> >> > >Yes. > > > >>The documentation says it prevents >>locking when multiple processes are accessing the index. >> >> > >Locking usually works correctly, but *sometimes* I get lockfiles that >are never removed. What I don''t understand is why one of the proceses is >still able to write to the index then, while the others aren''t. > > >
Jan Prill wrote:> Hi, Andreas, > > maybe a stupid guess: Maybe because this one process is the locking one? > > Regards > JanI don''t know. I have two fastcgi processes, and it seems that they are actually both able to write to the index (at least I never got exceptions when I created a post in the forum), only the third process (mail receiver) isn''t. Andreas -- Posted via http://www.ruby-forum.com/.
Hi Andreas, I am using a modified version of your Ferret code in my application, which is still in development mode, so no actual user testing has been done. I also encountered the locking problem. What I did was remove the auto_flush => true and handle flushing myself. So in the create_doc method I placed: index << doc index.flush at the end and removed your last line ''doc''. Which also made the update method absolete, so I made update an alias of create_doc . I haven''t had any locks during my testing, but I haven''t done any real world testing yet. So I''m eager to know how you managed to fix it, once it is fixed. Kind regards, Nick Andreas S. wrote:> Hi, > > I have been using Ferret on ruby-forum.com for a few days. While the > searching works with reasonable performance, there are a lot of problems > related to indexing. Sometimes the process hangs with 100% CPU usage, > sometimes it leaves Lockfiles in the directory (causing other processes > to fail while one ist still working), sometimes it even crashes with a > segfault. > > I''m going to try to use only one indexing process to solve the locking > problems. A process that looks periodically for posts that need to be > indexed is probably the most robust solution. > > Andreas-- Posted via http://www.ruby-forum.com/.
Nick Snels wrote:> Hi Andreas, > > I am using a modified version of your Ferret code in my application, > which is still in development mode, so no actual user testing has been > done. I also encountered the locking problem. What I did was remove the > auto_flush => true and handle flushing myself.I tried that too, but had the same problem as before.> at the end and removed your last line ''doc''. Which also made the update > method absolete, so I made update an alias of create_doc . I haven''t had > any locks during my testing, but I haven''t done any real world testing > yet. So I''m eager to know how you managed to fix it, once it is fixed.I am now using a process that checks periodically for new/changed posts and updates the index. This seems to be the best solution. You can find the code in RForum SVN. Andreas -- Posted via http://www.ruby-forum.com/.