Julik
2006-Jun-30 02:59 UTC
[Ferret-talk] Substantial problems with write locking (and other flux)
I am having some great trouble keeping my Ferret indexer for ActiveRecord working. First the get_field_names disappears (now back), then I am collectig some major trouble with locking. Same thing here: exception 6 not handled: Could not obtain write lock when trying to write index A snippet like this just deadlocks retrying endlessly: begin @ferret_index << doc @ferret_index.flush() @ferret_index.close() rescue Exception => e # No, he couldn''t define a proper class for this if e.to_s.include?(''Could not obtain write lock'') reopen_index # opens the index again! retry else raise e end end How are we supposed to handle concurrency with a file store? I can''t find anything in the wiki and actually I am getting very frustrated. It''s the third gem update of Ferret and my plugin just got broken, I can''t repair it since. Also the habit of throwing Exceptions is somewhat obnoxious because they are not standard errors. Might Ferret once get his own error class tree? -- Posted via http://www.ruby-forum.com/.
David Balmain
2006-Jun-30 23:02 UTC
[Ferret-talk] Substantial problems with write locking (and other flux)
On 6/30/06, Julik <me at julik.nl> wrote:> I am having some great trouble keeping my Ferret indexer for > ActiveRecord working. > > First the get_field_names disappears (now back), then I am collectig > some major trouble with locking. Same thing here:I ported 20,000 lines of ruby code to C. I apologize if some parts of the API got left out in the process. People wanted a faster search library so that''s what I''m working on. It''s still beta, although it should still be alpha.> exception 6 not handled: Could not obtain write lock when trying to > write indexDo you have more than one process writing to the index? The C version of Ferret currently doesn''t wait long enough for the lock to be released. This behaviour will improve once I get the current release finished although I''m guessing that locking problems are always going to occur.> A snippet like this just deadlocks retrying endlessly: > > begin > @ferret_index << doc > @ferret_index.flush() > @ferret_index.close() > rescue Exception => e # No, he couldn''t define a proper class for this > if e.to_s.include?(''Could not obtain write lock'') > reopen_index # opens the index again! > retry > else > raise e > end > end> How are we supposed to handle concurrency with a file store? I can''t > find anything in the wiki and actually I am getting very frustrated. > It''s the third gem update of Ferret and my plugin just got broken, I > can''t repair it since. > > Also the habit of throwing Exceptions is somewhat obnoxious because they > are not standard errors. Might Ferret once get his own error class tree?I''m working on this too. I''ve done a significant rewrite of cFerret so that it will gel better with Ruby. I''ve basically rewritten 10,000 LOC which is why I haven''t had time to respond to the most recent tickets. All the problems people are having are going to be addressed. Exception handling will be significantly improved. Hopefully I''ll be able to eliminate segfaults. A windows version is also on the way. So please be patient. Or fix it yourself and send in a patch. Cheers, Dave
Julik
2006-Jul-01 01:49 UTC
[Ferret-talk] Substantial problems with write locking (and other flux)
David Balmain wrote:> It''s still beta, although it > should still be alpha.Got the message :-)> Do you have more than one process writing to the index? The C version > of Ferret currently doesn''t wait long enough for the lock to be > released. This behaviour will improve once I get the current release > finished although I''m guessing that locking problems are always going > to occur.No, it throws within a unit test which runs inside a single process. I ain''t even got to any real concurrency yet :-)> I''m working on this too. I''ve done a significant rewrite of cFerret so > that it will gel better with Ruby. I''ve basically rewritten 10,000 LOC > which is why I haven''t had time to respond to the most recent tickets.I realise I vented too much, so apologies for that. Great kudos to you for the work you are doing on Ferret. Would you please put out a gem with the "get_field_names" back in the picture? This is the only thing broken after some major patching I did on my plugin. Would you be so kind to give me a tip what "get_field_names" ought to return when casted to an Array so that I can replicate it''s functionality (I have the names of fields stored elsewhere, I just need to pass them when constructing the search). TIA. -- Posted via http://www.ruby-forum.com/.
Julik
2006-Jul-01 01:50 UTC
[Ferret-talk] Substantial problems with write locking (and other flux)
Julik wrote:> No, it throws within a unit test which runs inside a single process. I > ain''t even got to any real concurrency yet :-)And to be fair - the problem seems to be removed altogether by using auto_flush. So now the get_field_names thing is the only one and then I can hack further and switch an app to Ferret. -- Posted via http://www.ruby-forum.com/.
David Balmain
2006-Jul-01 07:49 UTC
[Ferret-talk] Substantial problems with write locking (and other flux)
On 7/1/06, Julik <me at julik.nl> wrote:> Julik wrote: > > > No, it throws within a unit test which runs inside a single process. I > > ain''t even got to any real concurrency yet :-) > > And to be fair - the problem seems to be removed altogether by using > auto_flush. So now the get_field_names thing is the only one and then I > can hack further and switch an app to Ferret.I''ve released the Gem. I''m not sure how much has changed since the last release as it''s been so long since I did any work on that branch, but it does have the get_field_names method and it raises a StandardError instead of Exception. But if I were you, I''d wait around for the next release of Ferret and possibly even join in the discussions on the API. There is going to be 0.10.x string of releases which will break backwards compatibility with 0.9.x and earlier. The goal is to get out a 1.0 release with a stable API. I''ll then come back and fix the bugs in 0.9.x for people who aren''t able to change over, but I won''t be supporting it for very long. Cheers, Dave