Ferret talk - Oct 2006 - Search result inconsistencies due to indexing

I seem to be having problems with getting my searcher to be consistent 
while indexing.  I am running the latest version of ferret (0.10.13) and 
I am using the Searchable plugin.  Currently the way it indexes is by 
using a callback in the model of either after_update or after_create to 
index the fields that I have setup to be indexed.

Right now I update the index about once every 4 or 5 minutes.  The 
problem occurs when I do a search and either no results ( or a very 
small amount like 2) are returned for particular keywords that 
previously had plenty of results.  I am thinking that it has something 
to do with indexing or possibly that Searchable does not optimize after 
it adds to the index.

I currently have about 10,000 records in my index and am fairly new to 
ferret.  Any assistance would be helpful.  Thanks.

Andy

-- 
Posted via http://www.ruby-forum.com/.

On Wed, Oct 25, 2006 at 06:41:14AM +0200, William Minkstein
wrote:
> I seem to be having problems with getting my searcher to be consistent 
> while indexing.  I am running the latest version of ferret (0.10.13) and 
> I am using the Searchable plugin.  Currently the way it indexes is by 
> using a callback in the model of either after_update or after_create to 
> index the fields that I have setup to be indexed.
huh, I didn''t ever hear about that plugin before - the DrB remote
indexing stuff is quite interesting indeed.
> Right now I update the index about once every 4 or 5 minutes.  The 
> problem occurs when I do a search and either no results ( or a very 
> small amount like 2) are returned for particular keywords that 
> previously had plenty of results.  I am thinking that it has something 
> to do with indexing or possibly that Searchable does not optimize after 
> it adds to the index.
Did you contact the plugin author about this ? 


cheers,
Jens

-- 
webit! Gesellschaft f?r neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Kr?mer       kraemer at webit.de
Schnorrstra?e 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

Jens Kraemer wrote:
> On Wed, Oct 25, 2006 at 06:41:14AM +0200, William Minkstein wrote:
>> I seem to be having problems with getting my searcher to be consistent 
>> while indexing.  I am running the latest version of ferret (0.10.13)
and
>> I am using the Searchable plugin.  Currently the way it indexes is by 
>> using a callback in the model of either after_update or after_create to
>> index the fields that I have setup to be indexed.
> 
> huh, I didn''t ever hear about that plugin before - the DrB remote
> indexing stuff is quite interesting indeed.
> 
The url for author''s page is: http://searchable.rubyforge.org/  There 
are things about it that are very convenient.
>> Right now I update the index about once every 4 or 5 minutes.  The 
>> problem occurs when I do a search and either no results ( or a very 
>> small amount like 2) are returned for particular keywords that 
>> previously had plenty of results.  I am thinking that it has something 
>> to do with indexing or possibly that Searchable does not optimize after
>> it adds to the index.
> 
> Did you contact the plugin author about this ?
> 
> 
Yes I did contact the author about this.  I checked the Searchable code 
that indexes records and it seems to be pretty consistent with how 
people are indexing with just Ferret itself.  I guess I was wondering if 
anyone else experienced having inconsistent search results after 
updating or adding records to their index.
> cheers,
> Jens
> 
> --
> webit! Gesellschaft f?r neue Medien mbH          www.webit.de
> Dipl.-Wirtschaftsingenieur Jens Kr?mer       kraemer at webit.de
> Schnorrstra?e 76                         Tel +49 351 46766  0
> D-01069 Dresden                          Fax +49 351 46766 66
Andy

-- 
Posted via http://www.ruby-forum.com/.

On 10/25/06, William (Andy) Minkstein <wminkstein at gmail.com>
wrote:
> Jens Kraemer wrote:
> > On Wed, Oct 25, 2006 at 06:41:14AM +0200, William Minkstein wrote:
> >> I seem to be having problems with getting my searcher to be
consistent
> >> while indexing.  I am running the latest version of ferret
(0.10.13) and
> >> I am using the Searchable plugin.  Currently the way it indexes is
by
> >> using a callback in the model of either after_update or
after_create to
> >> index the fields that I have setup to be indexed.
> >
> > huh, I didn''t ever hear about that plugin before - the DrB
remote
> > indexing stuff is quite interesting indeed.
> >
> The url for author''s page is: http://searchable.rubyforge.org/ 
There
> are things about it that are very convenient.
Hehe. I hadn''t seen this either. Seth Fitzsimmons, if you''re
out
there, nice work. I''m going to be adding a DRb server to Ferret soon.
I''ll definitely be checking out your code. If you''d like to
contribute, please do. :)
> >> Right now I update the index about once every 4 or 5 minutes.  The
> >> problem occurs when I do a search and either no results ( or a
very
> >> small amount like 2) are returned for particular keywords that
> >> previously had plenty of results.  I am thinking that it has
something
> >> to do with indexing or possibly that Searchable does not optimize
after
> >> it adds to the index.
> >
> > Did you contact the plugin author about this ?
> >
> >
>
> Yes I did contact the author about this.  I checked the Searchable code
> that indexes records and it seems to be pretty consistent with how
> people are indexing with just Ferret itself.  I guess I was wondering if
> anyone else experienced having inconsistent search results after
> updating or adding records to their index.
I haven''t seen this problem in version 0.10.13. The way I would go
about debugging it, though, is to store all the fields in the index.
Then if you look at a certain document in the index and it contains
the data you''re searching for but doesn''t get matched in the
search
results it is a bug. In this case you can send me a zipped up copy of
the index and I''ll fix the problem. Otherwise I''m not sure
there''s
much else we can do (unless you can give me ssh access to your
server).

Cheers,
Dave

Hey David,

Thanks for the quick response.  Sorry about the delay in responding, we 
had a deadline we had to meet for this app we''re developing and that
has
been taking up most of my life.

I am having a lot of trouble reproducing the bug I first reported.  I 
talked to the writer of the Searchable plugin (Seth) and he is stumped 
as well.  It seems that creating an index from scratch works fine and 
the bug only occurs when documents are added or deleted.

A weird thing happens though once searchable starts updating the index. 
It creates folders inside the index directory called development and 
production.  Seth claims that this is not desired behavior but I guess I 
was wondering if having files or directories like that in the index 
directory would cause it to behave strangely?

Another question I have is can certain characters in the text that is 
being indexed cause problems when trying to retrieve search results? 
Should I be filtering out any non-alphanumerics?  Not sure if that 
matters at all but like I said I am new to this.

Andy

David Balmain wrote:
> On 10/25/06, William (Andy) Minkstein <wminkstein at gmail.com>
wrote:
>> >
>> The url for author''s page is: http://searchable.rubyforge.org/
There
>> are things about it that are very convenient.
> 
> Hehe. I hadn''t seen this either. Seth Fitzsimmons, if
you''re out
> there, nice work. I''m going to be adding a DRb server to Ferret
soon.
> I''ll definitely be checking out your code. If you''d like
to
> contribute, please do. :)
> 
>>
>> Yes I did contact the author about this.  I checked the Searchable code
>> that indexes records and it seems to be pretty consistent with how
>> people are indexing with just Ferret itself.  I guess I was wondering
if
>> anyone else experienced having inconsistent search results after
>> updating or adding records to their index.
> 
> I haven''t seen this problem in version 0.10.13. The way I would go
> about debugging it, though, is to store all the fields in the index.
> Then if you look at a certain document in the index and it contains
> the data you''re searching for but doesn''t get matched in
the search
> results it is a bug. In this case you can send me a zipped up copy of
> the index and I''ll fix the problem. Otherwise I''m not
sure there''s
> much else we can do (unless you can give me ssh access to your
> server).
> 
> Cheers,
> Dave

-- 
Posted via http://www.ruby-forum.com/.