Ferret talk - Aug 2008 - Rép : ignored or forgotten values in reindexing

Hi,

Le 27 ao?t 08 ? 02:51, Chris Strom a ?crit :
> On Tue, Aug 26, 2008 at 7:53 PM, lists.jc.michel at symetrie.com
> <lists.jc.michel at symetrie.com> wrote:
>
>> Using acts_as_ferret with rails 2.1, I encounter some strange  
>> behaviour in
>> reindexing all records from my model.
>> Some seem to be ignored. The log generated by reindexing misses  
>> some values;
>> but no error is printed about parsing these strings.
>> I was alerted by some results missing.
>>
>
> I have seen this problem as well.  Same versions of ferret/aaf
> (0.11.6/0.4.3).  Differences are Rails 2.0 and using SQL Server.
I use posgresql, didn''t mention it.

> The limited amount of investigation that I have been able to do
> suggests a fence post error in the bulk index pagination.  Of 2000+
> objects, 2 are missing.  I''m only basing the fence post conjecture
on
> the number of missing objects & the default pagination window (1000).
> I have not even had a chance to investigate changing the window size,
> let alone investigate deeper.

I observed this behaviour with reindexing, even from script/console,  
without any pagination interference.

Jean-Christophe Michel
jc.michel at symetrie.com

On Wed, Aug 27, 2008 at 10:30 AM, lists.jc.michel at symetrie.com
<lists.jc.michel at symetrie.com> wrote:
>> The limited amount of investigation that I have been able to do
>> suggests a fence post error in the bulk index pagination.  Of 2000+
>> objects, 2 are missing.  I''m only basing the fence post
conjecture on
>> the number of missing objects & the default pagination window
(1000).
>> I have not even had a chance to investigate changing the window size,
>> let alone investigate deeper.
>
>
> I observed this behaviour with reindexing, even from script/console,
without
> any pagination interference.
Reindexing uses the BulkIndexer as well does it not?  Even from
script/console, unless I am mistaken.

-Chris

Hi!

I must admit that I don''t do regular testing with postgresql, maybe  
its some db related issue. I''ll look into it. Could you please open a  
ticket at projects.jkraemer.net/acts_as_ferret ?

cheers,
Jens

On 27.08.2008, at 16:30, lists.jc.michel at symetrie.com wrote:
> Hi,
>
> Le 27 ao?t 08 ? 02:51, Chris Strom a ?crit :
>> On Tue, Aug 26, 2008 at 7:53 PM, lists.jc.michel at symetrie.com
>> <lists.jc.michel at symetrie.com> wrote:
>>
>>> Using acts_as_ferret with rails 2.1, I encounter some strange  
>>> behaviour in
>>> reindexing all records from my model.
>>> Some seem to be ignored. The log generated by reindexing misses  
>>> some values;
>>> but no error is printed about parsing these strings.
>>> I was alerted by some results missing.
>>>
>>
>> I have seen this problem as well.  Same versions of ferret/aaf
>> (0.11.6/0.4.3).  Differences are Rails 2.0 and using SQL Server.
>
> I use posgresql, didn''t mention it.
>
>
>> The limited amount of investigation that I have been able to do
>> suggests a fence post error in the bulk index pagination.  Of 2000+
>> objects, 2 are missing.  I''m only basing the fence post
conjecture on
>> the number of missing objects & the default pagination window
(1000).
>> I have not even had a chance to investigate changing the window size,
>> let alone investigate deeper.
>
>
> I observed this behaviour with reindexing, even from script/console,  
> without any pagination interference.
>
> Jean-Christophe Michel
> jc.michel at symetrie.com
>
> _______________________________________________
> Ferret-talk mailing list
> Ferret-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/ferret-talk
>
--
Jens Kr?mer
webit! Gesellschaft f?r neue Medien mbH
Schnorrstra?e 76 | 01069 Dresden
Telefon +49351467660 | Telefax +493514676666
kraemer at webit.de | www.webit.de

Amtsgericht Dresden | HRB 15422
GF Sven Haubold

Le 28 ao?t 08 ? 09:25, Jens Kr?mer a ?crit :
> I must admit that I don''t do regular testing with postgresql,
maybe
> its some db related issue. I''ll look into it. Could you please
open
> a ticket at projects.jkraemer.net/acts_as_ferret ?
Sure ! but I''d like to diagnose a bit more. How can I get some error  
output instead of silently ignored rows ?

Hi,

Le 28 ao?t 08 ? 09:25, Jens Kr?mer a ?crit :
> I must admit that I don''t do regular testing with postgresql,
maybe
> its some db related issue. I''ll look into it. Could you please
open
> a ticket at projects.jkraemer.net/acts_as_ferret ?
Thanks for your quick reply.
I think I found a mistake of mine with this other thread :

Le 28 ao?t 08 ? 17:19, Sheldon Maloff a ?crit :
> Now, corrupt indexes in development is another issue. In  
> development, you are not running a DRb server. Each mongrel is  
> hitting the index directly. You typically have only one mongrel  
> running in development. But if you open an interactive script/ 
> console session, and play with your models side-by-side a running  
> mongrel, you WILL corrupt your Ferret index. That''s because both  
> the mongrel and the script/console will be writers to the same  
> index, something that Ferret doesn''t support. Heck, running a rake
> db:migrate along side a running mongrel will cause index  
> corruption, for the same reason: multiple writers.
In fact I probably had mongrel and console running together when I  
reindexed in development.
I shut down and dropped my index dir, then did bulk_index on my class  
in console.
Seems all my words were indexed.

Maybe we could think to some error messages issued by each index  
writer :
could be like
   logger.debug ''Man, chances are you are messing up your index by  
writing concurrently to it'' if (development) && writer_flag
The flag being a development only feature, enabled by default,  
disabled in production only.
What do you think of this ?