Ferret talk - Aug 2006 - [0.10.0] Index#search is not thread safe ?

This script (http://pastie.caboo.se/10371) give this result :

"1"
"0"
"0"
"0"

Why the other thread does not have the same result ?
Maybe, it''s not the correct way to use the index in a multi threaded 
environement but I don''t know how to do.

Any ideas ?

-- 
Posted via http://www.ruby-forum.com/.

On Sat, Aug 26, 2006 at 04:12:57PM +0200, Florent Solt
wrote:
> This script (http://pastie.caboo.se/10371) give this result :
> 
> "1"
> "0"
> "0"
> "0"
> 
> Why the other thread does not have the same result ?
> Maybe, it''s not the correct way to use the index in a multi
threaded
> environement but I don''t know how to do.
This should work, imho. I even tried to synchronize access to the index
like that:

guard = Mutex.new
[..]
and in the thread''s block:
  guard.synchronize do
    result = index.search(''id:42'')
    p result.total_hits
  end

but the output stayed the same.
> Any ideas ?
You might try to use a Searcher instance for your searching in the
threads. Maybe the problem lies only in the Index class.

Your best bet however is to create separate Index instances in each 
thread.

Jens

-- 
webit! Gesellschaft f?r neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Kr?mer       kraemer at webit.de
Schnorrstra?e 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

> 
> Your best bet however is to create separate Index instances in each
> thread.
> 
Thanks for your answer, but this example was inspirate by what I do with 
Rails and Ferret. And the main problem is that I have many ajax calls, 
each using the index... If I open/close for each ajax call an index, it 
will be too expensive, no ? What are your opinion ?

PS: I will try tomorrow with a Searcher, stay tuned :)

Thanks again.

-- 
Posted via http://www.ruby-forum.com/.

On Sun, Aug 27, 2006 at 07:43:49PM +0200, Florent Solt
wrote:
> > 
> > Your best bet however is to create separate Index instances in each
> > thread.
> > 
> 
> Thanks for your answer, but this example was inspirate by what I do with 
> Rails and Ferret. And the main problem is that I have many ajax calls, 
> each using the index... If I open/close for each ajax call an index, it 
> will be too expensive, no ? What are your opinion ?
if you''re using Rails, you usually won''t encounter
multithreaded access
to a single index instance, since Rails is not threaded (or at least is
not intended to run in a multithreaded way). 

So in any Rails app you''ll have at most multiple OS processes accessing
one physical index, but not multiple threads accessing one instance of 
class Index. Locking problems on physical index level may occur, though.
> PS: I will try tomorrow with a Searcher, stay tuned :)
A Searcher should not try to write-lock, so you''ll be on the safe side
as long as only the batch-indexing process will write to the index.

Jens

-- 
webit! Gesellschaft f?r neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Kr?mer       kraemer at webit.de
Schnorrstra?e 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

> 
> if you''re using Rails, you usually won''t encounter
multithreaded access
> to a single index instance, since Rails is not threaded (or at least is
> not intended to run in a multithreaded way).
But, tell me if I''m wrong, in dev env Rails use webrick, and webrick 
create
a thread for each client, so each ajax call. Rails does not create 
threads,
it''s true, but webrick does. And the instance of DispatchServlet is 
share
between all client thread, yes or not ?


-- 
Posted via http://www.ruby-forum.com/.

On Mon, Aug 28, 2006 at 11:46:55AM +0200, Florent Solt
wrote:
> 
> > 
> > if you''re using Rails, you usually won''t encounter
multithreaded access
> > to a single index instance, since Rails is not threaded (or at least
is
> > not intended to run in a multithreaded way).
> 
> But, tell me if I''m wrong, in dev env Rails use webrick, and
webrick
> create
> a thread for each client, so each ajax call. Rails does not create 
> threads,
> it''s true, but webrick does. And the instance of DispatchServlet
is
> share
> between all client thread, yes or not ?
I don''t know how Webrick handles Rails, but with lighty/fastcgi you
have
seperate fastcgi processes that don''t share anything and each only
handle one request at a time. 

Jens

-- 
webit! Gesellschaft f?r neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Kr?mer       kraemer at webit.de
Schnorrstra?e 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

> I don''t know how Webrick handles Rails, but with lighty/fastcgi
you have
> seperate fastcgi processes that don''t share anything and each only
> handle one request at a time.
Thanks a lot for you answer.

I thought both dev/prod environement had the same way to deal with 
threads and in production (with lighttpd/fastcgi) each rails process 
could handle several clients at the same time, but I was wrong. I''ve 
just re-read the FCGI dispatcher and you are right Jens. Thanks again.

-- 
Posted via http://www.ruby-forum.com/.

On 8/28/06, Jens Kraemer <kraemer at webit.de>
wrote:
> On Sat, Aug 26, 2006 at 04:12:57PM +0200, Florent Solt wrote:
> > This script (http://pastie.caboo.se/10371) give this result :
> >
> > "1"
> > "0"
> > "0"
> > "0"
> >
> > Why the other thread does not have the same result ?
> > Maybe, it''s not the correct way to use the index in a multi
threaded
> > environement but I don''t know how to do.
>
> This should work, imho. I even tried to synchronize access to the index
> like that:
>
> guard = Mutex.new
> [..]
> and in the thread''s block:
>   guard.synchronize do
>     result = index.search(''id:42'')
>     p result.total_hits
>   end
>
> but the output stayed the same.
This was a bug that has been fixed in the current working copy. The
fix will be released in version 0.10.2.

Cheers,
Dave

Thanks Dave, I''ll check it.

-- 
Posted via http://www.ruby-forum.com/.