Ferret talk - Mar 2007 - Warming up a new Searcher/Reader (Ferret 0.10.9 win32)

Hi,

I have a largish index [700MB]  which is updated from time to time,
requiring me to close and recreate the Ferret::Search::Searcher to use
the latest index.

My problem is that the first few searches on the new index are slow [by
comparison to before the close/recreate], I''m guessing because the new
index is being loaded into RAM by my OS and into Ferret as needed. 

I''m thinking of "warming up" the OS and Ferret by processing
a query or
two, and I''d appreciate any details about the way Ferret loads the
index, so that I can construct a good "warm up" query.

Kind Regards

Neville


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/ferret-talk/attachments/20070305/13b4bf16/attachment.html

On 3/5/07, Neville Burnell <Neville.Burnell at bmsoft.com.au>
wrote:
>
>
>
> Hi,
>
> I have a largish index [700MB]  which is updated from time to time,
> requiring me to close and recreate the Ferret::Search::Searcher to use the
> latest index.
>
> My problem is that the first few searches on the new index are slow [by
> comparison to before the close/recreate], I''m guessing because the
new index
> is being loaded into RAM by my OS and into Ferret as needed.
>
> I''m thinking of "warming up" the OS and Ferret by
processing a query or two,
> and I''d appreciate any details about the way Ferret loads the
index, so that
> I can construct a good "warm up" query.
Hi Neville,

Ferret loads the index for each field when it is searched. So if you
only search one field, only that field''s index will be loaded.

Once each fields index is loaded, ferret should be fully warmed up as
far as simple queries like phrase and boolean go. If you are sorting
search results, then sort indexes will also need to be built. A new
sort index is built for each sorted field depending on sort type (int,
float, string, byte) and sort direction (normal and reverse). So if
you are sorting your search results you will also need to try each
type of sort that you might use to warm up that part of the index.

That''s all I can think of at the moment. One thing I have planned for
the future is adding the ability to autoload all the indexes and to
save sort indexes rather than building them each time you open an
index reader.

Cheers,
Dave

-- 
Dave Balmain
http://www.davebalmain.com/

Thanks for the detail Dave!
> One thing I have planned for the future is adding the ability 
> to autoload all the indexes
Awesome - this sounds like just what I''m looking for.

Cheers,

Nev
> -----Original Message-----
> From: ferret-talk-bounces at rubyforge.org [mailto:ferret-talk-
> bounces at rubyforge.org] On Behalf Of David Balmain
> Sent: Tuesday, 6 March 2007 2:33 PM
> To: ferret-talk at rubyforge.org
> Subject: Re: [Ferret-talk] Warming up a new Searcher/Reader (Ferret
> 0.10.9win32)
> 
> On 3/5/07, Neville Burnell <Neville.Burnell at bmsoft.com.au> wrote:
> >
> >
> >
> > Hi,
> >
> > I have a largish index [700MB]  which is updated from time to time,
> > requiring me to close and recreate the Ferret::Search::Searcher to
> use the
> > latest index.
> >
> > My problem is that the first few searches on the new index are slow
> [by
> > comparison to before the close/recreate], I''m guessing
because the
> new index
> > is being loaded into RAM by my OS and into Ferret as needed.
> >
> > I''m thinking of "warming up" the OS and Ferret by
processing a query
> or two,
> > and I''d appreciate any details about the way Ferret loads the
index,
> so that
> > I can construct a good "warm up" query.
> 
> Hi Neville,
> 
> Ferret loads the index for each field when it is searched. So if you
> only search one field, only that field''s index will be loaded.
> 
> Once each fields index is loaded, ferret should be fully warmed up as
> far as simple queries like phrase and boolean go. If you are sorting
> search results, then sort indexes will also need to be built. A new
> sort index is built for each sorted field depending on sort type (int,
> float, string, byte) and sort direction (normal and reverse). So if
> you are sorting your search results you will also need to try each
> type of sort that you might use to warm up that part of the index.
> 
> That''s all I can think of at the moment. One thing I have planned
for
> the future is adding the ability to autoload all the indexes and to
> save sort indexes rather than building them each time you open an
> index reader.
> 
> Cheers,
> Dave
> 
> --
> Dave Balmain
> http://www.davebalmain.com/
> _______________________________________________
> Ferret-talk mailing list
> Ferret-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/ferret-talk