Figured it out. It''s interesting, academically.
I was flushing every time I added something to the index. I forget exactly
why I thought that was a good idea, but that''s what I was doing.
Apparently, that''s a bad idea under 0.11. Worked fine under 0.10, not
so
fine under 0.11. Like, 75x less fine.
Now I''m just flushing after every batch. It''s back to
reasonable,
5-seconds-for-10,000-small-records performance.
Thanks for all your help, Dave.
Schnitz
On 3/5/07, David Balmain <dbalmain.ml at gmail.com>
wrote:>
> On 3/5/07, Matt Schnitz <matt at mattschnitz.com> wrote:
> > Hi - I upgraded to Ferret 0.11.3 from 0.10.13.
> >
> > I used to index 10,000 records in 10 secs. Now it takes 13 minutes.
> > (That''s a factor of ~75x)
> >
> > Did something change in the flush semantics, or something?
>
> Hi Matt,
>
> The opening of an index takes a little longer now. I guess if you have
> the index set to :auto_flush then it could take a fair bit longer but
> I didn''t expect it to take that much longer. Unfortunately this
> slowdown was a price I had to pay to prevent the segfault and
> FileNotFound errors that people where getting. Having said that, you
> shouldn''t have :auto_flush set when you are batch indexing anyway.
>
> If you send me a benchmark which approximates what you are doing,
I''d
> be happy to take a look at it for you and tell you how to make it
> faster or add a fix to Ferret if the problem does happen to be at this
> end.
>
> Cheers,
> Dave
>
> --
> Dave Balmain
> http://www.davebalmain.com/
> _______________________________________________
> Ferret-talk mailing list
> Ferret-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/ferret-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/ferret-talk/attachments/20070305/7e9e0574/attachment.html