Hi,
I am experiencing very poor performance when updating my index. For
example, to update the index for 10 documents, it is taking 3 to 4
seconds. My index is currently very small... with probably less than
100 docs in it.
I have created my index as follows:
GIFT_INDEX = Index::Index.new(:path => "#{index_dir}/gift", :key
=>
''id'', :auto_flush => true)
and I have an after_save filter in my model as follows:
def update_index
INDEX << self.to_doc
end
Is there anything I can do to improve this performance?
Thanks,
Tom
On 3/1/06, Tom Davies <atomgiant at gmail.com> wrote:> Hi, > > I am experiencing very poor performance when updating my index. For > example, to update the index for 10 documents, it is taking 3 to 4 > seconds. My index is currently very small... with probably less than > 100 docs in it. > > I have created my index as follows: > > GIFT_INDEX = Index::Index.new(:path => "#{index_dir}/gift", :key => > ''id'', :auto_flush => true) > > and I have an after_save filter in my model as follows: > > def update_index > INDEX << self.to_doc > end > > > Is there anything I can do to improve this performance?Hi Tom, That sounds very slow. How large are the documents? The first thing you can do is turn off auto_flush. That should substantially speed things up. If you only have one thread you won''t need auto_flush. If you have more then one thread then I''d suggest having a dedicated indexing thread (and again you won''t need auto_flush). If things are still too slow after that, I''m nearly finished with the C rewrite of ferret. A linux version should be out some time next week. This will be at least 10 times as fast. Hope that helps, Cheers, Dave> > Thanks, > Tom > > _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk >
That''s great, David. Is the C version of ferret similar enough to the current version so that someone could develop on windows or osx with the current version of ferret and deploy on linux with cFerret? On 3/1/06, David Balmain <dbalmain.ml at gmail.com> wrote:> On 3/1/06, Tom Davies <atomgiant at gmail.com> wrote: > > Hi, > > > > I am experiencing very poor performance when updating my index. For > > example, to update the index for 10 documents, it is taking 3 to 4 > > seconds. My index is currently very small... with probably less than > > 100 docs in it. > > > > I have created my index as follows: > > > > GIFT_INDEX = Index::Index.new(:path => "#{index_dir}/gift", :key => > > ''id'', :auto_flush => true) > > > > and I have an after_save filter in my model as follows: > > > > def update_index > > INDEX << self.to_doc > > end > > > > > > Is there anything I can do to improve this performance? > > Hi Tom, > > That sounds very slow. How large are the documents? > > The first thing you can do is turn off auto_flush. That should > substantially speed things up. If you only have one thread you won''t > need auto_flush. If you have more then one thread then I''d suggest > having a dedicated indexing thread (and again you won''t need > auto_flush). If things are still too slow after that, I''m nearly > finished with the C rewrite of ferret. A linux version should be out > some time next week. This will be at least 10 times as fast. > > Hope that helps, > > Cheers, > Dave > > > > > > Thanks, > > Tom > > > > _______________________________________________ > > Ferret-talk mailing list > > Ferret-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/ferret-talk > > > > _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk >
On 3/1/06, Gary Elliott <garypelliott at gmail.com> wrote:> That''s great, David. Is the C version of ferret similar enough to the > current version so that someone could develop on windows or osx with > the current version of ferret and deploy on linux with cFerret? >It''s similar enough that it won''t take long to port but it won''t be exactly the same. The pure ruby version is being phased out, but I do plan have a windows version of cferret. Hopefully a keen windows developer will lend a hand. Unfortunately I don''t have the microsoft C compiler.
Hi Dave,
That is good news about the Linux version. I tried turning off
autoflush, but that did not appear to have a very noticable
difference.
I just added the following benchmark code around the index update:
def update_index
Gift.benchmark("updating index") do
INDEX << self.to_doc
end
end
I am attaching a trace of running this against 28 docs, which is the
entire index. The documents are not large, as you will see from the
trace. Each index update is averaging around .4 seconds. Does that
seem acceptable? I am hoping this turns out to be some sort of
configuration error.
Tom
On 3/1/06, Gary Elliott <garypelliott at gmail.com>
wrote:> That''s great, David. Is the C version of ferret similar enough to
the
> current version so that someone could develop on windows or osx with
> the current version of ferret and deploy on linux with cFerret?
>
> On 3/1/06, David Balmain <dbalmain.ml at gmail.com> wrote:
> > On 3/1/06, Tom Davies <atomgiant at gmail.com> wrote:
> > > Hi,
> > >
> > > I am experiencing very poor performance when updating my index.
For
> > > example, to update the index for 10 documents, it is taking 3 to
4
> > > seconds. My index is currently very small... with probably less
than
> > > 100 docs in it.
> > >
> > > I have created my index as follows:
> > >
> > > GIFT_INDEX = Index::Index.new(:path =>
"#{index_dir}/gift", :key =>
> > > ''id'', :auto_flush => true)
> > >
> > > and I have an after_save filter in my model as follows:
> > >
> > > def update_index
> > > INDEX << self.to_doc
> > > end
> > >
> > >
> > > Is there anything I can do to improve this performance?
> >
> > Hi Tom,
> >
> > That sounds very slow. How large are the documents?
> >
> > The first thing you can do is turn off auto_flush. That should
> > substantially speed things up. If you only have one thread you
won''t
> > need auto_flush. If you have more then one thread then I''d
suggest
> > having a dedicated indexing thread (and again you won''t need
> > auto_flush). If things are still too slow after that, I''m
nearly
> > finished with the C rewrite of ferret. A linux version should be out
> > some time next week. This will be at least 10 times as fast.
> >
> > Hope that helps,
> >
> > Cheers,
> > Dave
> >
> >
> > >
> > > Thanks,
> > > Tom
> > >
> > > _______________________________________________
> > > Ferret-talk mailing list
> > > Ferret-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/ferret-talk
> > >
> >
> > _______________________________________________
> > Ferret-talk mailing list
> > Ferret-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/ferret-talk
> >
>
> _______________________________________________
> Ferret-talk mailing list
> Ferret-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/ferret-talk
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ferret.log
Type: application/octet-stream
Size: 23845 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/ferret-talk/attachments/20060301/41c1b379/ferret-0001.obj
Hi Tom, It does seem rather slow. The unit tests are adding a few hundred documents and they''re finishing in under ten seconds on my machine. I''d suggest you don''t worry about it too much though as the new version of Ferret will solve any performance problems you''re having. Cheers, Dave On 3/1/06, Tom Davies <atomgiant at gmail.com> wrote:> Hi Dave, > > That is good news about the Linux version. I tried turning off > autoflush, but that did not appear to have a very noticable > difference. > > I just added the following benchmark code around the index update: > def update_index > Gift.benchmark("updating index") do > INDEX << self.to_doc > end > end > > I am attaching a trace of running this against 28 docs, which is the > entire index. The documents are not large, as you will see from the > trace. Each index update is averaging around .4 seconds. Does that > seem acceptable? I am hoping this turns out to be some sort of > configuration error. > > Tom > > > > On 3/1/06, Gary Elliott <garypelliott at gmail.com> wrote: > > That''s great, David. Is the C version of ferret similar enough to the > > current version so that someone could develop on windows or osx with > > the current version of ferret and deploy on linux with cFerret? > > > > On 3/1/06, David Balmain <dbalmain.ml at gmail.com> wrote: > > > On 3/1/06, Tom Davies <atomgiant at gmail.com> wrote: > > > > Hi, > > > > > > > > I am experiencing very poor performance when updating my index. For > > > > example, to update the index for 10 documents, it is taking 3 to 4 > > > > seconds. My index is currently very small... with probably less than > > > > 100 docs in it. > > > > > > > > I have created my index as follows: > > > > > > > > GIFT_INDEX = Index::Index.new(:path => "#{index_dir}/gift", :key => > > > > ''id'', :auto_flush => true) > > > > > > > > and I have an after_save filter in my model as follows: > > > > > > > > def update_index > > > > INDEX << self.to_doc > > > > end > > > > > > > > > > > > Is there anything I can do to improve this performance? > > > > > > Hi Tom, > > > > > > That sounds very slow. How large are the documents? > > > > > > The first thing you can do is turn off auto_flush. That should > > > substantially speed things up. If you only have one thread you won''t > > > need auto_flush. If you have more then one thread then I''d suggest > > > having a dedicated indexing thread (and again you won''t need > > > auto_flush). If things are still too slow after that, I''m nearly > > > finished with the C rewrite of ferret. A linux version should be out > > > some time next week. This will be at least 10 times as fast. > > > > > > Hope that helps, > > > > > > Cheers, > > > Dave > > > > > > > > > > > > > > Thanks, > > > > Tom > > > > > > > > _______________________________________________ > > > > Ferret-talk mailing list > > > > Ferret-talk at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/ferret-talk > > > > > > > > > > _______________________________________________ > > > Ferret-talk mailing list > > > Ferret-talk at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/ferret-talk > > > > > > > _______________________________________________ > > Ferret-talk mailing list > > Ferret-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/ferret-talk > > > > > _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk > > >
David Balmain wrote:> I''d suggest you don''t worry about it too much though as the new > version of Ferret will solve any performance problems you''re having.I''m looking forward to trying the new version, as I have quite a lot of problems with the current version on ruby-forum.com. -- Posted via http://www.ruby-forum.com/.
Andreas, do you just mean performance problems or other bugs as well? On 3/3/06, Andreas S. <f at andreas-s.net> wrote:> David Balmain wrote: > > I''d suggest you don''t worry about it too much though as the new > > version of Ferret will solve any performance problems you''re having. > > I''m looking forward to trying the new version, as I have quite a lot of > problems with the current version on ruby-forum.com. > > -- > Posted via http://www.ruby-forum.com/. > _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk >
Gary Elliott wrote:> Andreas, do you just mean performance problems or other bugs as well?Other bugs as well, occasional crashes, stale lockfiles, exploding index file size (see bugtracker on Ferret website). -- Posted via http://www.ruby-forum.com/.
On 3/3/06, Andreas S. <f at andreas-s.net> wrote:> David Balmain wrote: > > I''d suggest you don''t worry about it too much though as the new > > version of Ferret will solve any performance problems you''re having. > > I''m looking forward to trying the new version, as I have quite a lot of > problems with the current version on ruby-forum.com.Once the new version is out I''ll be able to be a lot more responsive when it comes to fixing bugs and addressing other problems. I should warn though that the new version is an alpha release and it''s going to take some work before it''s even as stable as the current ruby version. It''ll get there though.> > -- > Posted via http://www.ruby-forum.com/. > _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk >