Hi all, there was a recent thread[1] on rails-deploy about Ferret in which a lot of people complained of problems using it in production. I''ve been using Ferret (with DRb) for many months now with no serious issues. I''m assuming the posters know what they''re doing so I''m guessing they''re just using Ferret in higher-scale environments than me. I spoke to someone in person yesterday who claimed that Ferret over DRb couldn''t keep up with their use rate and had been investigating replicating the ferret database between two machines. With all these bad experiences, I''d like to hear about some good experiences. Anyone care to comment? Anyone using it under huge load? Care to provide some numbers and some notes about how you''ve made it work? John. [1] http://groups.google.com/group/rubyonrails-deployment/browse_thread/thread/980fe7cb20cb97dd/bc798b52f439020c -- http://www.brightbox.co.uk - UK Ruby on Rails hosting
On Jan 25, 2008, at 7:02 AM, John Leach wrote:> Hi all, > > there was a recent thread[1] on rails-deploy about Ferret in which a > lot > of people complained of problems using it in production. > > I''ve been using Ferret (with DRb) for many months now with no serious > issues. I''m assuming the posters know what they''re doing so I''m > guessing they''re just using Ferret in higher-scale environments than > me. > > I spoke to someone in person yesterday who claimed that Ferret over > DRb > couldn''t keep up with their use rate and had been investigating > replicating the ferret database between two machines. > > With all these bad experiences, I''d like to hear about some good > experiences. Anyone care to comment? Anyone using it under huge > load? > Care to provide some numbers and some notes about how you''ve made it > work?I had used ferret w/ DrB at Technorati on a project that had several indexes of 5-10M documents. To make it work well I had to limit the update rates to the index (I think I took it down to about 1-2/s). To go to higher updates rates I would have had to change how we were writing and serving indexes. -ryan
I''m using Ferret/aaf with the DRb server under a medium load at Ravelry.com. I think that we peak at 10-12 queries per second and a little less than 1 update per second. My biggest problem has been indexing speed. I''ve been gradually switching over to Sphinx (http://www.sphinxsearch.com/) for indexes that don''t have to be updated in realtime (places where I can afford several minutes of lag). Emulating near realtime index updates in Sphinx is a little hacky but I find that it is worth it. Casey On Fri, 25 Jan 2008, Ryan King wrote:> On Jan 25, 2008, at 7:02 AM, John Leach wrote: >> Hi all, >> >> there was a recent thread[1] on rails-deploy about Ferret in which a >> lot >> of people complained of problems using it in production. >> >> I''ve been using Ferret (with DRb) for many months now with no serious >> issues. I''m assuming the posters know what they''re doing so I''m >> guessing they''re just using Ferret in higher-scale environments than >> me. >> >> I spoke to someone in person yesterday who claimed that Ferret over >> DRb >> couldn''t keep up with their use rate and had been investigating >> replicating the ferret database between two machines. >> >> With all these bad experiences, I''d like to hear about some good >> experiences. Anyone care to comment? Anyone using it under huge >> load? >> Care to provide some numbers and some notes about how you''ve made it >> work? > > I had used ferret w/ DrB at Technorati on a project that had several > indexes of 5-10M documents. To make it work well I had to limit the > update rates to the index (I think I took it down to about 1-2/s). > > To go to higher updates rates I would have had to change how we were > writing and serving indexes. > > -ryan > _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk >
I am successfully using ferret with a few caveats, none of which to my knowledge would be solved by using another solution. I believe many corruption problems are associated with a bug in aaf that causes multiple indexes to be built at the same time in the same place. I''m not sure if that''s been addressed as I didn''t see any response after reporting it months ago. My primary issues is that the ferret index needs to be optimized before it performs well. Since optimization locks the index from reading, and can take 30 seconds on my index, this severely limits how often I can update my index. I am working on a solution by running two ferret servers, but this is requiring extensively modifying the aaf plug-in. I have investigated Sphynx, but I don''t think that it solves my problem of large amounts of constant updates. -- Alex Neth Liivid Inc / cribQ www.liivid.com / www.cribq.com alex at liivid.com +1 206 499 4995 +86 13761577188 On Jan 25, 2008, at 1:53 PM, ferret-talk-request at rubyforge.org wrote:> Message: 5 > Date: Fri, 25 Jan 2008 15:02:50 +0000 > From: John Leach <john at johnleach.co.uk> > Subject: [Ferret-talk] ferret success stories? > To: ferret-talk at rubyforge.org > Message-ID: <1201273370.18749.8.camel at dogen.thepride.> > Content-Type: text/plain > > Hi all, > > there was a recent thread[1] on rails-deploy about Ferret in which > a lot > of people complained of problems using it in production. > > I''ve been using Ferret (with DRb) for many months now with no serious > issues. I''m assuming the posters know what they''re doing so I''m > guessing they''re just using Ferret in higher-scale environments > than me. > > I spoke to someone in person yesterday who claimed that Ferret over > DRb > couldn''t keep up with their use rate and had been investigating > replicating the ferret database between two machines. > > With all these bad experiences, I''d like to hear about some good > experiences. Anyone care to comment? Anyone using it under huge > load? > Care to provide some numbers and some notes about how you''ve made it > work? > > John. > > [1] http://groups.google.com/group/rubyonrails-deployment/ > browse_thread/thread/980fe7cb20cb97dd/bc798b52f439020c > > -- > http://www.brightbox.co.uk - UK Ruby on Rails hosting >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/ferret-talk/attachments/20080125/732e332c/attachment.html
Hi Alex, sorry about not responding to that mail, been pretty busy with other stuff at that time. I just dugg it out of my mail archive now and yes, I''d like to have a look at your locking code to prevent parallel index rebuilds :-) Regarding your optimization troubles - I''m afraid I don''t know what''s going wrong there with your index being slow when you don''t optimize it (unless you''re sorting your search results by something else than relevancy - that is indeed known to be slow when the index isn''t optimized). Did you try to tweak some of Ferret''s more obscure indexing parameters like :merge_factor (lowering it (the Ferret shortcut suggests a value of 2 or 3 for better search performance) will let Ferret merge segments more frequently so on average there are less files in the index which should improve search performance)? Cheers, Jens On Fri, Jan 25, 2008 at 04:00:33PM -0800, Alex Neth wrote:> I am successfully using ferret with a few caveats, none of which to > my knowledge would be solved by using another solution. > > I believe many corruption problems are associated with a bug in aaf > that causes multiple indexes to be built at the same time in the same > place. I''m not sure if that''s been addressed as I didn''t see any > response after reporting it months ago. > > My primary issues is that the ferret index needs to be optimized > before it performs well. Since optimization locks the index from > reading, and can take 30 seconds on my index, this severely limits > how often I can update my index. I am working on a solution by > running two ferret servers, but this is requiring extensively > modifying the aaf plug-in. > > I have investigated Sphynx, but I don''t think that it solves my > problem of large amounts of constant updates. > > -- > Alex Neth > Liivid Inc / cribQ > www.liivid.com / www.cribq.com > alex at liivid.com > +1 206 499 4995 > +86 13761577188 > > > > > On Jan 25, 2008, at 1:53 PM, ferret-talk-request at rubyforge.org wrote: > > >Message: 5 > >Date: Fri, 25 Jan 2008 15:02:50 +0000 > >From: John Leach <john at johnleach.co.uk> > >Subject: [Ferret-talk] ferret success stories? > >To: ferret-talk at rubyforge.org > >Message-ID: <1201273370.18749.8.camel at dogen.thepride.> > >Content-Type: text/plain > > > >Hi all, > > > >there was a recent thread[1] on rails-deploy about Ferret in which > >a lot > >of people complained of problems using it in production. > > > >I''ve been using Ferret (with DRb) for many months now with no serious > >issues. I''m assuming the posters know what they''re doing so I''m > >guessing they''re just using Ferret in higher-scale environments > >than me. > > > >I spoke to someone in person yesterday who claimed that Ferret over > >DRb > >couldn''t keep up with their use rate and had been investigating > >replicating the ferret database between two machines. > > > >With all these bad experiences, I''d like to hear about some good > >experiences. Anyone care to comment? Anyone using it under huge > >load? > >Care to provide some numbers and some notes about how you''ve made it > >work? > > > >John. > > > >[1] http://groups.google.com/group/rubyonrails-deployment/ > >browse_thread/thread/980fe7cb20cb97dd/bc798b52f439020c > > > >-- > >http://www.brightbox.co.uk - UK Ruby on Rails hosting > > >> _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk-- Jens Kr?mer http://www.jkraemer.net/ - Blog http://www.omdb.org/ - The new free film database