I''m having a strange problem accessing a 1.7GB Ferret index from within Mongrel (1.0.1) on Linux. On OSX a Ferret search through Rails takes a fraction of a second. From the command line, bypassing Mongrel, the search takes about the same amount of time. On Fedora Core 4 a Ferret search from the command line takes a fraction of a second, but the same search through Mongrel never returns. The mongrel just spins and using 50% or so of the CPU. Has anyone seen anything like this? Is there some kind of limit on the size of a file that Mongrel allows Rails to access in Linux? Help is greatly appreciated. I had no problem on Linux searching a Ferret index of about 1.5GB. Erik
On Sun, 15 Apr 2007 16:04:06 -0400 Erik Morton <eimorton at gmail.com> wrote:> I''m having a strange problem accessing a 1.7GB Ferret index from > within Mongrel (1.0.1) on Linux. On OSX a Ferret search through Rails > takes a fraction of a second. From the command line, bypassing > Mongrel, the search takes about the same amount of time. On Fedora > Core 4 a Ferret search from the command line takes a fraction of a > second, but the same search through Mongrel never returns. The > mongrel just spins and using 50% or so of the CPU. > > Has anyone seen anything like this? Is there some kind of limit on > the size of a file that Mongrel allows Rails to access in Linux? Help > is greatly appreciated. I had no problem on Linux searching a Ferret > index of about 1.5GB.No, mongrel doesn''t set any limits, and I don''t think it could set those kinds of OS limits without more extensive Ruby support. It''s also odd that you have no problems on OSX but do have them on Linux. When it comes to file IO problems it''s usually the other way around. There''s a couple things you can do to get to the bottom of this. Since you have a reproducible test scenario, you can simply try your query, make the CPU go 50% and then attach to it with: strace -p PID This will print out the system calls being done by that ruby process and should give you and the ferret author or myself an idea of what to do next. You may also have to delve into using gdb, but that''s kind of complex. Hit me up if strace doesn''t help as there''s a way to attach to a ruby process via gdb and then force it to throw a ruby exception. At a minimum you could attach and do the command backtrace to see where in the C callstack it''s stuck. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/
Zed, Thank you very much for the reply. The solution, as you will see, is very humbling for me. I installed strace and it looks like the problem is that there is a recursive permission denied error in an infinite loop that is hanging Mongrel: open("/var/www/apps/search/current/config/../indexes/final/ segments_5", O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied) My fix is obviously to change the ownership of my index files from root to the correct user. The infinite loop seems like a Ferret issue. Do you agree? Thanks again Zed. Erik On Apr 15, 2007, at 7:06 PM, Zed A. Shaw wrote:> On Sun, 15 Apr 2007 16:04:06 -0400 > Erik Morton <eimorton at gmail.com> wrote: > >> I''m having a strange problem accessing a 1.7GB Ferret index from >> within Mongrel (1.0.1) on Linux. On OSX a Ferret search through Rails >> takes a fraction of a second. From the command line, bypassing >> Mongrel, the search takes about the same amount of time. On Fedora >> Core 4 a Ferret search from the command line takes a fraction of a >> second, but the same search through Mongrel never returns. The >> mongrel just spins and using 50% or so of the CPU. >> >> Has anyone seen anything like this? Is there some kind of limit on >> the size of a file that Mongrel allows Rails to access in Linux? Help >> is greatly appreciated. I had no problem on Linux searching a Ferret >> index of about 1.5GB. > > No, mongrel doesn''t set any limits, and I don''t think it could set > those kinds of OS limits without more extensive Ruby support. It''s > also odd that you have no problems on OSX but do have them on Linux. > When it comes to file IO problems it''s usually the other way around. > > There''s a couple things you can do to get to the bottom of this. > Since > you have a reproducible test scenario, you can simply try your query, > make the CPU go 50% and then attach to it with: > > strace -p PID > > This will print out the system calls being done by that ruby process > and should give you and the ferret author or myself an idea of what to > do next. > > You may also have to delve into using gdb, but that''s kind of complex. > Hit me up if strace doesn''t help as there''s a way to attach to a ruby > process via gdb and then force it to throw a ruby exception. At a > minimum you could attach and do the command backtrace to see where in > the C callstack it''s stuck. > > -- > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > http://www.zedshaw.com/ > http://www.awprofessional.com/title/0321483502 -- The Mongrel Book > http://mongrel.rubyforge.org/ > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On Sun, 15 Apr 2007 20:12:21 -0400 Erik Morton <eimorton at gmail.com> wrote:> Zed, > Thank you very much for the reply. The solution, as you will see, is > very humbling for me. I installed strace and it looks like the > problem is that there is a recursive permission denied error in an > infinite loop that is hanging Mongrel: > > open("/var/www/apps/search/current/config/../indexes/final/ > segments_5", O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied)Yeah, that''s bad. Probably lots of other ferret related errors like that.> My fix is obviously to change the ownership of my index files from > root to the correct user. > > The infinite loop seems like a Ferret issue. Do you agree?Yep, tell them about it. That''s basic IO operations. It means that the author is calling open and not checking for error conditions on the return. EVERY time someone has that habit they write code that''s easily hacked. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/
Erik Morton
2007-Apr-16 17:48 UTC
[Ferret-talk] Ferret bug when non-root user tries to read from an index owned by root?
I noticed an issue where a Mongrel would hang at 50% CPU utilization indefinitely while attempting to query a Ferret index. I discovered that the index directory was owned by root and Mongrel was running as a non-root user. I straced the Mongrel and result was an infinite loop of:>> open("/var/www/apps/search/current/config/../indexes/final/ >> segments_5", O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied)My fix was obviously to change the permissions of the directory. Is the infinite loop of permission denied errors a bug in Ferret? Thanks. Erik