Backgroundrb devel - Jul 2006 - BackgrounDRb not working properly with files ?

Hey there,

I''m kinda new to BackgrounDRb but I''m facing a big problem
which I
can''t seem to wrap my head around.

The short version of it is : code that works when tested inside RoR''s
console doesn''t work when executed by a BackgrounDRb worker.

So I''ve tried a lot of things, including checking that the
"load_rails" was true in the config, and I''m starting to
wonder if
this is not some kind of problem between BackgrounDRb and files or
BackgrounDRb and file_column (which I''m using here).


Any idea or help is greatly appreciated.

My models are basically :
- Clip has_many records
- Record belongs_to Clip with :path as the file_column field.

What I''m trying to do here is import a batch of files from a directory
that''s passed as the args to the worker and add them as a new records
to new clips.

Again, this code is working perfectly from inside the console, but not
when I activate this worker through the application (the worker does
start, show the progress bar and completes the action, the clips and
records are created but the files are not attached correctly to the
records.

My hinch is that the worker cannot understand that :path is a
file_column and that the file object (fl) is not passed correctly, but
I could be completely off-target.


Here''s my worker''s do_work :

def do_work(args)
    # This method is called in it''s own new thread when you
    # call new worker. args is set to :args

      #We define the variables used to monitor the progress
      #@progress is used by the controller for the progress bar,
      #file_number is used to calculate the progress
      @progress = 0
      file_number = 0

      #Checking if the arg is indeed a directory
      if File.directory?(args)

        path = args

        #we get number_of_files to calculate the progress
        number_of_files = Dir.entries(path).size

        #Let''s get into that dir and look inside
        Dir.open(path) do |dir|
          dir.each do |entry|
            #We ignore ".", ".." (included in ruby dir
listings)
            #we also ignore sub-directories and hidden files
            unless  ["..", "."].include?(entry) or
File.directory?(entry) or entry.match(''^\.'')
              puts "treating file : "+entry

              #First we create a new clip with the filename
              cl = Clip.new(:title => entry)

              #We get a File object out of the path
              fl = File.open(path+entry)

              #We create a new record. path is the file_column defined
in the model.
              rec = cl.records.new(:path => fl)

              #... and we save everything.
              rec.save
              cl.save
            end
              file_number += 1
              @progress = file_number * 100 / number_of_files
          end
        end

      else
        puts "Gimme a directory !"
      end
  end


Any help greatly apreciated.

Tim

Hi!


On Jul 19, 2006, at 6:49 AM, Tim Anglade wrote:
> Hey there,
>
> I''m kinda new to BackgrounDRb but I''m facing a big
problem which I
> can''t seem to wrap my head around.
>
> Here''s my worker''s do_work :
>
> def do_work(args)
>     # This method is called in it''s own new thread when you
>     # call new worker. args is set to :args
>
>       #We define the variables used to monitor the progress
>       #@progress is used by the controller for the progress bar,
>       #file_number is used to calculate the progress
>       @progress = 0
>       file_number = 0
>
>       #Checking if the arg is indeed a directory
>       if File.directory?(args)
>
>         path = args
>
>         #we get number_of_files to calculate the progress
>         number_of_files = Dir.entries(path).size
>
>         #Let''s get into that dir and look inside
>         Dir.open(path) do |dir|
>           dir.each do |entry|
>             #We ignore ".", ".." (included in ruby dir
listings)
>             #we also ignore sub-directories and hidden files
>             unless  ["..", "."].include?(entry) or
> File.directory?(entry) or entry.match(''^\.'')
>               puts "treating file : "+entry
>
>               #First we create a new clip with the filename
>               cl = Clip.new(:title => entry)
>
>               #We get a File object out of the path
>               fl = File.open(path+entry)
>
>               #We create a new record. path is the file_column defined
> in the model.
>               rec = cl.records.new(:path => fl)
>
>               #... and we save everything.
>               rec.save
>               cl.save
>             end
>               file_number += 1
>               @progress = file_number * 100 / number_of_files
>           end
>         end
>
>       else
>         puts "Gimme a directory !"
>       end
>   end
>
>
> Any help greatly apreciated.
>
> Tim
Hey Tim-

	The first thing I would think of is that you are not passing an  
absolute path to this worker. Can you verify? What is the exact path  
you pass in to kick this worker off?  There should not be an issue  
with using any of the code you have in a worker, it should work fine.  
This line:
> if File.directory?(args)

Makes me think that the rest of your code is just getting skipped  
because the directory may not exist? Can you show me the call you use  
in rails to kick this off? Also I would put some @logger.debug  
statements inside your worker there so you can see which parts of the  
code actually get run when you kick it off.

Thanks
-Ezra

Hey there, thanks for the quick response !

On 7/19/06, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
>
> Hey Tim-
>
>         The first thing I would think of is that you are not passing an
> absolute path to this worker. Can you verify? What is the exact path
> you pass in to kick this worker off?  There should not be an issue
> with using any of the code you have in a worker, it should work fine.
> This line:
>
> > if File.directory?(args)
>
>
> Makes me think that the rest of your code is just getting skipped
> because the directory may not exist? Can you show me the call you use
> in rails to kick this off? Also I would put some @logger.debug
> statements inside your worker there so you can see which parts of the
> code actually get run when you kick it off.
>
> Thanks
> -Ezra
>
Thanks for the input, alas the code is not getting skipped, as I can
see debugging lines (I''ve got some ''puts'' thrown
around), and the new
clips and records are being created.

Also, there is no error thrown anywhere, and the problem obviously
comes from this line:
rec = cl.records.new(:path => fl)

which obviously does not work as intended _and_  still does not throw
an error... At first I thought (logically) that the error was on
file_column''s side, but at the same time, this line does work as
intended in the console, so the origin of the problem does seem to
come from BackgrounDRb.
Is there any reason why files would not be passed correctly from a
worker to file_column ?

I''m thinking this is some kind of linking error between the
backgroundrb daemon and the rails models. BackgrounDRb might very well
think that :path is a standard field and throw fl at him, while if use
the same line within Rails, it will trigger file_column functions to
assign the field correctly.


Thanks again and Cheers,
Tim

Yup, that did the trick.

Nice one, I obviously wouldn''t have been able to catch that by
myself...
I''ll tune it to load only file_column.


Many, many thanks !

On 7/19/06, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
> Hi~
>
> On Jul 19, 2006, at 8:43 AM, Tim Anglade wrote:
>
> > Hey there, thanks for the quick response !
> >
> > On 7/19/06, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> >
> >>
> >> Hey Tim-
> >>
> >>         The first thing I would think of is that you are not
> >> passing an
> >> absolute path to this worker. Can you verify? What is the exact
path
> >> you pass in to kick this worker off?  There should not be an issue
> >> with using any of the code you have in a worker, it should work
fine.
> >> This line:
> >>
> >> > if File.directory?(args)
> >>
> >>
> >> Makes me think that the rest of your code is just getting skipped
> >> because the directory may not exist? Can you show me the call you
use
> >> in rails to kick this off? Also I would put some @logger.debug
> >> statements inside your worker there so you can see which parts of
the
> >> code actually get run when you kick it off.
> >>
> >> Thanks
> >> -Ezra
> >>
> > Thanks for the input, alas the code is not getting skipped, as I can
> > see debugging lines (I''ve got some ''puts''
thrown around), and the new
> > clips and records are being created.
> >
> > Also, there is no error thrown anywhere, and the problem obviously
> > comes from this line:
> > rec = cl.records.new(:path => fl)
> >
> > which obviously does not work as intended _and_  still does not throw
> > an error... At first I thought (logically) that the error was on
> > file_column''s side, but at the same time, this line does work
as
> > intended in the console, so the origin of the problem does seem to
> > come from BackgrounDRb.
> > Is there any reason why files would not be passed correctly from a
> > worker to file_column ?
> >
> > I''m thinking this is some kind of linking error between the
> > backgroundrb daemon and the rails models. BackgrounDRb might very well
> > think that :path is a standard field and throw fl at him, while if use
> > the same line within Rails, it will trigger file_column functions to
> > assign the field correctly.
> >
> >
> > Thanks again and Cheers,
> > Tim
>
>
> Tim-
>
>         Ok maybe there is something else going on. BackgrounDRb by defalt
> even with the load_rails set to true doers not actually load all of
> rails. It just loads the database connection via boot.rb and all your
> model files. So what I think is happening is that the file_column
> plugin is not being loaded : / So lets try this... Open up the script/
> backgroundrb/start script and add the follwoing line right after the
> require boot.rb line:
>
> require File.dirname(__FILE__) + "/../../config/environment.rb"
>
>
>         That wilkl make sure all of rails is loaded including your plugins.
> Now you need to realize that this will also make the backgroundrb
> process take up about 22Mb of ram since it is loading all of rails
> now. This isn''t too big of a problem its just a tad inefficient.
If
> this causes memory problems for you then you might consider writing a
> few lines to require in just the file column plugin instead of all of
> rails. But I think if you do this then you will get your worker to
> function correctly.
>
>
> Thanks-
> -Ezra
>
>