Rails - Feb 2006 - Memory Issue while looping through CSV file

I have a site on a shared host at TextDrive. I have an import process
that imports a CSV file into the database using a ruby script executed
using script/runner. TextDrive has a memory limit which I am hitting
and the process gets killed. I get through to about 400 records before
it dies. I guess each time I instantate a new object for each record I
am using more memory but thought I was using reusing the same memory
allocated. Anyway, I''m a little new with Ruby and Rails so if you can
tell me where I''m going wrong that would be much appreciated. I have
included the code below. Thanks!

csv_file = CSV.open(CSV_PATH, "r")

csv_file.each do | pattern_temp |

  unless pattern_temp[0].nil?

		pattern_record = {
				:code => pattern_temp[0].delete("\""),
				:designer => pattern_temp[1],
				:name => pattern_temp[2],
				:designs => pattern_temp[3],
				:price => pattern_temp[4].delete("$"),
				:set => pattern_temp[5].delete("\"\r\n"),
				:set_price => pattern_temp[6].delete("$"),
				:themes => pattern_temp[7],
				:bundle_with_code => pattern_temp[8],
		                :matrix => pattern_temp[9],
				:description => pattern_temp[10],
				:active => pattern_temp[11],
				:panel => pattern_temp[12]
		}

    puts "Processing #{pattern_record[:code]}..."
    pattern = Pattern.find_by_pattern_code(pattern_record[:code]) || Pattern.new

    if pattern_record[:active] == "Yes"
		  add_pattern(pattern, pattern_record) # function
		elsif pattern_record[:active] == "No" and !pattern.new_record?
      puts "Removing #{pattern_record[:code]}..."
      Pattern.destroy(pattern.id)
    end
  end

end

Does anybody have any ideas on this below. Can I provide any more
information that would help someone?

Thanks in advance,
Nicholas

---------- Forwarded message ----------
Date: Feb 6, 2006 12:14 PM
Subject: Memory Issue while looping through CSV file
To: rails@lists.rubyonrails.org


I have a site on a shared host at TextDrive. I have an import process
that imports a CSV file into the database using a ruby script executed
using script/runner. TextDrive has a memory limit which I am hitting
and the process gets killed. I get through to about 400 records before
it dies. I guess each time I instantate a new object for each record I
am using more memory but thought I was using reusing the same memory
allocated. Anyway, I''m a little new with Ruby and Rails so if you can
tell me where I''m going wrong that would be much appreciated. I have
included the code below. Thanks!

csv_file = CSV.open(CSV_PATH, "r")

csv_file.each do | pattern_temp |

  unless pattern_temp[0].nil?

                pattern_record = {
                                :code =>
pattern_temp[0].delete("\""),
                                :designer => pattern_temp[1],
                                :name => pattern_temp[2],
                                :designs => pattern_temp[3],
                                :price =>
pattern_temp[4].delete("$"),
                                :set =>
pattern_temp[5].delete("\"\r\n"),
                                :set_price =>
pattern_temp[6].delete("$"),
                                :themes => pattern_temp[7],
                                :bundle_with_code => pattern_temp[8],
                                :matrix => pattern_temp[9],
                                :description => pattern_temp[10],
                                :active => pattern_temp[11],
                                :panel => pattern_temp[12]
                }

    puts "Processing #{pattern_record[:code]}..."
    pattern = Pattern.find_by_pattern_code(pattern_record[:code]) || Pattern.new

    if pattern_record[:active] == "Yes"
                  add_pattern(pattern, pattern_record) # function
                elsif pattern_record[:active] == "No" and
!pattern.new_record?
      puts "Removing #{pattern_record[:code]}..."
      Pattern.destroy(pattern.id)
    end
  end

end

Not a solution but a related plea for help from a new boy...

Can someone tell me please, if I have a csv file Foo stored in
public/csvfiles, how do I open this, read its rows and then present them
on an browser screen...
1: what "requires" do I need and where does it go?  [require
"csv" in
the CLASS file?]
2: what code opens and reads the file row by row, and where does this
go? [do I read the file row by row into an array within the controller?]
3: how do I then display the data on screen?

Sample code would be most welcomed, plus, if possible, a reference to
any further related reading on the web.

Many thanks.

On Feb 10, 2006, at 2:31 PM, Les Bowker wrote:
>
> Can someone tell me please, if I have a csv file Foo stored in
> public/csvfiles, how do I open this, read its rows and then present  
> them
> on an browser screen...
> 1: what "requires" do I need and where does it go?  [require
"csv" in
> the CLASS file?]
> 2: what code opens and reads the file row by row, and where does this
> go? [do I read the file row by row into an array within the  
> controller?]
> 3: how do I then display the data on screen?
>
> Sample code would be most welcomed, plus, if possible, a reference to
> any further related reading on the web.
Here''s a script I use to import from CSV that came from Excel  
spreadsheets that came from MS Access.

http://www.rafb.net/paste/results/C1r7hF26.html

>From the command-line:
gem install fastercsv

At the bottom of config/environment.rb, add:
require_gem ''fastercsv''


Let''s say you have a controller action called display_data.
Further, let''s say that you already know which filename you want. You
could also pass it as a parameter, etc, etc.


def display_data
  # @csv_rows will be an Array of rows from the CSV file.  The rows
are Arrays of columns.
  # Further, let''s assume that the first row contains column names.

  # Read the CSV file from disk, and load it up into @csv_rows.
  @csv_rows = FasterCSV.read("public/csvfiles/example.csv")
  @columns = @csv_rows.shift # Push the top row out into @columns
end

Now, in display_data.rhtml:

<table>
<thead>
  <% @columns.each do |col| -%>
  <th><%= col.titleize -%></th>
  <% end -%>
</thead>
<tbody>
  <%= render :partial => ''csv_row'', :collection =>
@csv_rows %>
</tbody>
</table>

In "_csv_row.rhtml":
<tr>
<% csv_row.each do |entry| -%>
  <td><%= h entry %></td>
<% end -%>
</tr>

There you go. The output will be an html table matching your CSV file.
The only non-trivial part is dealing with the fact that you''ve got
more than one collection of things to iterate over. (First the column
headings, then the rows and ''cells'')

--Wilson.

On 2/10/06, Les Bowker <les@microdimensional.co.uk>
wrote:
> Not a solution but a related plea for help from a new boy...
>
> Can someone tell me please, if I have a csv file Foo stored in
> public/csvfiles, how do I open this, read its rows and then present them
> on an browser screen...
> 1: what "requires" do I need and where does it go?  [require
"csv" in
> the CLASS file?]
> 2: what code opens and reads the file row by row, and where does this
> go? [do I read the file row by row into an array within the controller?]
> 3: how do I then display the data on screen?
>
> Sample code would be most welcomed, plus, if possible, a reference to
> any further related reading on the web.
>
> Many thanks.
>
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>