Rails - Apr 2006 - Transactions and migrations (lots of records)???

Hi all -

I know this must be really easy to do, but I can''t for the life of me 
figure it out.

I have about 100,000 rows I need to migrate over into rails using a 
migration.

Right now, I select all of them out of the old database and do a series 
of:

...
new.col = old.col
...
new.save
...

Problem is each one is it''s own transaction.  And it''s slow. 
I''d like to
group them into sets of say 50-100 to speed things up a bit.

But the only example regarding transactions I can find are long the lines 
of:

   transaction do
     david.withdrawal(100)
     mary.deposit(100)
   end

Which doesn''t really help me.

What I want to do is start a transaction before I start looping then watch 
a counter and every 50th time, commit, then open another one.  Then have a 
final commit.

I''ll deal with errors elsewhere... right now I just need it to go
faster.

I have a feeling I want begin_db_transaction and commit_db_transaction, 
but I can''t figure out how to use them in the way I want.

Help! :-)

-philip

Here is a thought of how you might be able to do it, but I haven''t
tested it.  This is just pseudocode, but someone might be able to elaborate
further.

maxcount = Order.count
(1..maxcount).step(100) { |i|
	transaction do
		orders = Order.find(:all, :limit => 100, :offset => maxcount)
		## Then do processing here

	end
}

-----Original Message-----
From: rails-bounces@lists.rubyonrails.org
[mailto:rails-bounces@lists.rubyonrails.org]On Behalf Of Philip
Hallstrom
Sent: Tuesday, April 25, 2006 7:23 PM
To: rails@lists.rubyonrails.org
Subject: [Rails] Transactions and migrations (lots of records)???


Hi all -

I know this must be really easy to do, but I can''t for the life of me 
figure it out.

I have about 100,000 rows I need to migrate over into rails using a 
migration.

Right now, I select all of them out of the old database and do a series 
of:

...
new.col = old.col
...
new.save
...

Problem is each one is it''s own transaction.  And it''s slow. 
I''d like to
group them into sets of say 50-100 to speed things up a bit.

But the only example regarding transactions I can find are long the lines 
of:

   transaction do
     david.withdrawal(100)
     mary.deposit(100)
   end

Which doesn''t really help me.

What I want to do is start a transaction before I start looping then watch 
a counter and every 50th time, commit, then open another one.  Then have a 
final commit.

I''ll deal with errors elsewhere... right now I just need it to go
faster.

I have a feeling I want begin_db_transaction and commit_db_transaction, 
but I can''t figure out how to use them in the way I want.

Help! :-)

-philip
_______________________________________________
Rails mailing list
Rails@lists.rubyonrails.org
http://lists.rubyonrails.org/mailman/listinfo/rails
Please see the following link for the BlueCross BlueShield of Tennessee E-mail
disclaimer:  http://www.bcbst.com/email_disclaimer.shtm

Oops, that code should have been...

maxcount = Order.count
(0..maxcount).step(100) { |i|
	transaction do
		orders = Order.find(:all, :limit => 100, :offset => i)
		## Then do processing here

	end
}

-----Original Message-----
From: Campano, David 
Sent: Wednesday, April 26, 2006 10:44 AM
To: ''rails@lists.rubyonrails.org''
Subject: RE: [Rails] Transactions and migrations (lots of records)???


Here is a thought of how you might be able to do it, but I haven''t
tested it.  This is just pseudocode, but someone might be able to elaborate
further.

maxcount = Order.count
(1..maxcount).step(100) { |i|
	transaction do
		orders = Order.find(:all, :limit => 100, :offset => maxcount)
		## Then do processing here

	end
}

-----Original Message-----
From: rails-bounces@lists.rubyonrails.org
[mailto:rails-bounces@lists.rubyonrails.org]On Behalf Of Philip
Hallstrom
Sent: Tuesday, April 25, 2006 7:23 PM
To: rails@lists.rubyonrails.org
Subject: [Rails] Transactions and migrations (lots of records)???


Hi all -

I know this must be really easy to do, but I can''t for the life of me 
figure it out.

I have about 100,000 rows I need to migrate over into rails using a 
migration.

Right now, I select all of them out of the old database and do a series 
of:

...
new.col = old.col
...
new.save
...

Problem is each one is it''s own transaction.  And it''s slow. 
I''d like to
group them into sets of say 50-100 to speed things up a bit.

But the only example regarding transactions I can find are long the lines 
of:

   transaction do
     david.withdrawal(100)
     mary.deposit(100)
   end

Which doesn''t really help me.

What I want to do is start a transaction before I start looping then watch 
a counter and every 50th time, commit, then open another one.  Then have a 
final commit.

I''ll deal with errors elsewhere... right now I just need it to go
faster.

I have a feeling I want begin_db_transaction and commit_db_transaction, 
but I can''t figure out how to use them in the way I want.

Help! :-)

-philip
_______________________________________________
Rails mailing list
Rails@lists.rubyonrails.org
http://lists.rubyonrails.org/mailman/listinfo/rails
Please see the following link for the BlueCross BlueShield of Tennessee E-mail
disclaimer:  http://www.bcbst.com/email_disclaimer.shtm