Rails - Oct 2005 - (long!) An "Enterprise" Rails story and a help with nested transaction support

Hello, everyone, 
A tiny bit of background first:  I''m managing one of those so-called
"Enterprise" projects: N * 1,000 dedicated users spread over multiple
states (probably 80+ locations at deployment), 24/7 uptime required,
$multi-million project budget (see also: training), multiple platforms
for both servers and clients, etc.

We started out presuming we were going to be forced to use a Java stack
(primarily due to 3rd parties we are partnered with).  We implemented a
vertical slice of the app (db tables, Hibernate mappings, pojos/beans,
Struts config, JSPs, AJAX stuff, unit tests) for one group of models.
Only, actually we couldn''t get that far in a reasonable amount of time:
we began to get bogged down in "shotgun code smell" (aka "the
domino
effect") with all the inter-layer coupling, and AJAX was damned-near
impossible to get working with the Struts/JSP layer we were using.  That
and we still hadn''t had the time yet to make a decision on which
testing
tools to use (JUnit, Cactus, HTTPUnit, JHTTPsomethingorother, etc.).

I came into this project a big Rails fan, but, given the external
requirements, I understood that avenue had been foreclosed.

As we began to run the boat aground on the big Java stack our developers
(who were of their own accord looking at how Rails does the AJAX views
for help in figuring out how to do such things in Java) started asking
"Why don''t we just do this stuff in Ruby?"  These are Java
developers,
hired for their Java skills, asking why not move to Ruby on Rails?

At about that time, some of the external constraints started to loosen.
It may not be *necessary* to deploy the project in Java, after all.
But, we''d need a good case for switching architectures.

Shortly thereafter our boss (The Director) said, "How would you guys
feel about taking a couple of weeks to implement some of this in Ruby to
see how well it would work?"  There wasn''t a hint of a
"no" at the
table.

Our technical lead hacked a quick vertical version of one model class in
Rails that had already been implemented in Java.  He found an 8:1
reduction in code size going from Java to Rails.

About a week ago (just a couple of days after our boss gave us the
"go"
to try a Rails version of the module we''d already written in Java), we
started Rails development, using 3 developers (2 of which had only a day
or two''s worth of knowledge of Ruby), one very part-time DBA, and me
fielding the occasional question nobody else wants to deal with (see
below...).

After a week we have ~85% of the Rails development done for the module,
including a slick AJAX interface (which we never managed in Java), and a
subset of our needed unit/functional tests (see below...).  Best
estimate for equivalent effort we put in on the Java side (because our
data modelling can be reused, clearly) is about 6 weeks.

Here are some figures from today''s most recent SVN pulls:

Java version:

10361 lines of Java code
1143  lines of JSP
8082  lines of XML
1267  lines of build configuration
-----------------------------------------------------------
20853 TOTAL lines of stuff


Rails version:

494   lines of code (386 "LOC" per rake stats)
254   lines of RHTML
75    lines of configuration (includes comments in routes.rb)
0     lines of build configuration
-----------------------------------------------------------
823   TOTAL lines of stuff


Code reduction alone is right around 20:1, and overall lines produced
(config, templates, code) the ratio is just over 25:1.  I''m using the
larger (494) number for Rails LOC, because I counted the Java LOC by a
simple ''wc -l'' and so doesn''t get rid of comments,
whitespace, whatever.
We also are using 2 DB platforms, hence the ability to get a full 75
lines of config for the Rails app.

So, bottom line from where we sit:  the guys complaining that reported
10:1 savings over Java was bullsh*t were right on target:  10:1 is way
too low from what we''re seeing.

I''m sure there will be plenty of people saying "Well, you should
have
used {JSF, Shale, Tapestry, Spring, Echo{1,2}, Castor, Cayenne, etc."
(fwiw, we were using draft EJB3.0 w/ annotations), but I''ve seen zero
evidence that any combination of the most cutting edge Java components
will get us down to a functional application using <= 823 lines of total
stuff -- really, not even w/in a factor of 5 of that from all I''ve read
lately.

Note that I haven''t even mentioned to this point the app server
(JBoss),
which includes a few lines of XML complexity of its own:

& Mon Oct 10 10:49:57 jrbradle@rick
~/svn/phoenix/srv$ find . -type f -name ''*.xml'' | xargs wc -l
| grep total
44472 total

That and add in the sheer heft that is running JBoss up under
CruiseControl and our build server has to be downright stoked with RAM.

  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

Enough.  As Arlo Guthrie would say, "But that''s not what I came
here to
talk to you about." I came to talk to you about nested transactions in
ActiveRecord.  More background?  Glad you asked.

Our project is building out a healthcare application, one which will be
open-sourced within a consortium of mental health care providers around
the US.  The reason the consortium exists (and that it''s not just an
open source free-for-all) is that we, a community-based mental
healthcare nonprofit, want to promote research in the field (see also
industry buzzword "evidence-based treatment).  We will give away the
system, even host it if you need it, so long as you provide your patient
data (anonymized!) for use in a research database to be used by
researchers in academia and elsewhere to develop new mental health care
protocols.  The protocols feed back into the system, lather, rinse,
repeat.

Great fun, until you run into HIPAA, Sarbanes-Oxley, and the restrictive
auditing requirements for drug trials.  For anything related to patient
data we not only have to log who did what to the data, but also who saw
the data, and we typically have to preserve the data in the database.
This ultimately requires (among other things) a database-level no
deletions policy.  In our data model one of the ways this is implemented
is by constraints (we''re currently targeting both Oracle and PostgreSQL
though others will be supported later) which say for certain tables
"sorry, you can''t delete that record!".  Since access to the
database
can come from outside the Rails application (there is a policy document
which enumerates what we will be supporting), app-level constraints do
not suffice.

Which brings me to the crux of the matter.  To actually test the
functionality we''ve implemented, using fixtures, we discovered
something
interesting (and, in retrospect) obvious.  When we specify fixtures on a
protected table, the fixtures will load in properly, we use
transactional tests so that our individual unit tests run in isolation,
and then at the end of the test the fixtures attempt to unload from the
database but can''t -- because the data can''t be deleted from
the table.
Further, the next test case which uses those fixtures is in a bit of a
bind because the fixture records in question are already in the table.

This is one of those cases, to me, where we don''t have a choice but to
use the constraints in the database.  And it''s important for us to be
able to test that deletions can''t be performed -- otherwise how will we
know if the application behaves properly under the real production
constraints?  It really looks like we need a further layer of
transactions (a nested transaction) around the overall test case.
Start a transaction, load the fixtures, wrap each unit test in a
transaction (like we''re currently doing) with a rollback, and at the
end
roll back the big transaction.

So I did some legwork (on the 0.13.1 version, not yet on the svn trunk,
though I spot-checked some things in the Trac browser and didn''t see
any
eyebrow-raisers).  I see that Transaction::Simple and ActiveRecord::Base
handle the core transactional support, ultimately delegating to the
actual connection adapters to do a
"BEGIN"/"ROLLBACK"/"COMMIT".  Support
for nested transactions would introduce the notion of savepoints, and
rolling back thereto.

Note that right now the only thing I''m aware of that''s holding
us up
from saying "We did in Ruby on Rails in 8 days, in 1/20th the amount of
code, what took us more than 6 weeks to do in Java" is the inability to
test our actual database setup, and that smells like it''s just limited
by the lack of nested transactions.  Everything else we we''ve tried
we''ve had no problem getting to work.

My fears about adding in nested transaction support entirely by myself:

 1- I''d be tempted to do "just what works", which would
probably be
 bastardizing the pretty AR way into just hammering nested transactions
 in for OCI and postgres to get our tests to work.  That would break on
 the next release, and wouldn''t be much help to anyone else.

 2- Somebody else is probably working on this already.  In which case
 I''d be working at cross purposes and our version would have to be
 dumped on the next release anyway.

 3- I''d probably invent an unwieldy abstraction that wouldn''t
work well
 for the rest of AR -- and I probably wouldn''t be able to test it well
 for the various adapters in use out there.  I don''t really have a 100%
 handle on the Test::Unit changes that AR does, nor quite what
 everything in the fixture flow is doing -- I think I could get there
 quickly though.

 4- SVN trunk is a moving target, that''s for sure.

Anyway, I''m looking for some advice on maybe the best way to do
implement this.

If someone''s working on this already, can I be of help?  If not, what
would be the best way to communicate savepoint information up and down
through AR to/from the various connection adapters.  What about adapters
which don''t support nested transactions?  On the testing side,
presumably there needs to be a class variable to turn things on/off.

Just looking for some guidance.

Thanks for any insights.  Oh, and see you at RubyConf!

Rick
-- 
 http://www.rickbradley.com    MUPRN: 67
                       |  fixes needed" Heh,
   random email haiku  |  it was on my todo list
                       |  for the past few months.

On Monday 10 October 2005 19:32, Rick Bradley wrote:
> Which brings me to the crux of the matter.  To actually test the
> functionality we''ve implemented, using fixtures, we discovered
> something interesting (and, in retrospect) obvious.  When we specify
> fixtures on a protected table, the fixtures will load in properly, we
> use
> transactional tests so that our individual unit tests run in
> isolation, and then at the end of the test the fixtures attempt to
> unload from the database but can''t -- because the data
can''t be
> deleted from the table. Further, the next test case which uses those
> fixtures is in a bit of a bind because the fixture records in
> question are already in the table.
[snip]
> Anyway, I''m looking for some advice on maybe the best way to do
> implement this.
Don''t! From what you wrote, I take it that you don''t need
nested
transactions for your application by itself. Rather, you have a problem 
testing your database interaction and nested transactions look like a 
way to solve it. Correct? I suggest that you go at the immediate 
problem: making your tests work.

I think the best approach would be to add a decorator facility to 
Test::Unit so that you can wrap actions around test suites. In 
particular, this would allow you to clean up the database in whatever 
way that is necessary after each suite has been run. While you''re at 
it, it would be useful to enable constraints only after fixture data 
has been loaded into the database. This would avoid having to order 
fixtures in a way that doesn''t violate constraints; something
that''s
not always even possible.

Michael

-- 
Michael Schuerig                              Life is just as deadly
mailto:michael-q5aiKMLteq4b1SvskN2V4Q@public.gmane.org                          
As it looks
http://www.schuerig.de/michael/          --Richard Thompson, Sibella

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 10, 2005, at 10:32 AM, Rick Bradley wrote:
> Enough.  As Arlo Guthrie would say, "But that''s not what I
came
> here to
> talk to you about." I came to talk to you about nested transactions in
> ActiveRecord.  More background?  Glad you asked.
Your case study is a real pleasure!

> Which brings me to the crux of the matter.  To actually test the
> functionality we''ve implemented, using fixtures, we discovered  
> something
> interesting (and, in retrospect) obvious.  When we specify fixtures  
> on a
> protected table, the fixtures will load in properly, we use
> transactional tests so that our individual unit tests run in  
> isolation,
> and then at the end of the test the fixtures attempt to unload from  
> the
> database but can''t -- because the data can''t be deleted
from the
> table.
> Further, the next test case which uses those fixtures is in a bit of a
> bind because the fixture records in question are already in the table.
Fixtures aren''t smart enough about its transactions: they
needn''t issue
deletes considering they''re rolling back anyway.  Ryan Davis recently
pressed a similar point regarding the order of deletes and inserts.

I''m fixing this up now as well as applying portions of tickets #1911
and
#2292 from Duane Johnson and Rick Olson, so your fixtures issues will be
resolved before the first 1.0 release candidate.

> This is one of those cases, to me, where we don''t have a choice
but to
> use the constraints in the database.  And it''s important for us to
be
> able to test that deletions can''t be performed -- otherwise how  
> will we
> know if the application behaves properly under the real production
> constraints?  It really looks like we need a further layer of
> transactions (a nested transaction) around the overall test case.
> Start a transaction, load the fixtures, wrap each unit test in a
> transaction (like we''re currently doing) with a rollback, and at  
> the end
> roll back the big transaction.
Fortunately, I think neither nested transactions nor savepoints are
required.  If you find that you need them later, I have some code that
emulates nested transactions with savepoints on PostgreSQL and MySQL.
I opted to leave it out of ActiveRecord because it needs declarations
such as :supports, :requires, :requires_new, etc. to intelligently  
decide
whether to flatten, nest, or deny transactions.

> If someone''s working on this already, can I be of help?  If not,
what
> would be the best way to communicate savepoint information up and down
> through AR to/from the various connection adapters.  What about  
> adapters
> which don''t support nested transactions?  On the testing side,
> presumably there needs to be a class variable to turn things on/off.
I keep a stack of savepoints in the adapter and perform the appropriate
SQL and push/pop for begin/commit/rollback.

> Thanks for any insights.  Oh, and see you at RubyConf!
Likewise.  Perhaps we can hammer out fixtures by then.

Best,
jeremy
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (Darwin)

iD8DBQFDSrPTAQHALep9HFYRApADAJwJL43o7BZNK0Jh9Ih615UPt1BAVACgtg7q
93WJEz/IggM86J48TgOPNFM=3X1D
-----END PGP SIGNATURE-----

On 10/10/05, Rick Bradley <rick-xSCPAUIMY+WN9aS15agKxg@public.gmane.org>
wrote:
<snip>
> This ultimately requires (among other things) a database-level no
> deletions policy.  In our data model one of the ways this is implemented
> is by constraints (we''re currently targeting both Oracle and
PostgreSQL
> though others will be supported later) which say for certain tables
> "sorry, you can''t delete that record!".  Since access to
the database
> can come from outside the Rails application (there is a policy document
> which enumerates what we will be supporting), app-level constraints do
> not suffice.
>
> Which brings me to the crux of the matter.  To actually test the
> functionality we''ve implemented, using fixtures, we discovered
something
> interesting (and, in retrospect) obvious.  When we specify fixtures on a
> protected table, the fixtures will load in properly, we use
> transactional tests so that our individual unit tests run in isolation,
> and then at the end of the test the fixtures attempt to unload from the
> database but can''t -- because the data can''t be deleted
from the table.
> Further, the next test case which uses those fixtures is in a bit of a
> bind because the fixture records in question are already in the table.
Rick, first off - Awesome Post!!!  Exciting real world numbers.

Secondly, what about dropping the constraint in the test teardown or
in a helper method.  You would need to re-apply the contrainst(s) in
setup or in test helper method

* Load fixtures
* Apply DB contraints
* Run Tests
* Remove DB contraints
* Purge test data

rinse and repeat.

Ken Barker

Rick Bradley wrote:
> Which brings me to the crux of the matter.  To actually test the
> functionality we''ve implemented, using fixtures, we discovered
something
> interesting (and, in retrospect) obvious.  When we specify fixtures on a
> protected table, the fixtures will load in properly, we use
> transactional tests so that our individual unit tests run in isolation,
> and then at the end of the test the fixtures attempt to unload from the
> database but can''t -- because the data can''t be deleted
from the table.
> Further, the next test case which uses those fixtures is in a bit of a
> bind because the fixture records in question are already in the table.
> If someone''s working on this already, can I be of help?  If not,
what
> would be the best way to communicate savepoint information up and down
> through AR to/from the various connection adapters.  What about adapters
> which don''t support nested transactions?  On the testing side,
> presumably there needs to be a class variable to turn things on/off.
If I understand correctly what you''re looking for is a way to make
fixture-based tests work correctly in the face of database constraints,
rather than specifically to implement nested transactions -- though
it''d
be nice to have that feature.

I''ve come across this problem on Oracle too, in the same way as you
have, but also on MySQL with foreign keys, where something based on
nested transactions wouldn''t be possible. You might have seen the brief
thread this morning on just this topic.

It seems that what''s needed is some more general way of dealing with
fixtures and database constraints: where nested transactions are
available, use that method, but something else otherwise.

I''m not sure what that something else might be; I suggested generated
scripts to disable and enable constraints, hooked in at the appropriate
moments to allow fixtures to load and unload, and the tests to run with
all the constraints enabled. There might be issues with database
privileges, but perhaps that is acceptable on a testing-only database.



Chris.

* Ken Barker (ken.barker-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) [051010
13:45]:
> Secondly, what about dropping the constraint in the test teardown or
> in a helper method.  You would need to re-apply the contrainst(s) in
> setup or in test helper method
> 
> * Load fixtures
> * Apply DB contraints
> * Run Tests
> * Remove DB contraints
> * Purge test data
Ken -- 
That''s an interesting suggestion.  I''d have to think about how
easy it
would be for us to factor our various types of constraints on our tables
to fit such a model.  We use a Ruby-based build tool which we wrote
which helps manage dependencies, including which tables to load in which
order.  It would introduce some complexity, but we already have a
headache in the testability arena.

I''m wondering if a Test::Unit::TestCase class-level (as in once per
test
case, not once per unit test) setup/teardown method would be of general
interest in order to get this sort of thing done?  

I guess my overall questions would be:  

 1 - Are there more uses for a once-per-testcase setup/teardown method
 or for nested transactions in testing? 
 
 and 

 2 - Can we easily split our database schemas into table defs and
 constraints?  How hard is it then to load and unload those constraints
 from within AR (given that the constraint scripts are per-DB and it
 seems a bit heavy-handed to say ''case ... when
"postgres"'' inside AR.
 Hm.  Perhaps that''s just a per-adapter bit of functionality.

Ken -- you raise an interesting question.

Rick
-- 
 http://www.rickbradley.com    MUPRN: 751
                       |  large supply of these
   random email haiku  |  CDR''s we''re passing our
                       |  savings on to you.

* Chris Andrews (chris-+s4wOdoq3VEdnm+yROfE0A@public.gmane.org) [051010
13:55]:
> I''ve come across this problem on Oracle too, in the same way as
you
> have, but also on MySQL with foreign keys, where something based on
> nested transactions wouldn''t be possible. You might have seen the
brief
> thread this morning on just this topic.
> 
> It seems that what''s needed is some more general way of dealing
with
> fixtures and database constraints: where nested transactions are
> available, use that method, but something else otherwise.
> 
> I''m not sure what that something else might be; I suggested
generated
> scripts to disable and enable constraints, hooked in at the appropriate
> moments to allow fixtures to load and unload, and the tests to run with
> all the constraints enabled. There might be issues with database
> privileges, but perhaps that is acceptable on a testing-only database.
It sounds like what you and Ken are both talking about is very similar
-- some way to do some work around the fixtures portion of a
Test::Unit::TestCase, where we could do our constraint shepherding at
the right time.   I can see how pure nesting wouldn''t necessarily work
for everyone (although nesting transactions may be useful in other
contexts), depending upon the DB backend.

If anyone else is already working along these lines I''d like to hear
their thoughts.

Rick
-- 
 http://www.rickbradley.com    MUPRN: 371
                       |  are excellent choice
   random email haiku  |  for them (I recommend
                       |  Front Page frequently).

Rick Bradley wrote:
> * Chris Andrews (chris-+s4wOdoq3VEdnm+yROfE0A@public.gmane.org) [051010
13:55]:
>>I''m not sure what that something else might be; I suggested
generated
>>scripts to disable and enable constraints, hooked in at the appropriate
>>moments to allow fixtures to load and unload, and the tests to run with
>>all the constraints enabled. There might be issues with database
>>privileges, but perhaps that is acceptable on a testing-only database.
> 
> 
> It sounds like what you and Ken are both talking about is very similar
> -- some way to do some work around the fixtures portion of a
> Test::Unit::TestCase, where we could do our constraint shepherding at
> the right time.   I can see how pure nesting wouldn''t necessarily
work
> for everyone (although nesting transactions may be useful in other
> contexts), depending upon the DB backend.
Yes, it seems we''re on the same track. I''m also wondering
about
automating the constraint drop and restore: it should be possible to
teach the AR adapter to enumerate all the foreign key constraints and
drop them, then restore them after loading the fixtures.

Certainly the Oracle data dictionary would allow this, and ''show create
table'' in pre-5.0 MySQL should too. Will have to investigate.


Chris.

On Oct 10, 2005, at 11:32 AM, Jeremy Kemper wrote:
> Fixtures aren''t smart enough about its transactions: they
needn''t
> issue
> deletes considering they''re rolling back anyway.  Ryan Davis
recently
> pressed a similar point regarding the order of deletes and inserts.
>
> I''m fixing this up now as well as applying portions of tickets  
> #1911 and
> #2292 from Duane Johnson and Rick Olson, so your fixtures issues  
> will be
> resolved before the first 1.0 release candidate.
Hurray!

You are my hero. :-)

-- 
-- Tom Mornini

* Jeremy Kemper (jeremy-w7CzD/W5Ocjk1uMJSBkQmQ@public.gmane.org) [051010
15:05]:
> Your case study is a real pleasure!
 
Glad I could entertain -- having access to Rails has been the real
pleasure for us.
> Fixtures aren''t smart enough about its transactions: they
needn''t issue
> deletes considering they''re rolling back anyway.  Ryan Davis
recently
> pressed a similar point regarding the order of deletes and inserts.
> 
> I''m fixing this up now as well as applying portions of tickets
#1911 and
> #2292 from Duane Johnson and Rick Olson, so your fixtures issues will be
> resolved before the first 1.0 release candidate.
I''ll keep an eye on the timeline and see when this stuff lands.  There
are some other things we like out of svn too, but we''re not using svn
Rails for wider development yet; if we need to move to it to get our
tests rolling we would.

I may well take someone else''s suggestion about using a decorator for
Test::Unit::Testcase in the meantime, but it will take some work on our
constraints to be usable.
> Fortunately, I think neither nested transactions nor savepoints are
> required.  If you find that you need them later, I have some code that
> emulates nested transactions with savepoints on PostgreSQL and MySQL.
> I opted to leave it out of ActiveRecord because it needs declarations
> such as :supports, :requires, :requires_new, etc. to intelligently
> decide whether to flatten, nest, or deny transactions.
I was afraid of the same as I started looking into the details of the
implementation (and hence my desire to post first, code later).  Thanks
for confirming my fears.
 
> Likewise.  Perhaps we can hammer out fixtures by then.
All in all this thread has had some of my best news for the day.

Thanks!

Rick
-- 
 http://www.rickbradley.com    MUPRN: 962
                       |  in advance of
   random email haiku  |  flights has increased from five
                       |  percent to percent.

On Oct 10, 2005, at 10:32 AM, Rick Bradley wrote:
> Code reduction alone is right around 20:1, and overall lines produced
> (config, templates, code) the ratio is just over 25:1.  I''m using
the
> larger (494) number for Rails LOC, because I counted the Java LOC by a
> simple ''wc -l'' and so doesn''t get rid of
comments, whitespace,
> whatever.
> We also are using 2 DB platforms, hence the ability to get a full 75
> lines of config for the Rails app.
>
> So, bottom line from where we sit:  the guys complaining that reported
> 10:1 savings over Java was bullsh*t were right on target:  10:1 is way
> too low from what we''re seeing.
This is great data!
> I''m sure there will be plenty of people saying "Well, you
should have
> used {JSF, Shale, Tapestry, Spring, Echo{1,2}, Castor, Cayenne, etc."
> (fwiw, we were using draft EJB3.0 w/ annotations), but I''ve seen
zero
> evidence that any combination of the most cutting edge Java components
> will get us down to a functional application using <= 823 lines of  
> total
> stuff -- really, not even w/in a factor of 5 of that from all I''ve
> read
> lately.
WebObjects is pretty terse (for Java), and I found a 3392 total lines  
o'' stuff (WebObjects) vs. 1866 total lines o'' stuff (Rails) in
even a
small app. Actual meat of the code difference was approximately 3:1  
excluding (r)html and wod files.

Various people predicted that rails would look worse on a larger  
project (hah!), but I''m glad to see that my wee little app was more  
like a best case for Java.

I''ll do a second WO analysis when I re-implement a larger WO app in  
rails toward the end of the year or early next year.

[1] http://deirdre.net/posts/2005/08/webobjects-to-ruby-on-rails/
[2] http://weblog.rubyonrails.com/archives/2005/08/23/comparing- 
webobjects-to-ruby-on-rails

-- 
_Deirdre                                             http://deirdre.net

It sounds like you need to 
1.  Back up the entire database
2.  Run your insertion tests.
3.  Delete the database.
4.  Restore the database from backup.

This is often needed in database testing, as testing changes the database, and
you want to
regression test from a constant database.

The obvious solution is to use database tools to backup and restore the
database.  I''d
consider using whole-disk tools like volcopy.  That way, you start out each time
with the
exact same bits in the exact same places on the disk.  This helps greatly if you
are doing
benchmarking (performance testing).  Given the size of your system, I think some
performance measurements will be mandatory.

Warren Seltzer