Re: Activerecord include queries - performance

Gregory Seidman
Thu, 03 Jan 2008 23:55:11 -0000

On Thu, Jan 03, 2008 at 11:50:46AM -0800,
bluescreen303-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
wrote:
> I noticed ActiveRecord uses some join queries to perform :include
> stuff.
> While this certainly fixes the N+1 problem, I''m wondering about
the
> amount of sparse data this generates.
[...]
> Now, I''m no database expert so I might be overlooking something.
> Also I haven''t really done any benchmarks.
> But like I said, it looks to me as if the current ActiveRecord
> behaviour is quite sparse for transfering/parsing the data as soon as
> you 3 or 4 tables, with long rows, and many "children".
> Am I overlooking something?
>
> Any thoughts on this matter would be nice.

You are unconsciously assuming that the cost of sending extra data is
usually greater than the cost of associating records outside the database,
not to mention multiple database optimize and execute cycles. A properly
indexed database is going to perform the association of records very
efficiently for you, since that''s one of the things for which RDBMS
software is designed and optimized. Note that YMMV with poorly written
RDBMSs; I''ve had MySQL''s query optimizer (not the execution
engine, the
query optimizer) completely hang when trying to deal with joins across a
lot of tables (18, in this case).

The real cost of sending lots of data is, of course, more than just
bandwidth. There is the cost of manifesting the result rows in the RDBMS
and serializing them, which can cause thrashing if the result set is really
big and have an even greater cost. There is the parsing cost for the database
driver to bring the received data into memory. There is the cost of hydrating
the models from the result set in memory.

These costs must be balanced against the smaller costs of the same thing
done multiple times, including query overhead, to produce the multiple
result sets, then performing the record association in memory.
Incidentally, if you have the right indices for a WHERE IN clause to be
efficient, you have the right indices for the joins to be efficient, so
that''s a wash.

The real question is where the line is such that the cost of dealing with
the number of rows, M, produced by a multi-table join is greater than the
cost of multiple queries and associating records in code. I think you need
a whole lot of superfluous rows to win with multiple queries, but I could
be wrong. This is where benchmarking comes in. Please do investigate.

> Thanks,
> Mathijs
--Greg