Re: Database recommendations?

Paul Butcher
Thu, 22 Feb 2007 21:13:05 -0000

Ilan Berci wrote:
> I think your real problem is trying to diagnose why your slave is
> crashing before you assume that you are running up against the upper
> limits of MySQL replication.

I''d certainly welcome advice on how to perform this diagnosis. Part of
the problem that we have is that MySQL provides us with almost no help
whatsoever in diagnosing the problem.

We know what was the last statement which went through before the crash,
but it''s "just a query". As far as we can tell, the problem
is that the
master and the slave got "out of whack" at some point in the past (and
we have no means to tell when this point was). The point of failure is
some stage after that when an otherwise entirely innocuous query runs.

Diagnosing the problem is also not helped by the fact that it has only
happened twice in 14 months! Most of the time everything runs just fine.
And we''ve not found a way to reproduce it (we do have automated stress
tests, but they don''t seem to do the trick).

We''re thinking of moving database at least in part because, quite apart
from this problem, we know that we''re going to hit a wall with MySQL
replication anyway. The slave exists to provide a failover if the master
dies. Let''s imagine that happens 6 months down the line. We can
failover
to the slave (assuming that the replication is working at the time!),
but at some point we''ll need to recreate a replication relationship
from
the new master to a new slave. MySQL replication requires us to take the
master down while we create the slave. With a database which is an order
of magnitude larger than we have right now, which is the size we expect
it to be in 6 months, this is going to mean being down for an hour -
which is completely unacceptable for our app.

So we''ve been thinking of moving anyway - the current problems
we''ve
been experiencing have just accelerated our thinking.

>  What are the details of your master/slave
> setup?, (a)synchronous?, distributed query load to slave?,
> auto-generated keys? load?.

Most of our tables are InnoDB, but we do have a couple of MyISAM tables
(because we use MySQL fulltext indices on those tables). The slave runs
read-only and replicates from the binary log on the master (which is a
very standard MySQL replication setup as far as I can tell). Just about
all tables are "standard" RoR style tables, so they have autoincrement
"id" columns, but are pretty much "plain vanilla" apart from
that. No
stored procedures, no transactions (other than the implicit transactions
that InnoDB creates as part of its normal operation). We do use foreign
key constraints though (although I can''t see why that should cause any
problems).

> It should be fairly straight forward to look at the db logs on the slave
> to see the last statement that went through before the crash..   I also
> guess it''s obvious to assume that your slave is a dedicated db
server..
> right?   Is the crash dependant upon load?

Yes, the slave is a dedicated server. It''s hard to say whether the
crash
is dependent on load - as it''s only happened twice it''s hard
to see a
pattern!

Thanks for any suggestions you can offer!

paul.butcher->msgCount++

--
Posted via http://www.ruby-forum.com/.