Rails - Jul 2006 - Does Berkeley DB and Rails seem like a good match?

I''ve been reading up on Berkeley DB and I''ve been refelcting
on the
Rails philosophy and it just seemed to me like the two should go 
together hand in glove.  Rails seeks to abstract the programer from 
squirrely database coding which is great.  My only problem is that 
you''re still left with a large component, namely the databsase server 
(MySQL, PostgreSQL..etc) which is treated like a black box...these 
databases are hugely complex and maybe shouldn''t be treated like blck 
boxes, maybe a small light-weight database, essentially a giant hashing 
table (i.e. Berkeley DB) would result in lower complexity and better 
performance?

what does everyone think?  I''m happy to be shot out of the water by 
those with more experience than me :)

-- 
Posted via http://www.ruby-forum.com/.

SQLite works very well in this regard.

http://www.sqlite.org/
http://wiki.rubyonrails.com/rails/pages/SQLite


Stephen Gerstacker

-----Original Message-----
From: rails-bounces@lists.rubyonrails.org
[mailto:rails-bounces@lists.rubyonrails.org] On Behalf Of Andrew Gibson
Sent: Friday, July 07, 2006 11:27
To: rails@lists.rubyonrails.org
Subject: [Rails] Does Berkeley DB and Rails seem like a good match?

I''ve been reading up on Berkeley DB and I''ve been refelcting
on the
Rails philosophy and it just seemed to me like the two should go 
together hand in glove.  Rails seeks to abstract the programer from 
squirrely database coding which is great.  My only problem is that 
you''re still left with a large component, namely the databsase server 
(MySQL, PostgreSQL..etc) which is treated like a black box...these 
databases are hugely complex and maybe shouldn''t be treated like blck 
boxes, maybe a small light-weight database, essentially a giant hashing 
table (i.e. Berkeley DB) would result in lower complexity and better 
performance?

what does everyone think?  I''m happy to be shot out of the water by 
those with more experience than me :)

-- 
Posted via http://www.ruby-forum.com/.
_______________________________________________
Rails mailing list
Rails@lists.rubyonrails.org
http://lists.rubyonrails.org/mailman/listinfo/rails

Andrew Gibson wrote:
> I''ve been reading up on Berkeley DB and I''ve been
refelcting on the
> Rails philosophy and it just seemed to me like the two should go 
> together hand in glove.  Rails seeks to abstract the programer from 
> squirrely database coding which is great.  My only problem is that 
> you''re still left with a large component, namely the databsase
server
> (MySQL, PostgreSQL..etc) which is treated like a black box...these 
> databases are hugely complex and maybe shouldn''t be treated like
blck
> boxes, maybe a small light-weight database, essentially a giant hashing 
> table (i.e. Berkeley DB) would result in lower complexity and better 
> performance?
> 
> what does everyone think?  I''m happy to be shot out of the water
by
> those with more experience than me :)
Rails needs a database that supports SQL.

SQLite might be the lightweight database you are looking for.

   http://www.sqlite.org/

Not sure of the quality of the information, but there''s advice on using
SQLite with Rails on the wiki here:

   http://wiki.rubyonrails.org/rails/pages/HowtoUseSQLite

regards

   Justin

On 7/7/06, Andrew Gibson <gibson_andrew@yahoo.com>
wrote:
> I''ve been reading up on Berkeley DB and I''ve been
refelcting on the
> Rails philosophy and it just seemed to me like the two should go
> together hand in glove.  Rails seeks to abstract the programer from
> squirrely database coding which is great.  My only problem is that
> you''re still left with a large component, namely the databsase
server
> (MySQL, PostgreSQL..etc) which is treated like a black box...these
> databases are hugely complex and maybe shouldn''t be treated like
blck
> boxes, maybe a small light-weight database, essentially a giant hashing
> table (i.e. Berkeley DB) would result in lower complexity and better
> performance?
>
> what does everyone think?  I''m happy to be shot out of the water
by
> those with more experience than me :)
I don''t know about Berkeley DB, but SQLite works fine.  As long as you
have few updates relative to your reads, and you don''t need
super-performance, it''s a decent choice.  I''ve had a Rails
site
running for several months now using that db.  It''s certainly simple.

http://www.sqlite.org/

Honestly though, I''ll be using Postgres from now on for anything
except the simplest of apps.

-- James

Ok, just out of curiosity, why do you want/need postgres over sqlite?

what isn''t sqlite giving you?
> http://www.sqlite.org/
> 
> Honestly though, I''ll be using Postgres from now on for anything
> except the simplest of apps.
> 
> -- James

-- 
Posted via http://www.ruby-forum.com/.

Andrew Gibson wrote:
> Ok, just out of curiosity, why do you want/need postgres over sqlite?
> 
> what isn''t sqlite giving you?
Take a look at the concurrency issues mentioned here:

http://www.sqlite.org/faq.html#q7

It appears that Sqlite locks the entire database for any writes, while other 
databases lock individual rows or the entire table, allowing more clients to 
update the database simultaneously.

Regards,
Blair

-- 
Blair Zajac, Ph.D.
http://www.orcaware.com/svn/

Fair point, which brings me back to Berkeley DB, you can check out the 
info here:
http://www.sleepycat.com/docs/ref/toc.html
http://en.wikipedia.org/wiki/Berkeley_db


I''m not trying to be stubborn here, I would realy like to try and start
a dicussion about this...



it''s not an SQL based data solution but it sounds like it''s
got top
notch performance (256 terrabytes addressable,
> 
> Take a look at the concurrency issues mentioned here:
> 
> http://www.sqlite.org/faq.html#q7
> 
> It appears that Sqlite locks the entire database for any writes, while 
> other
> databases lock individual rows or the entire table, allowing more 
> clients to
> update the database simultaneously.
> 
> Regards,
> Blair

-- 
Posted via http://www.ruby-forum.com/.

Andrew Gibson wrote:
> Fair point, which brings me back to Berkeley DB, you can check out the 
> info here:
> http://www.sleepycat.com/docs/ref/toc.html
> http://en.wikipedia.org/wiki/Berkeley_db
> 
> 
> I''m not trying to be stubborn here, I would realy like to try and
start
> a dicussion about this...
> 
> 
> 
> it''s not an SQL based data solution but it sounds like
it''s got top
> notch performance (256 terrabytes addressable,
...but as I wrote earlier, Rails (ActiveRecord) requires a database with 
SQL support.

You could put a different back end behind ActionPack, but you would lose 
an awful lot.

regards

   Justin

Andrew Gibson wrote:
> Fair point, which brings me back to Berkeley DB, you can check out the 
> info here:
> http://www.sleepycat.com/docs/ref/toc.html
> http://en.wikipedia.org/wiki/Berkeley_db
>
>
> I''m not trying to be stubborn here, I would realy like to try and
start
> a dicussion about this...
>   
I''ve got a little bit of anecdotal information about this topic. 
Assaph
Mehr wrote an ActiveRecord adapter for KirbyBase, called Ackbar.  Now, 
KirbyBase does not use SQL as it''s query language; it uses Ruby blocks.
So, inside the guts of Ackbar, Assaph had to do all kinds of strange 
morphing to convert the SQL string received from ActiveRecord into a 
Ruby block that KirbyBase could understand.

He got this working with Rails 1.0.  Unfortunately, Rails 1.1 broke it 
and I''m not sure Assaph is going to attempt to get Ackbar working with 
Rails 1.1.  It is not a trivial task to keep Ackbar in sync with 
ActiveRecord changes.

Now, writing an ActiveRecord adapter might be easier for Berkeley DB, 
since it''s query language is much simpler than KirbyBase''s.

As a side note, just for fun, a few days ago I started playing around 
with building a Rails app and using KirbyBase as the database back-end, 
but *not* using ActiveRecord (i.e. my model did not inherit from 
ActiveRecord''s Base class).  It was fairly easy to get basic #find 
functionality working all the way through the controller to the view.  
However, when I attempted to do #updates, what I started to miss was all 
of the cool stuff, like errors being propagated from the model back to 
the view and showing up all nice and pretty.

It''s definitely do-able to use a non-ActiveRecord backend in Rails, but
you will either have to live without a lot of the niceties that the 
ActiveRecord-Rails integration gives you, or try to duplicate it in your 
own code.

Maybe ActiveResource will prove to be another alternative.

Jamey Cribbs

Confidentiality Notice: This email message, including any attachments, is for
the sole use of the intended recipient(s) and may contain confidential and/or
privileged information. If you are not the intended recipient(s), you are hereby
notified that any dissemination, unauthorized review, use, disclosure or
distribution of this email and any materials contained in any attachments is
prohibited. If you receive this message in error, or are not the intended
recipient(s), please immediately notify the sender by email and destroy all
copies of the original message, including attachments.

Now we''re cookin'' with gas here:)

 In terms of interacting with the databse, there are already a couple of 
Ruby libs for talking to Berkeley DB...  you can find them on rubyforge 
and also on Darwin Ports

the reason why I started thinking along these lines was that it just 
seemed like an awful lot of overhead to create the ORM layer to wrap 
around an entire seperate database program, especially when as rubyists 
we want to code as much as possible in Ruby..so why not find a data 
storage solution which allows to always stay in Ruby-land and that acts 
as a giant hash-table?

I guess I should stop running off my mouth at this point though and 
maybe roll of my sleeves and try to integrate one of these libs into 
Rails and see how it all works out

Ron M wrote:
> Justin Forder wrote:
>  > Andrew Gibson wrote:
>  >> I''ve been reading up on Berkeley DB and I''ve
been refelcting ...
>  >
>  > Rails needs a database that supports SQL.
> 
> You exaggerate.
> 
> I''ve been doing fine with an application that uses a web service
> (the "solr" front end to lucene) for all its models rather than
> a database.
> 
> AFAIK the only subset of Rails that wants a SQL database is the
> ActiveRecord part.  And even when I do use a database with Rails,
> I tend not to use ActiveRecord stuff, since I find ActiveRecord
> rather annoying (relationships between tables defined in
> the ruby class instead of by foreign key constraints; can''t
> understand user-defined types in the database; etc).

-- 
Posted via http://www.ruby-forum.com/.

Justin Forder wrote:
 > Andrew Gibson wrote:
 >> I''ve been reading up on Berkeley DB and I''ve been
refelcting ...
 >
 > Rails needs a database that supports SQL.

You exaggerate.

I''ve been doing fine with an application that uses a web service
(the "solr" front end to lucene) for all its models rather than
a database.

AFAIK the only subset of Rails that wants a SQL database is the
ActiveRecord part.  And even when I do use a database with Rails,
I tend not to use ActiveRecord stuff, since I find ActiveRecord
rather annoying (relationships between tables defined in
the ruby class instead of by foreign key constraints; can''t
understand user-defined types in the database; etc).

Ron M wrote:
> Justin Forder wrote:
>  > Andrew Gibson wrote:
>  >> I''ve been reading up on Berkeley DB and I''ve
been refelcting ...
>  >
>  > Rails needs a database that supports SQL.
> 
> You exaggerate.
> 
> I''ve been doing fine with an application that uses a web service
> (the "solr" front end to lucene) for all its models rather than
> a database.
> 
> AFAIK the only subset of Rails that wants a SQL database is the
> ActiveRecord part.  And even when I do use a database with Rails,
> I tend not to use ActiveRecord stuff, since I find ActiveRecord
> rather annoying (relationships between tables defined in
> the ruby class instead of by foreign key constraints; can''t
> understand user-defined types in the database; etc).
You are quite right that it is ActiveRecord that requires SQL - I made 
this clearer in my second post in this thread. I would be happier if 
Rails separated its persistence solution from its domain modelling 
features, but at the moment they come in one bundle, ActiveRecord. 
That''s why I said, in my other post, that Andrew would be losing a lot 
by discarding ActiveRecord.

regards

   Justin

You do know that MySQL has the BDB table type, don''t you? The only
thing
I use the BDB database for is SVN.

Joe

-- 
Posted via http://www.ruby-forum.com/.

> Now we''re cookin'' with gas here:)
agricultural methane, i''d hope..
> the reason why I started thinking along these lines was that it just 
> seemed like an awful lot of overhead to create the ORM layer to wrap 
> around an entire seperate database program
dynamic object persistence is still a research topic.. i dont remember SQL being
proclaimed god and ORM his right hand, so much as it was the most pragmatic
solution at the current time
> , especially when as rubyists 
> we want to code as much as possible in Ruby..
one can write an entire fully featured and functional site, without touching
SQL, or HTML, or bash, or CSS, or JavaScript. to me this seems like a pretty
cool achievement. finally web development can complete with things like Tcl/Tk
and QT and Smalltalk in terms of what it offers to the _developer_.
> so why not find a data 
> storage solution which allows to always stay in Ruby-land and that acts 
> as a giant hash-table?
go for it.. the name ActiveHash isnt taken, btw ;). it would be great to have
this option well integrated.

> I guess I should stop running off my mouth at this point though and 
> maybe roll of my sleeves and try to integrate one of these libs into 
> Rails and see how it all works out

carmen wrote:
>> Now we''re cookin'' with gas here:)
> 
> agricultural methane, i''d hope..
???
> one can write an entire fully featured and functional site, without 
> touching [...] HTML [...]
Really?

Joe

-- 
Posted via http://www.ruby-forum.com/.

On 7-jul-2006, at 21:08, Andrew Gibson wrote:
> the reason why I started thinking along these lines was that it just
> seemed like an awful lot of overhead to create the ORM layer to wrap
> around an entire seperate database program, especially when as  
> rubyists
> we want to code as much as possible in Ruby..so why not find a data
> storage solution which allows to always stay in Ruby-land and that  
> acts
> as a giant hash-table?
Three words: queries, joins and concurrency.

When you have joins (what SQL databases do very well) you can depend  
on the DB for managing
the datastore in multiple related tables. When you have BDB (or such)  
you have to join the rows yourself,
because all convenience that you got is a Giant Hashtable. An SQL  
database is an airplane, BDB is a hot air balloon
(but a very robust one). Another problem is typing.

Try that with BDB (pseudocode):

find_all_entries whose date_created is between (today and 3 days ago)

The problem with joins is that you end up pulling lots of data out of  
the DB to do most simple comparisons/computations
as soon as you need more than one entity.

MovableType uses BDB because it''s the only choice for hosts where an  
SQL database is not available. Granted, it allowed
me to use it back in the day when all I had was my student account,  
but I would not go with BDB for most of data-driven apps now.

You can take a look at all the scaffolding that is used to support  
BDB in MT and you will realise that with many tables it becomes
more trouble than it''s worth.