Rails - Aug 2014 - Issue with hstore returning as string on postgres database, rails app

Summary: rails app using postgres randomly gets into a state where hstore 
array is returned as a string, and cannot be fixed without rebuilding the 
database

We have an app using rails 4.04 with ruby 1.9.3, postgres database version 
9.2.6. Every day, in the test environment, we drop the database, create a 
new database, and restore the data from a production backup using 
pg_restore. The hstore extension is installed, and there is some hstore 
data in the database. The app doesn't get much usage in the test 
environment right now. For the purposes of this problem, we can say it gets 
no usage other than deployment.

The deployment process seems to work perfectly well about 99 times out of 
100, but every once in a while, the app suddenly doesn't work. Trying to 
open the app, you get an error in the log like: 
ActionView::Template::Error(undefeined metghod 'keys' for 
"\"width\"=>"131.0\"".String)

The problem is that it is returning an hstore array as a string. From a 
non-working instance, here is some text from an irb session:

    irb(main):002:0> r=Row.first
    unknown OID: data(-1892003119) (SELECT  "rows".* FROM
"rows"   ORDER BY
"rows"."id" ASC LIMIT 1)
    => #<Row id: 1, data:
"\"test123\"=>\"5\"", created_at:
"2013-10-31
18:56:45", updated_at: "2013-10-31 18:57:05">

From a working instance, here is the same query:

    irb(main):002:0> r=Row.first
    => #<Row id: 1, data: {"test123"=>"5"},
created_at: "2013-10-31
18:56:45", updated_at: "2013-10-31 18:57:05">

You can see that in the broken instance, it is returning the hstore data as 
a string. Some pertinent info is that once it gets in this broken state, 
you can drop and restore the database, or rename the database, and as long 
as it's on the same postgres sever, it's still broken - you have to
rebuild
the whole database for it to work. And if you point the app (change 
database.yml) to point to a different server, it works.

This happens with no code changes. Same ruby on rails code works, then on 
redeployment it doesn't work. We cannot figure out any repro scenario. It 
works, it works, it works, then we do the same thing we've done a hundred 
times and it breaks.

Has anyone ever seen anything like this?

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit
https://groups.google.com/d/msgid/rubyonrails-talk/563a0ede-b78e-4b71-b17b-6b6e689e52f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.