Rails - Dec 2010 - Having problems with rails connectivity to postgres (libpq?)

I have some daemons that connect to a PG database and are pretty busy.
 Every few days at least one of them starts failing with this error.

PGError: lost synchronization with server: got message type
"something"

Sometimes the message type is an empty string. Other times it''s a
letter like "E" or something. It seems to vary from incident to
incident.

This is highly annoying because at this point the daemon becomes
useless. If I stop the daemon and start it again everything works
fine. A few days later some other daemon or maybe the same one will
fail with a similar message.

I am running the latest version of rails 3.0.3 on an up to date ubuntu
connected to postgres 8.4.

Any idea of what is going on?

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

I have recently deployed a similar setup, Ubuntu 10.04, pgsql 8.4,
rails 3.0.3, pg gem etc.

I have started to see this error occasionally, that request fails then
subsequent requests are OK, the pgsql log says the client
disconnected!

Sequel::DatabaseDisconnectError (PGError: server closed the connection
unexpectedly

I''ve not had problems before with older versions.

Take a look at you postgresql log and see if you can get clues there,
I''m not sure what is causing my problems either.
/var/log/posgresql/...

On Dec 17, 1:47 pm, Tim Uckun
<timuc...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> I have some daemons that connect to a PG database and are pretty busy.
>  Every few days at least one of them starts failing with this error.
>
> PGError: lost synchronization with server: got message type
"something"
>
> Sometimes the message type is an empty string. Other times it''s a
> letter like "E" or something. It seems to vary from incident to
> incident.
>
> This is highly annoying because at this point the daemon becomes
> useless. If I stop the daemon and start it again everything works
> fine. A few days later some other daemon or maybe the same one will
> fail with a similar message.
>
> I am running the latest version of rails 3.0.3 on an up to date ubuntu
> connected to postgres 8.4.
>
> Any idea of what is going on?
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

On Sat, Dec 18, 2010 at 11:05 AM, Jim Morris
<wolfmanjm-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> I have recently deployed a similar setup, Ubuntu 10.04, pgsql 8.4,
> rails 3.0.3, pg gem etc.
>
> I have started to see this error occasionally, that request fails then
> subsequent requests are OK, the pgsql log says the client
> disconnected!
>
In my case the application is unable to connect back to the database.
> Sequel::DatabaseDisconnectError (PGError: server closed the connection
> unexpectedly
>
I was thinking it might be an activerecord problem but if you are
having the same issue with sequel maybe it''s at a lower level than
that.

Somebody suggested it might be a connection pool or a thread pool
issue but I have no way of testing that.

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

I''ve spent the day trying to track this down. I am not seeing it with
my older rails/activerecord app, only with the newer rails 3.0.3/
sequel app.

I "think" it may be the ssl connection to postgresql. On newer Ubuntu
systems a local tcp connection uses ssl by default (kind of odd but
still).

What I did was to remove the host,username,password settings from
database.yml which forced it to use a local unix pipe to connect, so
far I have not seen this error, but it is pretty infrequent.

You can also switch off SSL in the posgresql,conf file., check to see
if ssl is set to true in that file, or when yo connect at the command
line with
> psql -h localhost yourdbname
if it prints out SSL connection (cipher: DHE-RSA-AES256-SHA, bits:
256) then it is using SSL.

Let me know.

On Dec 18, 4:56 am, Tim Uckun
<timuc...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Sat, Dec 18, 2010 at 11:05 AM, Jim Morris
<wolfma...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > I have recently deployed a similar setup, Ubuntu 10.04, pgsql 8.4,
> > rails 3.0.3, pg gem etc.
>
> > I have started to see this error occasionally, that request fails then
> > subsequent requests are OK, the pgsql log says the client
> > disconnected!
>
> In my case the application is unable to connect back to the database.
>
> > Sequel::DatabaseDisconnectError (PGError: server closed the connection
> > unexpectedly
>
> I was thinking it might be an activerecord problem but if you are
> having the same issue with sequel maybe it''s at a lower level than
> that.
>
> Somebody suggested it might be a connection pool or a thread pool
> issue but I have no way of testing that.
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

On Sun, Dec 19, 2010 at 7:26 PM, Jim Morris
<wolfmanjm-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> I''ve spent the day trying to track this down. I am not seeing it
with
> my older rails/activerecord app, only with the newer rails 3.0.3/
> sequel app.
I am seeing it using activerecord on rails 3.0.3

The problem just occured and although the client is constantly putting
out errors like this

	PGError: lost synchronization with server: got message type "o",
length 544501536
	PGError: lost synchronization with server: got message type "�",
length -65536
	PGError: lost synchronization with server: got message type "�",
length -65536
	PGError: lost synchronization with server: got message type "�",
length -65536
	PGError: lost synchronization with server: got message type "o",
length 544501536
       PGError: lost synchronization with server: got message type
"o", length 544501536
       PGError: lost synchronization with server: got message type
"o", length 544501536



there is nothing in the postgres log.
>
> I "think" it may be the ssl connection to postgresql. On newer
Ubuntu
> systems a local tcp connection uses ssl by default (kind of odd but
> still).
>
> What I did was to remove the host,username,password settings from
> database.yml which forced it to use a local unix pipe to connect, so
> far I have not seen this error, but it is pretty infrequent.
>
I can''t do this because my database is on another server.

> You can also switch off SSL in the posgresql,conf file., check to see
> if ssl is set to true in that file, or when yo connect at the command
> line with
>
>> psql -h localhost yourdbname
>
> if it prints out SSL connection (cipher: DHE-RSA-AES256-SHA, bits:
> 256) then it is using SSL.
It''s definitely using SSL.  I might try turning off the SSL and see if
that does any good.

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

Yea turn off ssl if you are on  local network and see if that
helps...

I suspect what is happening in my case, (and maybe yours), is the SSL
connection is "renegotiating" and failing, which is documented
elsewhere (google postgresql ssl renogotiation)

So far I have not seen this issue since I switched to a local
connection. Although I have two other older rails apps running
presumably with SSL and they have not exhibited this error.

On Dec 19, 2:32 am, Tim Uckun
<timuc...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Sun, Dec 19, 2010 at 7:26 PM, Jim Morris
<wolfma...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > I''ve spent the day trying to track this down. I am not seeing
it with
> > my older rails/activerecord app, only with the newer rails 3.0.3/
> > sequel app.
>
> I am seeing it using activerecord on rails 3.0.3
>
> The problem just occured and although the client is constantly putting
> out errors like this
>
>         PGError: lost synchronization with server: got message type
"o",
> length 544501536
>         PGError: lost synchronization with server: got message type "
", length -65536
>         PGError: lost synchronization with server: got message type "
", length -65536
>         PGError: lost synchronization with server: got message type "
", length -65536
>         PGError: lost synchronization with server: got message type
"o",
> length 544501536
>        PGError: lost synchronization with server: got message type
> "o", length 544501536
>        PGError: lost synchronization with server: got message type
> "o", length 544501536
>
> there is nothing in the postgres log.
>
>
>
> > I "think" it may be the ssl connection to postgresql. On
newer Ubuntu
> > systems a local tcp connection uses ssl by default (kind of odd but
> > still).
>
> > What I did was to remove the host,username,password settings from
> > database.yml which forced it to use a local unix pipe to connect, so
> > far I have not seen this error, but it is pretty infrequent.
>
> I can''t do this because my database is on another server.
>
> > You can also switch off SSL in the posgresql,conf file., check to see
> > if ssl is set to true in that file, or when yo connect at the command
> > line with
>
> >> psql -h localhost yourdbname
>
> > if it prints out SSL connection (cipher: DHE-RSA-AES256-SHA, bits:
> > 256) then it is using SSL.
>
> It''s definitely using SSL.  I might try turning off the SSL and
see if
> that does any good.
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

> I suspect what is happening in my case, (and maybe yours), is the SSL
> connection is "renegotiating" and failing, which is documented
> elsewhere (google postgresql ssl renogotiation)
>
> So far I have not seen this issue since I switched to a local
> connection. Although I have two other older rails apps running
> presumably with SSL and they have not exhibited this error.
I have turned it off. Let''s see if the problem occurs again.

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

It turns out you can also turn off ssl in the database.yml file by
adding

sslmode: disable

Although I have found no way to determine if the connection is SSL or
not.

On Dec 19, 1:30 pm, Tim Uckun
<timuc...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> > I suspect what is happening in my case, (and maybe yours), is the SSL
> > connection is "renegotiating" and failing, which is
documented
> > elsewhere (google postgresql ssl renogotiation)
>
> > So far I have not seen this issue since I switched to a local
> > connection. Although I have two other older rails apps running
> > presumably with SSL and they have not exhibited this error.
>
> I have turned it off. Let''s see if the problem occurs again.
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.