Mongrel users - Sep 2006 - mongrel thread safety and global variables

In environment.rb file, I initialize a connection to some data vending
servers, through TCP sockets.

The connection object is global and hence the code:

#environment.rb
$generic_connection = ConnectionClass.instance (singleton class)
$generic_connection.connect_me( this call will make the connection)


The above approach is to make sure that, only one connection is made to the
data vending servers.ConnectionClass is a library Class that, I have written
to handle connections and to ensure exception handling and all that stuff.

Now, in our rails code, whenever we need some data from these data vending
servers, we use this global variable to get the data.

#in controllers
$generic_connection.get_data(''<our internal protocol>'')



Since, we switched to mongrel, we are facing a strange issue. Our protocol
code, specify what kind of data we want from data vending servers.So, lets
say we want, x informatio, then we specify xxx as protocol code and get the
data. protocol code "yyy" would give completely different kind of
data.
Now..sometimes what happens is, an Ajax call requests xxx protocol code and
in the response we get data of protocol code yyy. Data vending servers are
threaded again, and for each connection it starts a new thread and each
request is served on a new thread.
Data vending servers are again mostly written in ruby.

Now, I am wondering where is the problem?

As i know, mongrel serves requests on new threads, so what happens to this
global variable that is shared among the threads.Is there a possiblity that,
problem is here?
I mean, lets say a thread is using this global variable for fetching data
for protocol code xxx and same time another thread requests for protocol
code yyy. My connection class, which is basically a proxy to these data
vending servers, may not be thread safe.If this is the problem, then can
anybody suggest better approach of making single connection to data vending
servers, without using shared global variables.



It could be a problem with our data vending servers also.But I am not so,
sure about that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060901/7313c88e/attachment.html

On 9/1/06, hemant <gethemant at gmail.com> wrote:
>
>
> In environment.rb file, I initialize a connection to some data vending
> servers, through TCP sockets.
>
> The connection object is global and hence the code:
>
> #environment.rb
> $generic_connection = ConnectionClass.instance (singleton class)
> $generic_connection.connect( this call will make the connection)
>
>
> The above approach is to make sure that, only one connection is made to
> the data vending servers.ConnectionClass is a library Class that, I have
> written to handle connections and to ensure exception handling and all that
> stuff.
>
> Now, in our rails code, whenever we need some data from these data vending
> servers, we use this global variable to get the data.
>
> #in controllers
> $generic_connection.get_data(''<our internal
protocol>'')
>
>
>
> Since, we switched to mongrel, we are facing a strange issue. Our protocol
> code, specify what kind of data we want from data vending servers.So, lets
> say we want, x informatio, then we specify xxx as protocol code and get the
> data. protocol code "yyy" would give completely different kind of
data.
> Now..sometimes what happens is, an Ajax call requests xxx protocol code and
> in the response we get data of protocol code yyy. Data vending servers are
> threaded again, and for each connection it starts a new thread and each
> request is served on a new thread.
> Data vending servers are again mostly written in ruby.
>
> Now, I am wondering where is the problem?
>
> As i know, mongrel serves requests on new threads, so what happens to this
> global variable that is shared among the threads.Is there a possiblity
> that, problem is here?
> I mean, lets say a thread is using this global variable for fetching data
> for protocol code xxx and same time another thread requests for protocol
> code yyy. My connection class, which is basically a proxy to these data
> vending servers, may not be thread safe.If this is the problem, then can
> anybody suggest better approach of making single connection to data vending
> servers, without using shared global variables.
>
>
>
> It could be a problem with our data vending servers also.But I am not so,
> sure about that.
>
>
For anyone interested here is the ConnectionClass code:

require ''socket''
require ''timeout''
require ''singleton''
require ''yaml''

class ConnectionClass
  include Singleton

  def connect
    p = YAML.load(File.read("#{RAILS_ROOT}/config/parameters.yaml"))
    @socket_file = p["socket_file"]
    return connect_to_proxy
  end


  def connect_to_proxy
    begin
      @proxy_con = UNIXSocket.open(@socket_file.strip)
      return @proxy_con
    rescue Exception => e
      return nil
    end
  end

  def get_data(request_string)
    req_array = request_string.split('':'')
    code = req_array[0].strip
    msg = req_array[1].strip
    symbol = req_array[2].strip

    return get_data_from_proxy(code,symbol)

  end


  def get_data_from_proxy(code,symbol)
    temp_string = code + ''<'' + symbol +
''>''

    ret_data = nil

    begin
      @proxy_con.puts(temp_string)
    rescue Errno::EPIPE
      unless connect_to_proxy.nil?
        @proxy_con.puts(temp_string)
      else
        return nil
      end
    rescue Exception => e
      return nil
    end

    begin
      timeout(8) do
        ret_data = @proxy_con.gets()
      end
    rescue Timeout::Error
      return nil
    rescue Exception => e
      return nil
    end


    if ret_data =~ /^600*/
      return nil
    elsif ret_data =~ /^620*/
      return nil
    else
      return ret_data
    end
  end

end

Evidently, when rails load(i.e mongrel starts), the only call made is,
$generic_connection.connect(from environment.rb), and hence within this
class, only @proxy_con should be shared among the threads IMHO.
So, what happens when subsequent calls to $generic_connection.get_data is
made from controller(ie mongrel threads)? while one thread is waiting for
@proxy_con.gets, another thread can call the same method, but the question
is, both the threads will share local variables defined within get_data and
get_data_from_proxy methods?

If i make, both the methods threaded, it should solve the problem? but i am
wondering, how mongrel would handle it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060901/ae07b793/attachment-0001.html

Will, mutex solve the problem?
I tried using them and seem to work.But as i said, the problem occurs very
less often.

There is another problem, if i use Mutex to synchronize, calls, then some
calls would take long and time and some would return immediately.so, a
request that takes long time, will block the other one.



On 9/1/06, hemant <gethemant at gmail.com> wrote:
>
> On 9/1/06, hemant <gethemant at gmail.com> wrote:
>
> >
> > In environment.rb file, I initialize a connection to some data vending
> > servers, through TCP sockets.
> >
> > The connection object is global and hence the code:
> >
> > #environment.rb
> > $generic_connection = ConnectionClass.instance (singleton class)
> > $generic_connection.connect( this call will make the connection)
> >
>
> The above approach is to make sure that, only one connection is made to
> the data vending servers.ConnectionClass is a library Class that, I have
> written to handle connections and to ensure exception handling and all that
> stuff.
>
> Now, in our rails code, whenever we need some data from these data vending
> servers, we use this global variable to get the data.
>
> #in controllers
> $generic_connection.get_data(''<our internal
protocol>'')
>
>
>
> Since, we switched to mongrel, we are facing a strange issue. Our protocol
> code, specify what kind of data we want from data vending servers.So, lets
> say we want, x informatio, then we specify xxx as protocol code and get the
> data. protocol code "yyy" would give completely different kind of
data.
> Now..sometimes what happens is, an Ajax call requests xxx protocol code and
> in the response we get data of protocol code yyy. Data vending servers are
> threaded again, and for each connection it starts a new thread and each
> request is served on a new thread.
> Data vending servers are again mostly written in ruby.
>
> Now, I am wondering where is the problem?
>
> As i know, mongrel serves requests on new threads, so what happens to this
> global variable that is shared among the threads.Is there a possiblity
> that, problem is here?
> I mean, lets say a thread is using this global variable for fetching data
> for protocol code xxx and same time another thread requests for protocol
> code yyy. My connection class, which is basically a proxy to these data
> vending servers, may not be thread safe.If this is the problem, then can
> anybody suggest better approach of making single connection to data vending
> servers, without using shared global variables.
>
>
>
> It could be a problem with our data vending servers also.But I am not so,
> sure about that.
>
>
> For anyone interested here is the ConnectionClass code:
>
> require ''socket''
> require ''timeout''
> require ''singleton''
> require ''yaml''
>
> class ConnectionClass
>   include Singleton
>
>   def connect
>     p =
YAML.load(File.read("#{RAILS_ROOT}/config/parameters.yaml"))
>     @socket_file = p["socket_file"]
>     return connect_to_proxy
>   end
>
>
>   def connect_to_proxy
>     begin
>       @proxy_con = UNIXSocket.open(@socket_file.strip)
>       return @proxy_con
>     rescue Exception => e
>       return nil
>     end
>   end
>
>   def get_data(request_string)
>     req_array = request_string.split('':'')
>     code = req_array[0].strip
>     msg = req_array[1].strip
>     symbol = req_array[2].strip
>
>     return get_data_from_proxy(code,symbol)
>
>   end
>
>
>   def get_data_from_proxy(code,symbol)
>     temp_string = code + ''<'' + symbol +
''>''
>
>     ret_data = nil
>
>     begin
>       @proxy_con.puts(temp_string)
>     rescue Errno::EPIPE
>       unless connect_to_proxy.nil?
>         @proxy_con.puts(temp_string)
>       else
>         return nil
>       end
>     rescue Exception => e
>       return nil
>     end
>
>     begin
>       timeout(8) do
>         ret_data = @proxy_con.gets()
>       end
>     rescue Timeout::Error
>       return nil
>     rescue Exception => e
>       return nil
>     end
>
>
>     if ret_data =~ /^600*/
>       return nil
>     elsif ret_data =~ /^620*/
>       return nil
>     else
>       return ret_data
>     end
>   end
>
> end
>
> Evidently, when rails load(i.e mongrel starts), the only call made is,
> $generic_connection.connect(from environment.rb), and hence within this
> class, only @proxy_con should be shared among the threads IMHO.
> So, what happens when subsequent calls to $generic_connection.get_data is
> made from controller(ie mongrel threads)? while one thread is waiting for
> @proxy_con.gets, another thread can call the same method, but the question
> is, both the threads will share local variables defined within get_data and
> get_data_from_proxy methods?
>
> If i make, both the methods threaded, it should solve the problem? but i
> am wondering, how mongrel would handle it.
>


-- 
nothing much to talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060901/274bbf9f/attachment.html

On Fri, 2006-09-01 at 17:36 +0530, hemant wrote:
> 
> In environment.rb file, I initialize a connection to some data vending
> servers, through TCP sockets.
> 
<snip>
> As i know, mongrel serves requests on new threads, so what happens to
> this global variable that is shared among the threads.Is there a
> possiblity that, problem is here?
> I mean, lets say a thread is using this global variable for fetching
> data for protocol code xxx and same time another thread requests for
> protocol code yyy. My connection class, which is basically a proxy to
> these data vending servers, may not be thread safe.If this is the
> problem, then can anybody suggest better approach of making single
> connection to data vending servers, without using shared global
> variables.
Well, Mongrel locks Rails with a Sync (Mutex caused leaks on some
platforms) so there should be only one user of your global variable at
any one time.  Also, the threads aren''t using your global after the
request is finished and the lock is released since rails is done
dispatching.

But, if you''re running multiple Mongrels, then you''ll get
concurrent
access to your backend server.  Unless that server is written to support
concurrent access then you''ll have these kinds of issues.

BTW, have you considered just using DRb?  It''s really fast, very easy
to
use, and it would probably solve many of these issues.


-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

>
> Well, Mongrel locks Rails with a Sync (Mutex caused leaks on some
> platforms) so there should be only one user of your global variable at
> any one time.  Also, the threads aren''t using your global after
the
> request is finished and the lock is released since rails is done
> dispatching.
>
> But, if you''re running multiple Mongrels, then you''ll get
concurrent
> access to your backend server.  Unless that server is written to support
> concurrent access then you''ll have these kinds of issues.
>
> BTW, have you considered just using DRb?  It''s really fast, very
easy to
> use, and it would probably solve many of these issues.
>
> <snip>
> --
> Zed A. Shaw
>

So, can''t be a issue with my ConnectionClass, right?

Basically this UNIXSocket that i connect(sorry for confusion, its actually a
UNIXSocket ) , is a proxy between actual data vending servers and
rails.Itserves multiple clients(not only rails client, other clients
connect to it
using TCPSockets) and is written to support multiple concurrent connections.
But if I can confirm that there is no issue, with rails connections, then we
can inform the other team(i am not author of this proxy server), that please
have a look at your code, and see if its thread safe.

We have been initially using DRb, but there was a issue with connection
pooling.Besides, fetching data in real time from data vending servers was
costly, hence this  proxy does caching and all that stuff.  As per my
understanding, the proxy is written like, for each connection it will create
a new thread, and a new thread for putting the data to other data vending
servers and another thread for receiving the response and sending back to
all the connected clients.It attaches unique ids to each request, to make
sure that, no data mis-match occurs.

But if it would have been written with druby, then it would be a little
difficult for other clients to connect.

Thanks for help Zed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060902/3f7d390c/attachment.html

On 9/1/06, hemant <gethemant at gmail.com> wrote:
[snip]
> So, can''t be a issue with my ConnectionClass, right?
>
> Basically this UNIXSocket that i connect(sorry for confusion, its actually
a
> UNIXSocket ) , is a proxy between actual data vending servers and rails.It
> serves multiple clients(not only rails client, other clients connect to it
> using TCPSockets) and is written to support multiple concurrent
connections.
> But if I can confirm that there is no issue, with rails connections, then
we
> can inform the other team(i am not author of this proxy server), that
please
> have a look at your code, and see if its thread safe.
UNIXSocket is like file sockets, right? that means One process
accessing it could put garbage IF other process jump into the socket.

Someone correct me if I''m wrong, but multiple mongrel running from the
same file socket will produce the behavior you''re getting (misplaced
stuff in the replies).
>
>  We have been initially using DRb, but there was a issue with connection
> pooling.Besides, fetching data in real time from data vending servers was
> costly, hence this  proxy does caching and all that stuff.  As per my
> understanding, the proxy is written like, for each connection it will
create
> a new thread, and a new thread for putting the data to other data vending
> servers and another thread for receiving the response and sending back to
> all the connected clients.It attaches unique ids to each request, to make
> sure that, no data mis-match occurs.
A pool of connections (one implemented around multiuser protocols...
TCPSocket) is my suggestion.

If your ConnectionClass could connect to your vending server using a
TCP Socket, it will work fine under multiple mongrels.
>
> But if it would have been written with druby, then it would be a little
> difficult for other clients to connect.
>
> Thanks for help Zed.
>
-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

We initially chose UNIXSocket over TCPSockets, because of performance
reasons.

As it turns out, its not so scalabale..so we may scrap it.


On 9/2/06, Luis Lavena <luislavena at gmail.com>
wrote:
>
> On 9/1/06, hemant <gethemant at gmail.com> wrote:
> [snip]
> > So, can''t be a issue with my ConnectionClass, right?
> >
> > Basically this UNIXSocket that i connect(sorry for confusion, its
> actually a
> > UNIXSocket ) , is a proxy between actual data vending servers and
> rails.It
> > serves multiple clients(not only rails client, other clients connect
to
> it
> > using TCPSockets) and is written to support multiple concurrent
> connections.
> > But if I can confirm that there is no issue, with rails connections,
> then we
> > can inform the other team(i am not author of this proxy server), that
> please
> > have a look at your code, and see if its thread safe.
>
> UNIXSocket is like file sockets, right? that means One process
> accessing it could put garbage IF other process jump into the socket.
>
> Someone correct me if I''m wrong, but multiple mongrel running from
the
> same file socket will produce the behavior you''re getting
(misplaced
> stuff in the replies).
>
> >
> >  We have been initially using DRb, but there was a issue with
connection
> > pooling.Besides, fetching data in real time from data vending servers
> was
> > costly, hence this  proxy does caching and all that stuff.  As per my
> > understanding, the proxy is written like, for each connection it will
> create
> > a new thread, and a new thread for putting the data to other data
> vending
> > servers and another thread for receiving the response and sending back
> to
> > all the connected clients.It attaches unique ids to each request, to
> make
> > sure that, no data mis-match occurs.
>
> A pool of connections (one implemented around multiuser protocols...
> TCPSocket) is my suggestion.
>
> If your ConnectionClass could connect to your vending server using a
> TCP Socket, it will work fine under multiple mongrels.
>
> >
> > But if it would have been written with druby, then it would be a
little
> > difficult for other clients to connect.
> >
> > Thanks for help Zed.
> >
>
> --
> Luis Lavena
> Multimedia systems
> -
> Leaders are made, they are not born. They are made by hard effort,
> which is the price which all of us must pay to achieve any goal that
> is worthwhile.
> Vince Lombardi
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
nothing much to talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060902/0d595bbd/attachment.html

Wanted to throw out there that we just launched our Rails application  
running on Apache 2.2/Mongrel. We''ve been working on it for a few  
months and it''s finally up and running - check out
http://www.blurb.com/

Blurb allows people to create, publish and sell professionally  
looking books online, although book creation software is not  
currently web-based, and is a client download for Windows/Mac.

Special thanks to Zed and the rest of the mongrel and cluster teams  
for providing such a great rails application server.

K

On 9/1/06, hemant <gethemant at gmail.com> wrote:
>
> We initially chose UNIXSocket over TCPSockets, because of performance
> reasons.
>
> As it turns out, its not so scalabale..so we may scrap it.
Ok, is your UNIXServer implemented in a multithread fashion? (so it
could hadle multiple clients to connect).

how its coded?

Like this:

require ''socket''
SOCKET = "/tmp/sample"
server_thread = Thread.start do # run server in a thread
  sock = UNIXServer.open(SOCKET)
  s1 = sock.accept
  p s1.recvfrom(124)
end

client = UNIXSocket.open(SOCKET)
client.send("hello", 0)
client.close
server_thread.join

I compared unix sockets to windows pipes... which is wrong, my mistake.

Have you tried connect to ruby clients in the same machine to the
running socket server?

If that works, then your problem could be... anything ;-) (just kidding)

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

Yes...its a multithreaded and supposed to server request flawlessly.

BTW, as a  quick check...i wrote a ruby program to fetch data for some 10000
iterations and and append it to a CSV file...and didn''t get a single
data
mangling error.

Next step, I will try 100 simultaneous threads, fetching the data and see if
there is a mis-match.



On 9/2/06, Luis Lavena <luislavena at gmail.com>
wrote:
>
> On 9/1/06, hemant <gethemant at gmail.com> wrote:
> >
> > We initially chose UNIXSocket over TCPSockets, because of performance
> > reasons.
> >
> > As it turns out, its not so scalabale..so we may scrap it.
>
> Ok, is your UNIXServer implemented in a multithread fashion? (so it
> could hadle multiple clients to connect).
>
> how its coded?
>
> Like this:
>
> require ''socket''
> SOCKET = "/tmp/sample"
> server_thread = Thread.start do # run server in a thread
> sock = UNIXServer.open(SOCKET)
> s1 = sock.accept
> p s1.recvfrom(124)
> end
>
> client = UNIXSocket.open(SOCKET)
> client.send("hello", 0)
> client.close
> server_thread.join
>
> I compared unix sockets to windows pipes... which is wrong, my mistake.
>
> Have you tried connect to ruby clients in the same machine to the
> running socket server?
>
> If that works, then your problem could be... anything ;-) (just kidding)
>
> --
> Luis Lavena
> Multimedia systems
> -
> Leaders are made, they are not born. They are made by hard effort,
> which is the price which all of us must pay to achieve any goal that
> is worthwhile.
> Vince Lombardi
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
nothing much to talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060902/44a94857/attachment.html

On Sat, 2006-09-02 at 17:00 +0530, hemant wrote:
> Yes...its a multithreaded and supposed to server request flawlessly.
>  
> BTW, as a  quick check...i wrote a ruby program to fetch data for some
> 10000 iterations and and append it to a CSV file...and didn''t get
a
> single data mangling error.
>  
> Next step, I will try 100 simultaneous threads, fetching the data and
> see if there is a mis-match.
And randomize it.  Ruby''s threads work fairly predictably so without a
random short think time to mix things up you won''t get good results.



-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Well, I think i have figured out the problem with my code (*but i am not
entirely sure..so please correct me if i am wrong)

If you look at above code, then you will see:

 begin
   timeout(6) do
    # read from socket
   end
rescue Timeout::Error
  # error handling
end


My assertion says, sometimes for certain protocol codes, the time to read
from the socket is more than 6 seconds and hence that "gets request"
is
timed out there. Now..in the next call, when i put some new request to the
socket...i get response for the previous request, because socket still
contains data that was written to it after the timeout. Ugly,right?

Solution I have thought is:
  1. Send a unique GUID as a extended modifier with each request and expect
that proxy will echo it to us.
  2. Somehow clear the socket, before read...not entirely sure abt
this.Somebody on ruby-lang  also suggested that, use Eventmachine.



On 9/2/06, Zed Shaw <zedshaw at zedshaw.com>
wrote:
>
> On Sat, 2006-09-02 at 17:00 +0530, hemant wrote:
> > Yes...its a multithreaded and supposed to server request flawlessly.
> >
> > BTW, as a  quick check...i wrote a ruby program to fetch data for some
> > 10000 iterations and and append it to a CSV file...and didn''t
get a
> > single data mangling error.
> >
> > Next step, I will try 100 simultaneous threads, fetching the data and
> > see if there is a mis-match.
>
> And randomize it.  Ruby''s threads work fairly predictably so
without a
> random short think time to mix things up you won''t get good
results.
>
>
>
> --
> Zed A. Shaw
> http://www.zedshaw.com/
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
nothing much to talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060904/3c61ee8b/attachment.html