Rails - Jun 2007 - rails/mongrel cluster - keeping media in sync across nodes with a CSS/LB

Hi all,

I figured I would see what other people would do and get some ideas on
what route I want to take.
Let me first give you a brief summary of my network and application.

I have a Cisco PIX FW and a Cisco 11501 CSS/LB, there are 5 network
nodes behind them. Three are webservers running apache/mongrel and 2
are mysql servers ( master/slave replication )

The three webservers have limited disk space.

I need a way to keep my data in sync across all nodes. A user would
upload artwork and other media after logging in, so since the data is
private I can''t create a VIP in the LB like http://mydomain.com/images
and route it to a specific node.

So essientially, I am trying to figure out the best way to keep my
media uploads in sync across all nodes. Right now I am using a mounted
NFS mount to a master server on each client, but I am afraid of the
bottleneck.

I can''t use a master upload server and Rsync hence the disk space
issue.

I am open to any suggestions that you guys have here, I don''t want to
have my media uploads "split-brained". Any thoughts on this would be
appreciated.

Regards,
-- Mike

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

We use Amazon S3 for storage, which works well across several  
servers.  It''s also instantaneous if you''re on EC2 (but
it''s still
limited beta).

Duane Johnson
(canadaduane)

On Jun 5, 2007, at 9:40 AM, Michael Steinfeld wrote:
>
> Hi all,
>
> I figured I would see what other people would do and get some ideas on
> what route I want to take.
> Let me first give you a brief summary of my network and application.
>
> I have a Cisco PIX FW and a Cisco 11501 CSS/LB, there are 5 network
> nodes behind them. Three are webservers running apache/mongrel and 2
> are mysql servers ( master/slave replication )
>
> The three webservers have limited disk space.
>
> I need a way to keep my data in sync across all nodes. A user would
> upload artwork and other media after logging in, so since the data is
> private I can''t create a VIP in the LB like
http://mydomain.com/images
> and route it to a specific node.
>
> So essientially, I am trying to figure out the best way to keep my
> media uploads in sync across all nodes. Right now I am using a mounted
> NFS mount to a master server on each client, but I am afraid of the
> bottleneck.
>
> I can''t use a master upload server and Rsync hence the disk space
> issue.
>
> I am open to any suggestions that you guys have here, I don''t want
to
> have my media uploads "split-brained". Any thoughts on this would
be
> appreciated.
>
> Regards,
> -- Mike
>
> >

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

You only have 3 options: database, shared file system, replication.
If you are not able to increase the disk space on the web servers then
replication is out.  If you are afraid of a bottleneck then shared
files are out.  And storing large files in a database is not any
better than shared files.  So I guess nothing works.

Really, the answer is to look at the requirements.  How big are the
files, how many hits, etc.  In most cases replicated files is the best
solution because you only take a hit on upload, not access.  But, that
requires you add disk.  How much will more disk cost?  Not much.  If
you have low enough volume then do not worry about NFS or database
blobs being the bottleneck.  If you have enough volume then move the
performance hit to the upload side, not the access side.

Michael

On Jun 5, 8:40 am, "Michael Steinfeld"
<mikeisgr...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> Hi all,
>
> I figured I would see what other people would do and get some ideas on
> what route I want to take.
> Let me first give you a brief summary of my network and application.
>
> I have a Cisco PIX FW and a Cisco 11501 CSS/LB, there are 5 network
> nodes behind them. Three are webservers running apache/mongrel and 2
> are mysql servers ( master/slave replication )
>
> The three webservers have limited disk space.
>
> I need a way to keep my data in sync across all nodes. A user would
> upload artwork and other media after logging in, so since the data is
> private I can''t create a VIP in the LB
likehttp://mydomain.com/images
> and route it to a specific node.
>
> So essientially, I am trying to figure out the best way to keep my
> media uploads in sync across all nodes. Right now I am using a mounted
> NFS mount to a master server on each client, but I am afraid of the
> bottleneck.
>
> I can''t use a master upload server and Rsync hence the disk space
issue.
>
> I am open to any suggestions that you guys have here, I don''t want
to
> have my media uploads "split-brained". Any thoughts on this would
be
> appreciated.
>
> Regards,
> -- Mike

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---