Rails - Dec 2009 - Approaches to improve response times for web pages who's HTML & content is loaded through Rails???

Hi,

I''m serving files (text, images) to a browser via my Rails application
as the user needs to be authenticated too. The files are on file
system, however references are in the database, as I''m using
PAPERCLIP. Using AUTHLOGIC for authentication. I''m finding the
response times aren''t good, as each request for HTML/image etc have to
go through the Rails layers I guess of AUTHLOGIC, PAPERCLIP then
stream back.

QUESTION: Given I want users to authenticate for HTML & associated
files (CSS, images) then page needs, is there any obvious way to get
an order of magnitude improvement? E.G.

a) I guess I could save in session state the list of children files a
HTML page needs & get its file location path, until the subsequent
requests come, and that would some a database lookup (at the expense
of using session state)?

b) is there a way to keep open a TCP session or something for the
entire web-page load? (clutching at straws here)

c) any way to stream all html & images/css as one file/stream to
browser?  (I assume not as this isn''t how the browser works)

d) Combine Javascript into one file (I''ve heard of this one and can
follow up)

e) Set future expiry headers on files (I''ve heard of this one and can
follow up)



Thanks



--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

PS.  On average each request is about 3000ms to 5000ms when being
served by RoR (authlogic/paperclip), as opposed to about 160ms for the
same content but put directly under /public.  I''m using mongrel.

On Dec 16, 4:22 pm, greghauptmann
<greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> Hi,
>
> I''m serving files (text, images) to a browser via my Rails
application
> as the user needs to be authenticated too. The files are on file
> system, however references are in the database, as I''m using
> PAPERCLIP. Using AUTHLOGIC for authentication. I''m finding the
> response times aren''t good, as each request for HTML/image etc
have to
> go through the Rails layers I guess of AUTHLOGIC, PAPERCLIP then
> stream back.
>
> QUESTION: Given I want users to authenticate for HTML & associated
> files (CSS, images) then page needs, is there any obvious way to get
> an order of magnitude improvement? E.G.
>
> a) I guess I could save in session state the list of children files a
> HTML page needs & get its file location path, until the subsequent
> requests come, and that would some a database lookup (at the expense
> of using session state)?
>
> b) is there a way to keep open a TCP session or something for the
> entire web-page load? (clutching at straws here)
>
> c) any way to stream all html & images/css as one file/stream to
> browser?  (I assume not as this isn''t how the browser works)
>
> d) Combine Javascript into one file (I''ve heard of this one and
can
> follow up)
>
> e) Set future expiry headers on files (I''ve heard of this one and
can
> follow up)
>
> Thanks
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

I believe that paperclip actually stores a copy of the images in the
public directory by default.  Once the html is rendered to the browser
the image requests should not go through the rails stack.

Let us know what firebug says.  Browser load times are dependent on
many factors, and often the rails stack is not the slowest piece of
the pie.  Once you prove there is some evidence that rails is
significantly slowing you down, then it would be worth trying to
optimize it.  But minimizing javascript, css, and images often has a
more profound effect on browser load times.  Then caching pages,
actions, etc helps immensely too.

Let us know the firebug or safari browser load time for each part of
the http requests.

Andrew

--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

thanks - so I do want to have the images etc authenticated as well by
the way, so this is why I haven''t got them under public either...

so the two test scenarios are based on doing a "save as" from a
website, and then:
(a) saving the web content directly under /public - which runs fast
(b) having the content in a protected area, and then having rails
serve it up via a controller - again that does (a) authentication via
authlogic and then (b) serves via paperclip which stores files on
disk, however there is a look to the database to find out where the
file is

I''m not sure how to attach the images here, however the results
basically show that on average, picking some of the images/css etc
that the browser fetches:
* a ~10kb javascript file:  10ms => 497ms
* a 1kb image: 5ms => 29ms
.
.
This is measured from a browser running on the same server as the ./
script/server -e production, mongrel server...



On Dec 16, 5:18 pm, Andrew Pace
<andrewpp...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> I believe that paperclip actually stores a copy of the images in the
> public directory by default.  Once the html is rendered to the browser
> the image requests should not go through the rails stack.
>
> Let us know what firebug says.  Browser load times are dependent on
> many factors, and often the rails stack is not the slowest piece of
> the pie.  Once you prove there is some evidence that rails is
> significantly slowing you down, then it would be worth trying to
> optimize it.  But minimizing javascript, css, and images often has a
> more profound effect on browser load times.  Then caching pages,
> actions, etc helps immensely too.
>
> Let us know the firebug or safari browser load time for each part of
> the http requests.
>
> Andrew
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

http://www.engineyard.com/blog/2009/your-pages-will-load-faster-with-rails/


On Dec 16, 2:19 pm, greghauptmann
<greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> thanks - so I do want to have the images etc authenticated as well by
> the way, so this is why I haven''t got them under public either...
>
> so the two test scenarios are based on doing a "save as" from a
> website, and then:
> (a) saving the web content directly under /public - which runs fast
> (b) having the content in a protected area, and then having rails
> serve it up via a controller - again that does (a) authentication via
> authlogic and then (b) serves via paperclip which stores files on
> disk, however there is a look to the database to find out where the
> file is
>
> I''m not sure how to attach the images here, however the results
> basically show that on average, picking some of the images/css etc
> that the browser fetches:
> * a ~10kb javascript file:  10ms => 497ms
> * a 1kb image: 5ms => 29ms
> .
> .
> This is measured from a browser running on the same server as the ./
> script/server -e production, mongrel server...
>
> On Dec 16, 5:18 pm, Andrew Pace
<andrewpp...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> > I believe that paperclip actually stores a copy of the images in the
> > public directory by default.  Once the html is rendered to the browser
> > the image requests should not go through the rails stack.
>
> > Let us know what firebug says.  Browser load times are dependent on
> > many factors, and often the rails stack is not the slowest piece of
> > the pie.  Once you prove there is some evidence that rails is
> > significantly slowing you down, then it would be worth trying to
> > optimize it.  But minimizing javascript, css, and images often has a
> > more profound effect on browser load times.  Then caching pages,
> > actions, etc helps immensely too.
>
> > Let us know the firebug or safari browser load time for each part of
> > the http requests.
>
> > Andrew
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

Some good points in the link - but I''m thinking it still
doesn''t
address the large fundamental increase in response time when having to
go through rails for the authentication etc.  In both cases in the
comparison for example there was no expires time populated.

I''m thinking maybe what i need to address is whether it''s
normal that
adding some extra code paths and a few database calls (authlogic and
paperclip lookup) should be taking this extra 300-400ms or so, or
whether there is something wrong here?  Perhaps I''ll need to under the
paperclip and authlogic code so I put some logging code in to measure
the times?


On Dec 16, 7:29 pm, Arun Srini
<arunro...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> http://www.engineyard.com/blog/2009/your-pages-will-load-faster-with-...
>
> On Dec 16, 2:19 pm, greghauptmann
<greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>
>
> > thanks - so I do want to have the images etc authenticated as well by
> > the way, so this is why I haven''t got them under public
either...
>
> > so the two test scenarios are based on doing a "save as"
from a
> > website, and then:
> > (a) saving the web content directly under /public - which runs fast
> > (b) having the content in a protected area, and then having rails
> > serve it up via a controller - again that does (a) authentication via
> > authlogic and then (b) serves via paperclip which stores files on
> > disk, however there is a look to the database to find out where the
> > file is
>
> > I''m not sure how to attach the images here, however the
results
> > basically show that on average, picking some of the images/css etc
> > that the browser fetches:
> > * a ~10kb javascript file:  10ms => 497ms
> > * a 1kb image: 5ms => 29ms
> > .
> > .
> > This is measured from a browser running on the same server as the ./
> > script/server -e production, mongrel server...
>
> > On Dec 16, 5:18 pm, Andrew Pace
<andrewpp...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> > > I believe that paperclip actually stores a copy of the images in
the
> > > public directory by default.  Once the html is rendered to the
browser
> > > the image requests should not go through the rails stack.
>
> > > Let us know what firebug says.  Browser load times are dependent
on
> > > many factors, and often the rails stack is not the slowest piece
of
> > > the pie.  Once you prove there is some evidence that rails is
> > > significantly slowing you down, then it would be worth trying to
> > > optimize it.  But minimizing javascript, css, and images often
has a
> > > more profound effect on browser load times.  Then caching pages,
> > > actions, etc helps immensely too.
>
> > > Let us know the firebug or safari browser load time for each part
of
> > > the http requests.
>
> > > Andrew
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

I am still not convinced the problem is with rails and not just
because there are a lot of images and files to load.

I like your thought to try to see exactly how long each process
takes.  The terminal in development mode can be helpful too.  It will
list the time for many of these things.

Here is another resource to help.   http://railslab.newrelic.com/scaling-rails

I don''t know the nature of the site, but if at all possible you should
try to use the http server to serve css, javascript, and images.  You
could even resort to other forms of authentication, using apache if
you had to.

--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

On Tue, Dec 15, 2009 at 11:13 PM, greghauptmann
<greg.hauptmann-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> PS.  On average each request is about 3000ms to 5000ms when being
> served by RoR (authlogic/paperclip), as opposed to about 160ms for the
> same content but put directly under /public.  I''m using mongrel.
You can get a lot more specifics about what''s happening time-wise
by installing the NewRelic monitoring plugin (http://newrelic.com/).

Highly recommended.

-- 
Hassan Schroeder ------------------------
hassan.schroeder-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
twitter: @hassan

--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

Are you re-checking authentication for each image? Could you find some
way to authenticate only once for all of your resources? Perhaps use
Basic or Digest authentication for your images, and then somehow tie
the password to the current session? If you can remove unnecessary
database hits, you may be able to shave off some ms. You may add a few
ms for HTTP authentication, though, so I''d take a close look.

Are you using a mongrel cluster? If this is all going through one
mongrel instance, it''s only handling one request at a time. Switching
to passenger may help, as it can spawn new instances to handle
simultaneous requests (depending on settings and your server''s
memory).

On Dec 16, 8:02 am, Hassan Schroeder
<hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Tue, Dec 15, 2009 at 11:13 PM, greghauptmann
>
> <greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > PS.  On average each request is about 3000ms to 5000ms when being
> > served by RoR (authlogic/paperclip), as opposed to about 160ms for the
> > same content but put directly under /public.  I''m using
mongrel.
>
> You can get a lot more specifics about what''s happening time-wise
> by installing the NewRelic monitoring plugin (http://newrelic.com/).
>
> Highly recommended.
>
> --
> Hassan Schroeder ------------------------
hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> twitter: @hassan
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

Thanks

* I''ll try new relic then to get some more incite
* re authentication I''m using authlogic so in fact after initial
authentication it should use session cookies.  However authlogic does
store sessions in database..
* re number of images etc and mongrel cluster, I''m only testing with
one mongrel, however I am here looking at per request stats
* re authenticating via apache:  if I knew how I could do this, bit
have it integrated with my apps user provioning I''d look at this. Any
leads here?  I couldn''t find anything when I looked.

Thanks



On Dec 17, 4:00 am, sax
<s...-c5KP9hz/kbxwMgcQ4+lbG0B+6BGkLq7r@public.gmane.org>
wrote:
> Are you re-checking authentication for each image? Could you find some
> way to authenticate only once for all of your resources? Perhaps use
> Basic or Digest authentication for your images, and then somehow tie
> the password to the current session? If you can remove unnecessary
> database hits, you may be able to shave off some ms. You may add a few
> ms for HTTP authentication, though, so I''d take a close look.
>
> Are you using a mongrel cluster? If this is all going through one
> mongrel instance, it''s only handling one request at a time.
Switching
> to passenger may help, as it can spawn new instances to handle
> simultaneous requests (depending on settings and your server''s
> memory).
>
> On Dec 16, 8:02 am, Hassan Schroeder
<hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> wrote:
>
>
>
> > On Tue, Dec 15, 2009 at 11:13 PM, greghauptmann
>
> > <greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > > PS.  On average each request is about 3000ms to 5000ms when being
> > > served by RoR (authlogic/paperclip), as opposed to about 160ms
for the
> > > same content but put directly under /public.  I''m using
mongrel.
>
> > You can get a lot more specifics about what''s happening
time-wise
> > by installing the NewRelic monitoring plugin (http://newrelic.com/).
>
> > Highly recommended.
>
> > --
> > Hassan Schroeder ------------------------
hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> > twitter: @hassan
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

tried new relic however it requires silver to do a web transaction
view, which seems to be a minimum $100 monthly even if I only want to
use it for 10 minutes...I bit expensive for a hobbyist programmer who
just wants to get time response time log points in to delineate times
spent in: rails generic code, authlogic, paperclip, database...   I
might try to work out how to put logging into these points myself
first.

Any advice welcome on how/where to put logging code that would:
a) be the first bit of rails code hit when the request comes in?
b) first point called out to authlogic for authentication (perhaps
this is just in the specific controller before the
"before_filter :require_user" line, and one at end point?
c) first point for paperclip call (would be just in the controller
before teh "webfile = Webfile.find_by_path( search_path )" line
perhaps)?
d) point where all the above calls complete and the streaming of data
back to the browser starts (perhaps just in the controller at the "
send_file webfile.file.path , :type =>
webfile.file_content_type, :disposition => ''inline''"
line)?



On Dec 17, 4:36 am, greghauptmann
<greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> Thanks
>
> * I''ll try new relic then to get some more incite
> * re authentication I''m using authlogic so in fact after initial
> authentication it should use session cookies.  However authlogic does
> store sessions in database..
> * re number of images etc and mongrel cluster, I''m only testing
with
> one mongrel, however I am here looking at per request stats
> * re authenticating via apache:  if I knew how I could do this, bit
> have it integrated with my apps user provioning I''d look at this.
Any
> leads here?  I couldn''t find anything when I looked.
>
> Thanks
>
> On Dec 17, 4:00 am, sax
<s...-c5KP9hz/kbxwMgcQ4+lbG0B+6BGkLq7r@public.gmane.org> wrote:
>
> > Are you re-checking authentication for each image? Could you find some
> > way to authenticate only once for all of your resources? Perhaps use
> > Basic or Digest authentication for your images, and then somehow tie
> > the password to the current session? If you can remove unnecessary
> > database hits, you may be able to shave off some ms. You may add a few
> > ms for HTTP authentication, though, so I''d take a close look.
>
> > Are you using a mongrel cluster? If this is all going through one
> > mongrel instance, it''s only handling one request at a time.
Switching
> > to passenger may help, as it can spawn new instances to handle
> > simultaneous requests (depending on settings and your
server''s
> > memory).
>
> > On Dec 16, 8:02 am, Hassan Schroeder
<hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> > wrote:
>
> > > On Tue, Dec 15, 2009 at 11:13 PM, greghauptmann
>
> > > <greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> > > > PS.  On average each request is about 3000ms to 5000ms when
being
> > > > served by RoR (authlogic/paperclip), as opposed to about
160ms for the
> > > > same content but put directly under /public.  I''m
using mongrel.
>
> > > You can get a lot more specifics about what''s happening
time-wise
> > > by installing the NewRelic monitoring plugin
(http://newrelic.com/).
>
> > > Highly recommended.
>
> > > --
> > > Hassan Schroeder ------------------------
hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> > > twitter: @hassan
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

On Wed, Dec 16, 2009 at 11:38 AM, greghauptmann
<greg.hauptmann-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> tried new relic however it requires silver to do a web transaction
> view, which seems to be a minimum $100 monthly
Mmmm. Haven''t been paying attention, apparently their business
model has, er, evolved  :-)

Alternatively, you might want to look at the rails-footnotes gem.

HTH,
-- 
Hassan Schroeder ------------------------
hassan.schroeder-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
twitter: @hassan

--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.

PS I should say what the free screen did show from a high level point
for the call I made was:

* average response time broken down by tier
* ~40ms Ruby
* ~140ms Mongrel wait
* ~3ms Database



On Dec 17, 5:47 am, Hassan Schroeder
<hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Wed, Dec 16, 2009 at 11:38 AM, greghauptmann
>
> <greg.hauptm...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > tried new relic however it requires silver to do a web transaction
> > view, which seems to be a minimum $100 monthly
>
> Mmmm. Haven''t been paying attention, apparently their business
> model has, er, evolved  :-)
>
> Alternatively, you might want to look at the rails-footnotes gem.
>
> HTH,
> --
> Hassan Schroeder ------------------------
hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> twitter: @hassan
--

You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.