Rails - Dec 2006 - Serving files on different file systems at the same time

Hello,

I''m experiencing a strange behaviour with lighty
I''m running lighty 1.14.3 on a debian box (kernel 2.6.18-3)

my server is serving big files (usually 350MB) coming from 2 different
filesystems : one on my debian box (fast internet connection : 100Mb/s),
let''s call it "fast fs" and the other on another box with
slow internet
connection (1Mb/s), let''s call it "slow fs"

slow fs is mounted on the fast box using NFS over VPN

when i download a file hosted on fast fs, it''s fast :)
when i download a file hosted on slow fs, it''s slow :/
so far so good

the weird behaviour is when i download a file from slow fs and, at the
same time, another file from fast fs, both files are downloaded slowly,
which is not the expected behaviour (i''d like the file from fast fs to
come fast and the file from slow fs to come slowly)

the result is : if any client is downloading a file from slow fs, every
other clients are downloading at the same (slow) rate as he is.

what''s going on ?
is it a fork/thread problem (i think lighty does not use threads, right ?)
it may look like some kind of FIFO problem or something (i don''t even
know if light is using a fifo)
i''m using fastcgi (RubyOnRails website), but i don''t think
it''s involved
when serving static files, is it ?

it''s probably not a QOS problem since the problem does not occurs when
the traffic between the fast box and the slow box is done by cp''ing a
file through NFS, that''s why i think lighty''s the problem.

Thanks in advance

PS : the site is password protected, but i could put a file from fast fs
and another file from slow fs on public availability for testing purpose
if you like to.

-- 
Mike
KameHouse Prod.

What you''re seeing is probably very simple.

Lighty is a single process server and as such it is sensitive to 
blocking syscalls, any such blocking syscall (read, write, stat, 
readdir) blocking long enough, in your case slow NFS access, will cause 
the entire process to be blocked by the kernel until that syscall is 
done doing whatever it is supposed to be doing.

What I would recommend is, depending on your exact needs, is to either 
use the config setting server.max-worker=XX where XX is the number of 
worker processes lighty will launch to serve your requests or to run 
multiple instances of lighty, one for your "fast" fs and one for your 
"slow" one.

NFS may not be the best choice for high performance file serving.

Marc
> Hello,
>
> I''m experiencing a strange behaviour with lighty
> I''m running lighty 1.14.3 on a debian box (kernel 2.6.18-3)
>
> my server is serving big files (usually 350MB) coming from 2 different
> filesystems : one on my debian box (fast internet connection : 100Mb/s),
> let''s call it "fast fs" and the other on another box
with slow internet
> connection (1Mb/s), let''s call it "slow fs"
>
> slow fs is mounted on the fast box using NFS over VPN
>
> when i download a file hosted on fast fs, it''s fast :)
> when i download a file hosted on slow fs, it''s slow :/
> so far so good
>
> the weird behaviour is when i download a file from slow fs and, at the
> same time, another file from fast fs, both files are downloaded slowly,
> which is not the expected behaviour (i''d like the file from fast
fs to
> come fast and the file from slow fs to come slowly)
>
> the result is : if any client is downloading a file from slow fs, every
> other clients are downloading at the same (slow) rate as he is.
>
> what''s going on ?
> is it a fork/thread problem (i think lighty does not use threads, right ?)
> it may look like some kind of FIFO problem or something (i don''t
even
> know if light is using a fifo)
> i''m using fastcgi (RubyOnRails website), but i don''t
think it''s involved
> when serving static files, is it ?
>
> it''s probably not a QOS problem since the problem does not occurs
when
> the traffic between the fast box and the slow box is done by
cp''ing a
> file through NFS, that''s why i think lighty''s the
problem.
>
> Thanks in advance
>
> PS : the site is password protected, but i could put a file from fast fs
> and another file from slow fs on public availability for testing purpose
> if you like to.
>
>

Just Marc wrote:
> 
> What you''re seeing is probably very simple.
> 
> Lighty is a single process server and as such it is sensitive to
> blocking syscalls, any such blocking syscall (read, write, stat,
> readdir) blocking long enough, in your case slow NFS access, will cause
> the entire process to be blocked by the kernel until that syscall is
> done doing whatever it is supposed to be doing.
> 
> What I would recommend is, depending on your exact needs, is to either
> use the config setting server.max-worker=XX where XX is the number of
> worker processes lighty will launch to serve your requests or to run
> multiple instances of lighty, one for your "fast" fs and one for
your
> "slow" one.
> 
> NFS may not be the best choice for high performance file serving.
> 
> Marc
> 
>> Hello,
>>
>> I''m experiencing a strange behaviour with lighty
>> I''m running lighty 1.14.3 on a debian box (kernel 2.6.18-3)
>>
>> my server is serving big files (usually 350MB) coming from 2 different
>> filesystems : one on my debian box (fast internet connection :
100Mb/s),
>> let''s call it "fast fs" and the other on another box
with slow internet
>> connection (1Mb/s), let''s call it "slow fs"
>>
>> slow fs is mounted on the fast box using NFS over VPN
>>
>> when i download a file hosted on fast fs, it''s fast :)
>> when i download a file hosted on slow fs, it''s slow :/
>> so far so good
>>
>> the weird behaviour is when i download a file from slow fs and, at the
>> same time, another file from fast fs, both files are downloaded slowly,
>> which is not the expected behaviour (i''d like the file from
fast fs to
>> come fast and the file from slow fs to come slowly)
>>
>> the result is : if any client is downloading a file from slow fs, every
>> other clients are downloading at the same (slow) rate as he is.
>>
>> what''s going on ?
>> is it a fork/thread problem (i think lighty does not use threads,
>> right ?)
>> it may look like some kind of FIFO problem or something (i
don''t even
>> know if light is using a fifo)
>> i''m using fastcgi (RubyOnRails website), but i don''t
think it''s involved
>> when serving static files, is it ?
>>
>> it''s probably not a QOS problem since the problem does not
occurs when
>> the traffic between the fast box and the slow box is done by
cp''ing a
>> file through NFS, that''s why i think lighty''s the
problem.
>>
>> Thanks in advance
>>
>> PS : the site is password protected, but i could put a file from fast
fs
>> and another file from slow fs on public availability for testing
purpose
>> if you like to.
>>
>>   
> 
> 
Thanks for your reply, Marc

the problem with the max-worker solution is to make sure that the
process serving slow fs file will not serve any fast fs file
i''ll try to run 2 instances of lighty and let you know about it

what would you recommend instead of NFS ?

-- 
Mike
KameHouse Prod.

Michael Baudino wrote:
> Hello,
> 
> I''m experiencing a strange behaviour with lighty
> I''m running lighty 1.14.3 on a debian box (kernel 2.6.18-3)
> 
> my server is serving big files (usually 350MB) coming from 2 different
> filesystems : one on my debian box (fast internet connection : 100Mb/s),
> let''s call it "fast fs" and the other on another box
with slow internet
> connection (1Mb/s), let''s call it "slow fs"
> 
> slow fs is mounted on the fast box using NFS over VPN
> 
> when i download a file hosted on fast fs, it''s fast :)
> when i download a file hosted on slow fs, it''s slow :/
> so far so good
> 
> the weird behaviour is when i download a file from slow fs and, at the
> same time, another file from fast fs, both files are downloaded slowly,
> which is not the expected behaviour (i''d like the file from fast
fs to
> come fast and the file from slow fs to come slowly)
> 
> the result is : if any client is downloading a file from slow fs, every
> other clients are downloading at the same (slow) rate as he is.
> 
> what''s going on ?
> is it a fork/thread problem (i think lighty does not use threads, right ?)
> it may look like some kind of FIFO problem or something (i don''t
even
> know if light is using a fifo)
> i''m using fastcgi (RubyOnRails website), but i don''t
think it''s involved
> when serving static files, is it ?
> 
> it''s probably not a QOS problem since the problem does not occurs
when
> the traffic between the fast box and the slow box is done by
cp''ing a
> file through NFS, that''s why i think lighty''s the
problem.
> 
> Thanks in advance
> 
> PS : the site is password protected, but i could put a file from fast fs
> and another file from slow fs on public availability for testing purpose
> if you like to.
> 
I tryed this max-worker thing and it appears to work very well
Thanks a lot to Marc and Sajendra for your answers

-- 
Mike
KameHouse Prod.

On Sun, 03 Dec 2006 14:21:37 +0100
Michael Baudino <gornack@snowboard.com> wrote:
> what would you recommend instead of NFS ?
Well...I think than saying that ''NFS is not good'' in this case
is wrong -
Basically, you can''t server files fast enough over a 1 Mbps link :D.

There are several protocols you can use to serve your files from a
network storage (or other server) (of course, there''s always attached
storage
and SANs ;) ) , but I think your main issue to solve is the slow link to your
storage. Work on that first, then worry about changing NFS out of the way.

_________________________
{Beto|Norberto|Numard} Meijome

"Always listen to experts.  They''ll tell you what can''t
be done, and why.
Then do it."
  Robert A. Heinlein

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.

On Sunday 03 05:21, Michael Baudino wrote:
> Thanks for your reply, Marc
>
> the problem with the max-worker solution is to make sure that the
> process serving slow fs file will not serve any fast fs file
> i''ll try to run 2 instances of lighty and let you know about it
>
> what would you recommend instead of NFS ?
Can you install a webserver(lighttpd/apache/etc...) on your NFS server and 
just proxy requests for slow fs files.  mod_proxy will not block on file IO, 
since it does non-blocking network IO.

If you can''t install a webserver on the NFS server, then try second
instance
of lighttpd with different config/port.  Have the front-end lighttpd proxy to 
second lighttpd for slow fs files.  That would offload the blocking 
operations onto a different process.

Also using HTTP instead of NFS over a slow connection, might improve the 
transfer speed.


-- 
Robert Jakabosky
Senior Software Developer
AlphaTrade.com
Phone: (604) 681-7503
Fax: (604) 681-7710
Cell: (604) 721-5283

DISCLAIMER:

This message is intended only for the use of the individual or entity to which 
it is addressed and may contain information which is privileged, confidential 
or subject to copyright. Internet communications cannot be guaranteed to be 
secure or error-free as information could be intercepted, corrupted, lost, 
arrive late or contain viruses. The information contained in this e-mail is 
drawn from sources believed to be reliable, but the accuracy and completeness 
of the information is not guaranteed, nor in providing it does AlphaTrade.com 
assume any liability. This footnote also confirms that this email message has 
been swept for the presence of computer viruses.

*************************************************