At my university we have a big storage problem. Currently we have 25 Linux servers each equiped with 3 TB of data. Totaling to 75TB of storage. We use each server as a NFS server and have clients mount up the filesystem. They are all connected by a GigE network. The problem we are facing is performance and scalibility. When a users access data, many times they access the same server. This is becoming a huge performance problem. The files sizes range from 8GB to 70GB. Would Lustre solve our performance and management problem? How would lustre scale? I have been reading the quickstart guide, is it easy to implement? TIA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080705/0044b3fc/attachment.html
On Jul 05, 2008 13:36 -0400, Mag Gam wrote:> At my university we have a big storage problem. Currently we have 25 Linux > servers each equiped with 3 TB of data. Totaling to 75TB of storage. We use > each server as a NFS server and have clients mount up the filesystem. They > are all connected by a GigE network. > > The problem we are facing is performance and scalibility. When a users > access data, many times they access the same server. This is becoming a huge > performance problem. The files sizes range from 8GB to 70GB. > > > Would Lustre solve our performance and management problem? How would lustre > scale? I have been reading the quickstart guide, is it easy to implement?As a general rule, Lustre performs very well with large files and linear IO patterns, which seems to match your usage. You can get very good scaling (nearly linear with each OST) and a single client+server can saturate a GigE network (110MB/s). Configuring Lustre 1.6 isn''t much different than NFS. Format each server, mount them, then mount the clients. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
hello andreas, i have totally different enviroment than mag''s we have many small file which less than 1Mb and i didn''t (actually i can''t) know exactly how many it is. it seems more than 10 millions of files.. in our situation, is lustre suitable?? On Mon, Jul 7, 2008 at 1:03 PM, Andreas Dilger <adilger at sun.com> wrote:> On Jul 05, 2008 13:36 -0400, Mag Gam wrote: > > At my university we have a big storage problem. Currently we have 25 > Linux > > servers each equiped with 3 TB of data. Totaling to 75TB of storage. We > use > > each server as a NFS server and have clients mount up the filesystem. > They > > are all connected by a GigE network. > > > > The problem we are facing is performance and scalibility. When a users > > access data, many times they access the same server. This is becoming a > huge > > performance problem. The files sizes range from 8GB to 70GB. > > > > > > Would Lustre solve our performance and management problem? How would > lustre > > scale? I have been reading the quickstart guide, is it easy to implement? > > As a general rule, Lustre performs very well with large files and linear > IO patterns, which seems to match your usage. You can get very good > scaling (nearly linear with each OST) and a single client+server can > saturate a GigE network (110MB/s). > > Configuring Lustre 1.6 isn''t much different than NFS. Format each server, > mount them, then mount the clients. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080707/ab4dd5eb/attachment.html
>> At my university we have a big storage problem. Currently we >> have 25 Linux servers each equiped with 3 TB of data. Totaling >> to 75TB of storage. We use each server as a NFS server and >> have clients mount up the filesystem. They are all connected >> by a GigE network.>> The problem we are facing is performance and scalibility. >> When a users access data, many times they access the same >> server. This is becoming a huge performance problem. The >> files sizes range from 8GB to 70GB.The impression I get from the above is that it is not clear what kind of performance and scalability problems you have. It is far from clear what is the status of any of the following "details": * Storage system structure and speed. * Network performance tuning. * How many users tend to access the same fileserver at once. * Speed of the clients. * Performance of the network switches/routers. * .... You have large scale storage issues, and perhaps the way they are expressed above is less clear than it should be. For example, in the best of cases you can get around 100MB/s reads out of a single NFS server to a single NFS (or Lustre) client, and 3GB will take 30s and 70GB over 10 minutes just to transfer at this speed, irrespective of filesystem performance and scalability. In a similar environment I have been pushing 10G switches/routers and 10G cards for servers and even for clients (and too bad that 10GBASE-T products are still a dream).>> Would Lustre solve our performance and management >> problem?Which management problem? Above you mention only performance. You also don''t mention any reliability/availalability goals, or backup. Also, the performance is not at all clear. Single client speed? Aggregate speed? Server load? ...>> How would lustre scale?To what? A petabyte? A thousand clients? A 10G network? ...>> I have been reading the quickstart guide, is it easy to >> implement?Implementing a high performance, scalable storage pool for 3-70GB files starting at 75TB is not a trivial task and requires a lot of clarity and experience irrespective of the network filesystem used... BTW, I have written some of my ineffable wisdom :-) on the subject here: http://WWW.sabi.co.UK/blog/0804apr.html#080417b However as AndreasD says, Lustre overall seems pretty good at medium-large scientific data storage systems, but the vague nature of your concerns makes a bit difficult to evaluate them. Perhaps those that have only a vague idea of their goals should read a few articles detailing users experiences with Lustre, there are quite a few pointers on the Lustre Wiki, and in various years of the HEPiX workshop.> hello andreas, i have totally different enviroment than mag''s > we have many small file which less than 1Mb and i didn''t > (actually i can''t) know exactly how many it is. it seems more > than 10 millions of files.. in our situation, is lustre > suitable??10^6 each less than 10^6 large. Looks like 1TB, for which an external FW800 or eSATA drive with a small-file friendly filesystem seems a good match. Perhaps JFS or ReiserFS; but ''fsck'' times are going however to be potentially long (days to weeks if the filesystems gets damaged). Lustre has the great advantage that it can ''fsck'' its OSTs in parallel, so perhaps dividing that 1TB into 50-100G OSTs may be of benefit as each filesystem would end up having a lot less files. Note however that Lustre in effect is based on the idea of clients mounting individual files (or file slices) instead of filetrees (with the MDS in the role of the automount directory server), so each file open involves at least two network round trips, which means that latency on metadata-intensive loads will be less awesome.>From the very vague goals you give above it is not at all clearto me whether any of this matters or not. As to the several and often underrated aspects on which a file system design can be evaluated, here is more of my incredible insight :-): http://WWW.sabi.co.UK/blog/0804apr.html#080415
thankyou all answered my question On Mon, Jul 7, 2008 at 12:45 PM, Peter Grandi <pg_lus at lus.for.sabi.co.uk> wrote:> >>> At my university we have a big storage problem. Currently we >>> have 25 Linux servers each equiped with 3 TB of data. Totaling >>> to 75TB of storage. We use each server as a NFS server and >>> have clients mount up the filesystem. They are all connected >>> by a GigE network. > >>> The problem we are facing is performance and scalibility. >>> When a users access data, many times they access the same >>> server. This is becoming a huge performance problem. The >>> files sizes range from 8GB to 70GB. > > The impression I get from the above is that it is not clear what > kind of performance and scalability problems you have. It is far > from clear what is the status of any of the following "details": > > * Storage system structure and speed. > * Network performance tuning. > * How many users tend to access the same fileserver at once. > * Speed of the clients. > * Performance of the network switches/routers. > * .... > > You have large scale storage issues, and perhaps the way they > are expressed above is less clear than it should be. > > For example, in the best of cases you can get around 100MB/s > reads out of a single NFS server to a single NFS (or Lustre) > client, and 3GB will take 30s and 70GB over 10 minutes just to > transfer at this speed, irrespective of filesystem performance > and scalability. In a similar environment I have been pushing > 10G switches/routers and 10G cards for servers and even for > clients (and too bad that 10GBASE-T products are still a dream). > >>> Would Lustre solve our performance and management >>> problem? > > Which management problem? Above you mention only performance. > You also don''t mention any reliability/availalability goals, > or backup. > > Also, the performance is not at all clear. Single client speed? > Aggregate speed? Server load? ... > >>> How would lustre scale? > > To what? A petabyte? A thousand clients? A 10G network? ... > >>> I have been reading the quickstart guide, is it easy to >>> implement? > > Implementing a high performance, scalable storage pool for > 3-70GB files starting at 75TB is not a trivial task and requires > a lot of clarity and experience irrespective of the network > filesystem used... > > BTW, I have written some of my ineffable wisdom :-) on the > subject here: > > http://WWW.sabi.co.UK/blog/0804apr.html#080417b > > However as AndreasD says, Lustre overall seems pretty good at > medium-large scientific data storage systems, but the vague > nature of your concerns makes a bit difficult to evaluate them. > > Perhaps those that have only a vague idea of their goals should > read a few articles detailing users experiences with Lustre, > there are quite a few pointers on the Lustre Wiki, and in > various years of the HEPiX workshop. > >> hello andreas, i have totally different enviroment than mag''s >> we have many small file which less than 1Mb and i didn''t >> (actually i can''t) know exactly how many it is. it seems more >> than 10 millions of files.. in our situation, is lustre >> suitable?? > > 10^6 each less than 10^6 large. Looks like 1TB, for which an > external FW800 or eSATA drive with a small-file friendly > filesystem seems a good match. Perhaps JFS or ReiserFS; but > ''fsck'' times are going however to be potentially long (days to > weeks if the filesystems gets damaged). Lustre has the great > advantage that it can ''fsck'' its OSTs in parallel, so perhaps > dividing that 1TB into 50-100G OSTs may be of benefit as each > filesystem would end up having a lot less files. > > Note however that Lustre in effect is based on the idea of > clients mounting individual files (or file slices) instead of > filetrees (with the MDS in the role of the automount directory > server), so each file open involves at least two network round > trips, which means that latency on metadata-intensive loads will > be less awesome. > > >From the very vague goals you give above it is not at all clear > to me whether any of this matters or not. > > As to the several and often underrated aspects on which a file > system design can be evaluated, here is more of my incredible > insight :-): > > http://WWW.sabi.co.UK/blog/0804apr.html#080415 > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >