My supervisor has this idea and I would like the input of the Lustre community as we are still very new to Lustre. We have 7 workstations, and the idea was to put into them 3 2TB drives, for a total of 42TB, and set them up as object servers, and another workstation as a meta data server. How feasible is this idea? And I know of the downfalls, what if a student reboots a machine, etc, but baring those events, would this setup work? David -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/91228e4e/attachment.html
On Wed, 2010-05-19 at 12:31 -0500, David Noriega wrote:> > We have 7 workstations, and the idea was to put into them 3 2TB > drives, for a total of 42TB, and set them up as object servers, and > another workstation as a meta data server.And what are the clients of this fileystem? So far you have only specified the servers. What machines will actually use this filesystem?> How feasible is this idea?Well, technically it''s possible as long as you can run a Lustre patched kernel on those workstations (which might contradict the requirements of the workstation users).> And I know of the downfalls, what if a student reboots a machine, etc, > but baring those events, would this setup work?What about the performance impact of Lustre on those students'' workstations? Are they going to be happy sharing the resources of their workstations with Lustre object storage services? Generally we advise against doing anything on a Lustre server except run Lustre. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/95d1ac1a/attachment.bin
The idea is to have an extra place to backup our file server. To run incremental backups on the weekends. Have a script to check if the workstations are online first. Eventually the main fileserver will be Lustre, but this would be a duplicate of the fileserver. So our services would not be depending on this. As it stands, the machines are hardly used, and a modified kernel is fine. On the occasion they are used, students tend to just check email, maybe matlab, but more often then not when using matlab its remotely off of a compute server. Or would it be better to use some other filesystem. What about what comes with Redhat/CentOS in their cluster packages? On Wed, May 19, 2010 at 12:47 PM, Brian J. Murrell <Brian.Murrell at oracle.com> wrote:> On Wed, 2010-05-19 at 12:31 -0500, David Noriega wrote: > > > > We have 7 workstations, and the idea was to put into them 3 2TB > > drives, for a total of 42TB, and set them up as object servers, and > > another workstation as a meta data server. > > And what are the clients of this fileystem? So far you have only > specified the servers. What machines will actually use this filesystem? > > > How feasible is this idea? > > Well, technically it''s possible as long as you can run a Lustre patched > kernel on those workstations (which might contradict the requirements of > the workstation users). > > > And I know of the downfalls, what if a student reboots a machine, etc, > > but baring those events, would this setup work? > > What about the performance impact of Lustre on those students'' > workstations? Are they going to be happy sharing the resources of their > workstations with Lustre object storage services? > > Generally we advise against doing anything on a Lustre server except run > Lustre. > > b. > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >-- Personally, I liked the university. They gave us money and facilities, we didn''t have to produce anything! You''ve never been out of college! You don''t know what it''s like out there! I''ve worked in the private sector. They expect results. -Ray Ghostbusters -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/2c7943ab/attachment.html
as playground is fine. even in this type of env , IMHO, u will need to do some raid for 3x2tb ost and mirror for mds Hope that U have enough memory and cpu power and network for MDS and OST. these are dedicate for lustre, u need other compute nodes. regards On 5/19/2010 1:31 PM, David Noriega wrote:> My supervisor has this idea and I would like the input of the > Lustre community as we are still very new to Lustre. > > We have 7 workstations, and the idea was to put into them 3 2TB > drives, for a total of 42TB, and set them up as object servers, and > another workstation as a meta data server. How feasible is this idea? > And I know of the downfalls, what if a student reboots a machine, etc, > but baring those events, would this setup work? > > David > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- Oracle Hung-Sheng Tsao, Ph.D. | Principal Sales Consultant Higher Education | +1.973.495.0840 Oracle * - North America Commercial Hardware* 400 Atrium Dr. Somerset, NJ 08873 Email Hungsheng.tsao at oracle.com <mailto:Hungsheng.Tsao at oracle.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/0e187b9c/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: ora_logo_small.gif Type: image/gif Size: 2059 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/0e187b9c/attachment.gif
I just had another idea but again since I know very little about how Lustre works, I''ll need some input. What if I take a single workstation and attach to it via iSCSI two Drobo disk arrays. Would it be possible to run both the metadata and the object storage off of a single machine? Two maxed out Drobo arrays gets 32TB of space. It costs more but would this be better then adding disks to existing workstations where we cant control the environment(ie users)? On Wed, May 19, 2010 at 1:01 PM, hungsheng Tsao <hungsheng.tsao at oracle.com>wrote:> as playground is fine. > even in this type of env , IMHO, u will need to do some raid for 3x2tb ost > and mirror for mds > Hope that U have enough memory and cpu power and network for MDS and OST. > these are dedicate for lustre, u need other compute nodes. > regards > > > > On 5/19/2010 1:31 PM, David Noriega wrote: > > My supervisor has this idea and I would like the input of the > Lustre community as we are still very new to Lustre. > > We have 7 workstations, and the idea was to put into them 3 2TB drives, > for a total of 42TB, and set them up as object servers, and another > workstation as a meta data server. How feasible is this idea? And I know of > the downfalls, what if a student reboots a machine, etc, but baring those > events, would this setup work? > > David > > ------------------------------ > > _______________________________________________ > Lustre-discuss mailing listLustre-discuss at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss > > > -- > [image: Oracle] > Hung-Sheng Tsao, Ph.D. | Principal Sales Consultant > Higher Education | +1.973.495.0840 > Oracle * - North America Commercial Hardware* > 400 Atrium Dr. Somerset, NJ 08873 > Email Hungsheng.tsao at oracle.com <Hungsheng.Tsao at oracle.com> > >-- Personally, I liked the university. They gave us money and facilities, we didn''t have to produce anything! You''ve never been out of college! You don''t know what it''s like out there! I''ve worked in the private sector. They expect results. -Ray Ghostbusters -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/ad303986/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 2059 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/ad303986/attachment.gif
On 2010-05-19, at 12:44, David Noriega wrote:> What if I take a single workstation and attach to it via iSCSI two Drobo disk arrays. Would it be possible to run both the metadata and the object storage off of a single machine? Two maxed out Drobo arrays gets 32TB of space. It costs more but would this be better then adding disks to existing workstations where we cant control the environment(ie users)?I run in this config for my home system all the time, though it has relatively low load. If you aren''t expecting the maximum performance out of the filesystem, then you can just use any old components to get enough space. For the extra $500 or so to add a dedicated server node (compared to $3000 for the Drobo enclosures + $2500 for the disks) you may as well avoid sharing this resource. However, the question arises whether you need Lustre at all, if you are just putting it on a single server? Nobody can accuse me of being anti-Lustre :-), but it definitely has more complexity than just using a single large filesystem (e.g. XFS) and NFS exporting it, or using rsync + ssh for your backups and not having any network exporting of the filesystem, if that meets your needs just as well. Lustre is best suited for the case where the performance/space requirements are larger than what a single server can provide, and I don''t think that matches your use case very well.> On Wed, May 19, 2010 at 1:01 PM, hungsheng Tsao <hungsheng.tsao at oracle.com> wrote: > as playground is fine. > even in this type of env , IMHO, u will need to do some raid for 3x2tb ost and mirror for mds > Hope that U have enough memory and cpu power and network for MDS and OST. > these are dedicate for lustre, u need other compute nodes. > regards > > > > On 5/19/2010 1:31 PM, David Noriega wrote: >> My supervisor has this idea and I would like the input of the Lustre community as we are still very new to Lustre. >> >> We have 7 workstations, and the idea was to put into them 3 2TB drives, for a total of 42TB, and set them up as object servers, and another workstation as a meta data server. How feasible is this idea? And I know of the downfalls, what if a student reboots a machine, etc, but baring those events, would this setup work? >> >> David >> >> _______________________________________________ >> Lustre-discuss mailing list >> >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >> > > -- > <ora_logo_small.gif> > Hung-Sheng Tsao, Ph.D. | Principal Sales Consultant > Higher Education | +1.973.495.0840 > Oracle - North America Commercial Hardware > 400 Atrium Dr. Somerset, NJ 08873 > Email Hungsheng.tsao at oracle.com > > > > > -- > Personally, I liked the university. They gave us money and facilities, we didn''t have to produce anything! You''ve never been out of college! You don''t know what it''s like out there! I''ve worked in the private sector. They expect results. -Ray Ghostbusters > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
On Wed, May 19, 2010 at 12:56:49PM -0500, David Noriega wrote:> The idea is to have an extra place to backup our file server. To run > incremental backups on the weekends. Have a script to check if the > workstations are online first. Eventually the main fileserver will be > Lustre, but this would be a duplicate of the fileserver. So our services > would not be depending on this.I strongly recommend not building infrastructure services on systems whose physical security you cannot safeguard. Your students could do much worse than merely reboot these workstations... If you care about the data, even as a backup (surely you care about having a good backup when you need it), then should not proceed with your plan. On the other hand, as a giant scratch space (e.g., for workspaces) that you don''t intend to backup/restore, running a Lustre cluster on those workstations would be just fine. It''d also be fine to share scratch space local storage via NFS and forget clustering. Nico --
[ ... ]>> What if I take a single workstation and attach to it via >> iSCSI two Drobo disk arrays. Would it be possible to run both >> the metadata and the object storage off of a single machine? >> [ ... for backups ... ]> [ ... ] However, the question arises whether you need Lustre > at all, if you are just putting it on a single server? Nobody > can accuse me of being anti-Lustre :-), but it definitely has > more complexity than just using a single large filesystem > (e.g. XFS) and NFS exporting it, or using rsync + ssh for your > backups and not having any network exporting of the > filesystem, if that meets your needs just as well.Ahhh that is really underselling Lustre. Sure, the main claim to fame is the spread-across-network angle, but there are some good reasons to use Lustre as a replacement for something like an ext4+NFS setup (and not necessarily in the case above, these are general points): #1 ''fsck'' times. Lustre allows splitting a large namespace onto multiple filesystems that can, if possible, be checked independently. A lot of people underestimate the issue of ''fsck'' scalability, and using a metafilesystem is a palliative. #2 network protocol. Lustre as a network filesystem it has a more POSIX-like protocol, which is arguably an advantage on the NFS network protocol (even if NFSv4 has the single port advantage). #3 quality of implementation. Regrettably the Linux NFS client is not that good, especially for writes. The Linux NFS server is good, but the Lustre server is perhaps a bit better, for example for recovery (if one has multiple servers for resilience as opposed to parallelism). Sure things like info/error message quality is abysmal, but then that''s pretty common (e.g. Kerberos5). #4 LNET. In some cases having LNET is quite useful regardless, as it runs transparently over several different network types, and allows some interesting forms of recovery. Some of these advantages (in particular #1) are compelling enough to suggest using Lustre as *local* filesystem, too bad that there are potential problems with that. If it weren''t for the latter I could well imagine having a fast SMP machine using Lustre locally for computations requiring particularly high bandwidth and low latency, and exporting it over the network for remote access when convenient. There is no real alternative between RSYNC-SSH and Lustre either for backups because: #1 SSH is very expensive and it is a very poor network protocol for data transfer, even if convenient. #2 One can very well anyhow use RSYNC+SSH to backup, and then give access to the backup over Lustre. As a rule I configure storage servers so that one can access the same data over several procotols, as in (with some poetic license): nfs://srv.example.com/mnt/fsys3/ ssh://srv.example.com/mnt/fsys3/ lustre://srv.example.com/mnt/fsys3/ smb://srv.example.com/mnt/fsys3/ rsync://srv.example.com/mnt/fsys3/ https://srv.example.com/mnt/fsys3/ ftps://srv.example.com/mnt/fsys3/ webdavs://srv.example.com/mnt/fsys3/> Lustre is best suited for the case where the performance/space > requirements are larger than what a single server can provide, > and I don''t think that matches your use case very well.Sure, parallelism is the big deal (despite the 1.x series lack of MDS scalability), but even in small implementations I''d often suggest a Lustre server or a resilient pair of Lustre servers rather than an ext4+NFS one, especially if the storage space is big and cannot be split logically (it has to be a single namespace), if only for the ''fsck'' benefits. The biggest limitation with Lustre is that it has been based for what I think are mostly political/commercial reasons on ext3 and ext4, instead on JFS (or XFS). But that''s not just for Lustre, most of the Linux world made that big mistake. But currently as (free sw) network file system go Lustre is probably without alternative (even if GlusterFS or Ceph might be quite interesting), even outside the parallel/cluster case.