Lustre discuss - Mar 2009 - Fwd: Simple servers as storage nodes

Hi.

Can anyone tell if regular servers can be used as the storage nodes?

Or any other solution (SAN/NAS) has to be used as the storage space?

Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090326/32ab9a8b/attachment.html

On Thu, 2009-03-26 at 18:58 +0200, Stas Oskin wrote:
> Hi.
> 
> Can anyone tell if regular servers can be used as the storage nodes?
A whitebox PC with an IDE disk in it *could* be a Lustre server, yes.
It won''t be a well performing server though and it won''t enjoy
the
benefits of Lustre''s failover abilities.
> Or any other solution (SAN/NAS) has to be used as the storage space?
Shared storage enables a failover configuration to be built.  Non-shared
storage doesn''t.

b.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090326/fcd49a30/attachment.bin

>> Can anyone tell if regular servers can be used as the storage
>> nodes?
> A whitebox PC with an IDE disk in it *could* be a Lustre
> server, yes.  It won''t be a well performing server though
I''d think that Sun X4500s (Thumpers) would make for very good
Lustre servers. But my boss went for DDN 9000s instead, too bad.
> and it won''t enjoy the benefits of Lustre''s failover
abilities.
>> Or any other solution (SAN/NAS) has to be used as the storage
>> space?
NAS is obviously inappropriate for Lustre storage :-).
> Shared storage enables a failover configuration to be built.
> Non-shared storage doesn''t.
I always ask myself why so many people think that OSS and path
failover is what matters. To me what matters is disk failover,
and having redundant frontends and paths to shared disks is
nowhere as useful as having redundant disk subsystems. Shared
storage is a problem, not a solution.

Given that sensible people use RAID10 (http://WWW.BAARF.com/)
anyhow having two DRBD-mirrored fully replicated storage systems
seems wonderful to me.

Lustre, Thumpers, RAID10 over DRBD seems the winning combination
to me for for large/fast/reliable/cost effective storage pools,
with fairly few exceptions. Consider this:

  http://www.mail-archive.com/lustre-discuss%40lists.lustre.org/msg02296.html

Hi.

Thanks for the advice everyone.

What I''m actually look for, is a solution that can take plain Linux
boxes,
and unify their space into a single volume, with optional replication of
every file.

So my question is, whether Lustre able to do so without any special
hardware.

Regard.

2009/3/29 Peter Grandi <pg_lus at lus.for.sabi.co.uk>
> >> Can anyone tell if regular servers can be used as the storage
> >> nodes?
>
> > A whitebox PC with an IDE disk in it *could* be a Lustre
> > server, yes.  It won''t be a well performing server though
>
> I''d think that Sun X4500s (Thumpers) would make for very good
> Lustre servers. But my boss went for DDN 9000s instead, too bad.
>
> > and it won''t enjoy the benefits of Lustre''s failover
abilities.
>
> >> Or any other solution (SAN/NAS) has to be used as the storage
> >> space?
>
> NAS is obviously inappropriate for Lustre storage :-).
>
> > Shared storage enables a failover configuration to be built.
> > Non-shared storage doesn''t.
>
> I always ask myself why so many people think that OSS and path
> failover is what matters. To me what matters is disk failover,
> and having redundant frontends and paths to shared disks is
> nowhere as useful as having redundant disk subsystems. Shared
> storage is a problem, not a solution.
>
> Given that sensible people use RAID10 (http://WWW.BAARF.com/)
> anyhow having two DRBD-mirrored fully replicated storage systems
> seems wonderful to me.
>
> Lustre, Thumpers, RAID10 over DRBD seems the winning combination
> to me for for large/fast/reliable/cost effective storage pools,
> with fairly few exceptions. Consider this:
>
>
> http://www.mail-archive.com/lustre-discuss%40lists.lustre.org/msg02296.html
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090330/1f0f26ad/attachment-0001.html

On Mon, 2009-03-30 at 13:43 +0300, Stas Oskin wrote:
> 
> What I''m actually look for, is a solution that can take plain
Linux
> boxes, and unify their space into a single volume, with optional
> replication of every file.
You will need to look elsewhere then.
> So my question is, whether Lustre able to do so without any special
> hardware.
No.  What you describe is not Lustre''s solution space.

b.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090330/9c0019d8/attachment.bin

Hi.

Are you aware of any other solutions that might achieve this?

Regards,

2009/3/30 Brian J. Murrell <Brian.Murrell at sun.com>
> On Mon, 2009-03-30 at 13:43 +0300, Stas Oskin wrote:
> >
> > What I''m actually look for, is a solution that can take plain
Linux
> > boxes, and unify their space into a single volume, with optional
> > replication of every file.
>
> You will need to look elsewhere then.
>
> > So my question is, whether Lustre able to do so without any special
> > hardware.
>
> No.  What you describe is not Lustre''s solution space.
>
> b.
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090330/e608912c/attachment.html

Hi.  It was pointed out that perhaps I was misunderstanding your
question/situation.

On Mon, 2009-03-30 at 09:16 -0400, Brian J. Murrell
wrote:
> On Mon, 2009-03-30 at 13:43 +0300, Stas Oskin wrote:
> > 
> > What I''m actually look for, is a solution that can take plain
Linux
> > boxes, and unify their space into a single volume, with optional
> > replication of every file.
So these Linux boxes you have, do they have a task already or are you
going to dedicate them to the job of serving their disk space out in a
uniform volume?  Do you have any expectations that these Linux boxes
will also be able to access that unified volume or will there be other
Linux machines accessing the unified volume?

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090330/70604ae4/attachment.bin

Hi.
>So these Linux boxes you have, do they have a task already or are you
>going to dedicate them to the job of serving their disk space out in a
>uniform volume?
Actually I''d prefer an approach where these boxes do other tasks, but
depends on the solution.
>Do you have any expectations that these Linux boxes
>will also be able to access that unified volume or will there be other
>Linux machines accessing the unified volume?
Both would be great.

Regards.

2009/3/30 Brian J. Murrell <Brian.Murrell at sun.com>
> Hi.  It was pointed out that perhaps I was misunderstanding your
> question/situation.
>
> On Mon, 2009-03-30 at 09:16 -0400, Brian J. Murrell wrote:
> > On Mon, 2009-03-30 at 13:43 +0300, Stas Oskin wrote:
> > >
> > > What I''m actually look for, is a solution that can take
plain Linux
> > > boxes, and unify their space into a single volume, with optional
> > > replication of every file.
>
> So these Linux boxes you have, do they have a task already or are you
> going to dedicate them to the job of serving their disk space out in a
> uniform volume?  Do you have any expectations that these Linux boxes
> will also be able to access that unified volume or will there be other
> Linux machines accessing the unified volume?
>
> b.
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090330/fb3e0a28/attachment.html

On Mon, 2009-03-30 at 19:15 +0300, Stas Oskin wrote:
> Hi.
Hello,
> Actually I''d prefer an approach where these boxes do other tasks,
but
> depends on the solution.
For a Lustre solution, you should dedicate the Linux boxes to sharing
their storage out.  For stability and performance, these machines should
not do any other tasks other than being a Lustre server.
> >Do you have any expectations that these Linux boxes
> >will also be able to access that unified volume or will there be
> other
> >Linux machines accessing the unified volume?
> 
> Both would be great.
Putting Lustre clients on the servers can trigger memory pressure
deadlocks in the situation where the client needs to flush (i.e. to the
server on the same node) some pages (again, due to memory pressure) yet
the server on the same node needs to allocate pages to receive the
client''s pages -- it will fail for the same reason the client is trying
to free pages.  For this reason we suggest against putting clients and
OSTs on the same machine.

Cheers,
b.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090330/721fac1e/attachment.bin

[ ... ]
> What I''m actually look for, is a solution that can take plain
> Linux boxes, and unify their space into a single volume, with
> optional replication of every file.
Well, this quest is rather vague -- "unify their space" does
that means the data space or the name space? Does the "optional
replication of every file" mean block level or file level
replication?

Lustre as to those unifies name spaces, sort of unifies data
space when striping (on a rather coarse level), and does no
replication, leaving it to the underlying storage (sw or hw)
layer, and several people use MD/DM RAID1 or DRBD to do that.

I have done a summary here:

  http://www.sabi.co.uk/blog/0804apr.html#080417

And this is a nice presentation that I used to push it within my
organization (even if the final achitecture was quite different):

  http://HEPiX.CASPUR.IT/storage/hep_pdf/2008/Spring/LustreClusterGSI.pdf

and I think that the "Even Cheaper" alternative they describe
is fully "generic box" based.
> So my question is, whether Lustre able to do so without any
> special hardware.
Well, Lustre works pretty well without special hardware (but
what does "special" mean?).

Conversely, if one wants a distributed/parallel/chunked file
system there is not much else in the "production ready" and
"free software" areas. I have read good things about GlusterFS
as a possible alternative with a somewhat different profile
though, and I am going to look into it.