Lustre discuss - Oct 2007 - Hardware/System Configuration Recommendation

On Oct 08, 2007  07:58 -0500, anhvu.q.le at exxonmobil.com
wrote:
> We have about 150TB of disk (in 50 luns) hanging off 4 HP DL380, each with
> 8.0GB of RAM serving 12 OSTs.  According to your calculation, 12 * 400MB
> 4.8GB of RAM being pinned just by the ext3 journal, then how much more RAM
> you recommend to be adequate?
This is the worst case scenario, but eventually that worst case happens...
As a rough estimate, 1GB/OST would be a good amount of RAM, and aligns
fairly well with what other customers are installing.  If you are running
in failover mode, expect 2x that amount.
> Please elaborate in shrinking the journal
> size case since I''m not familiar with it.
When the OST is cleanly unmounted, for each device:

	tune2fs -O ^has_journal {dev}
	tune2fs -J size=128 {dev}

where 128 = size of journal in MB.  You might want to benchmark that
under a heavy load - you could also use 256MB journals, but the amount
you need depends on the number of concurrent client IOs.
> On Oct 05, 2007  11:02 -0600, Daniel Leaberry wrote:
> > Aaron Knister wrote:
> > > I''m planning to hang 58 terabytes off of a PowerEdge
1950 with 4 CPUS
> > > and 8 gigs of memory. My MDS is a dual core Opteron with a 250gig
> > > raid1 metadata volume and 2GB of ram. Do you think this hardware
> > > configuration is sane?
> >
> > We have about 85TB of disk (in 24 luns) hanging off 4
PE2950''s with
> > those same specs. They are set up in failover pairs (each handles 6
> > luns) but I can''t run too long on a single machine before it
starts
> > thrashing when it takes over the other nodes 6 luns.
> 
> If you have 12 OSTs on a single node, that means up to 12 * 400MB = 4.8GB
> of RAM pinned just by the ext3 journal.  Either you need a lot more RAM
> than this (8TB for example), or you need to shrink the journal size like
> 128MB (tune2fs to remove then re-add it).  Using 128MB should be fine
> unless you have many hundreds of clients doing concurrent IO.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.