Hi all; Has anyone ever tried using a Fusion-IO ioDrive as a Lustre MDT? http://www.fusionio.com/products.aspx If so, I''d love to hear your experiences, particularly if you''ve used one in a production filesystem. We''re using a mirrored pair of SATA disks at present for our MDTs, and interactive response can get pretty crummy when our users perform metadata-heavy workloads. It seems this product could help us out in that area. Thanks, Craig Prescott UF HPC Center
Hi Craig, Assuming it presents itself as a regular LUN to the OS, I don''t think it would appear to operate any differently than a physical drive. I''ve always been extremely curious to try solid-state disks in a high-performance environment, so if anyone''s used one of these, I''m dying to hear how it worked for you. cheers, Klaus On 8/8/08 9:31 AM, "Craig Prescott" <prescott at hpc.ufl.edu>did etch on stone tablets:> > Hi all; > > Has anyone ever tried using a Fusion-IO ioDrive as > a Lustre MDT? > > http://www.fusionio.com/products.aspx > > If so, I''d love to hear your experiences, particularly > if you''ve used one in a production filesystem. We''re using > a mirrored pair of SATA disks at present for our MDTs, > and interactive response can get pretty crummy when > our users perform metadata-heavy workloads. It seems > this product could help us out in that area. > > Thanks, > Craig Prescott > UF HPC Center > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
I''ve visited their headquarters, and they''re a very much Linux-savvy organization. As an ext3 or XFS fs, they get 700MB/s out of one ioDrive... with three in a single system MD raided together they were getting 2GB/s out of IOZone atop an XFS fs. It is just presented as a block device. It''s an associative memory, so they have a special fs tailored to using it (and bypass the fs unnecessarily turning names into block addresses). That fs gets 800MB/s under IOZone (with one "disk"). Since the fs layer is somewhat independent in Lustre, I thought that might be a great optimization for this hardware. They probably won''t focus on the HPC market, as they are probably going after bigger fish... so this might be something the community has to do if it''s going to get done. Big issues w/ SSD''s include bottlenecks in controllers meant for spinning drives, and extra device driver layers that do unnecessary steps like elevator code (which SSD or the ioDrive don''t need as seek time is consistent). The ioDrive doesn''t carry or need this extra baggage. I''d like to see some experiment w/ Lustre and ioDrives... especially in the area of minimizing stripe sizes... since these drives don''t need to work w/ big blocks to get optimal performance. Chris On Fri, Aug 8, 2008 at 12:53 PM, Klaus Steden <klaus.steden at thomson.net> wrote:> > Hi Craig, > > Assuming it presents itself as a regular LUN to the OS, I don''t think it > would appear to operate any differently than a physical drive. I''ve always > been extremely curious to try solid-state disks in a high-performance > environment, so if anyone''s used one of these, I''m dying to hear how it > worked for you. > > cheers, > Klaus > > On 8/8/08 9:31 AM, "Craig Prescott" <prescott at hpc.ufl.edu>did etch on stone > tablets: > >> >> Hi all; >> >> Has anyone ever tried using a Fusion-IO ioDrive as >> a Lustre MDT? >> >> http://www.fusionio.com/products.aspx >> >> If so, I''d love to hear your experiences, particularly >> if you''ve used one in a production filesystem. We''re using >> a mirrored pair of SATA disks at present for our MDTs, >> and interactive response can get pretty crummy when >> our users perform metadata-heavy workloads. It seems >> this product could help us out in that area. >> >> Thanks, >> Craig Prescott >> UF HPC Center >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Thanks for the replies and insights, Chris and Klaus! What had attracted us to the ioDrive, particularly in the role of MDT, was the extremely high IOPS figures it claims for small I/Os. With our current mirrored-SATA disk MDTs, we find that when pdflush runs on the MDS during metadata-intense activities, the MDT becomes 100% utilized with tiny I/Os, and await skyrockets while the disks are swamped by all the requests pdflush is pushing to disk. While this is happening, the metadata-intense processes grind to a halt, as do interactive activities like opening files with editors, etc. Everything gets moving again once the pdflush is done, but this can last for tens of seconds, and our users do notice this. We''ve tuned the VM some on our MDS to mitigate this, but we''d like to make this behavior go away completely. If we can. The metadata-intense activities I refer to are usually due to our users managing their data interactively (i.e., removing enormous numbers of dinky files). Some of the applications they run do I/O inefficiently, too (lots of stat()/open()/write(tinyamount)/close()). It seems like a more capable MDT would help out such "bad" jobs, as well. Anyway, that is the source of our interest in the ioDrive as an MDT. During pdflush, we see less than 10MB/s of I/O being done to the MDT; if the ioDrive can do lots more than that at a much higher IOPS rating, it would seem that it would make a very nice MDT. Cheers, Craig Chris Worley wrote:> I''ve visited their headquarters, and they''re a very much Linux-savvy > organization. > > As an ext3 or XFS fs, they get 700MB/s out of one ioDrive... with > three in a single system MD raided together they were getting 2GB/s > out of IOZone atop an XFS fs. It is just presented as a block device. > > It''s an associative memory, so they have a special fs tailored to > using it (and bypass the fs unnecessarily turning names into block > addresses). That fs gets 800MB/s under IOZone (with one "disk"). > Since the fs layer is somewhat independent in Lustre, I thought that > might be a great optimization for this hardware. They probably won''t > focus on the HPC market, as they are probably going after bigger > fish... so this might be something the community has to do if it''s > going to get done. > > Big issues w/ SSD''s include bottlenecks in controllers meant for > spinning drives, and extra device driver layers that do unnecessary > steps like elevator code (which SSD or the ioDrive don''t need as seek > time is consistent). The ioDrive doesn''t carry or need this extra > baggage. > > I''d like to see some experiment w/ Lustre and ioDrives... especially > in the area of minimizing stripe sizes... since these drives don''t > need to work w/ big blocks to get optimal performance. > > Chris > On Fri, Aug 8, 2008 at 12:53 PM, Klaus Steden <klaus.steden at thomson.net> wrote: >> Hi Craig, >> >> Assuming it presents itself as a regular LUN to the OS, I don''t think it >> would appear to operate any differently than a physical drive. I''ve always >> been extremely curious to try solid-state disks in a high-performance >> environment, so if anyone''s used one of these, I''m dying to hear how it >> worked for you. >> >> cheers, >> Klaus >> >> On 8/8/08 9:31 AM, "Craig Prescott" <prescott at hpc.ufl.edu>did etch on stone >> tablets: >> >>> Hi all; >>> >>> Has anyone ever tried using a Fusion-IO ioDrive as >>> a Lustre MDT? >>> >>> http://www.fusionio.com/products.aspx >>> >>> If so, I''d love to hear your experiences, particularly >>> if you''ve used one in a production filesystem. We''re using >>> a mirrored pair of SATA disks at present for our MDTs, >>> and interactive response can get pretty crummy when >>> our users perform metadata-heavy workloads. It seems >>> this product could help us out in that area. >>> >>> Thanks, >>> Craig Prescott >>> UF HPC Center >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>