Hi, I am using 26x8TB disks per server. There are 60 servers in gluster cluster.
Each disk is a brick and configuration is 16+4 EC, 9PB single volume.
Clients are using fuse mounts.
Even with 1-2K files in a directory, ls from clients takes ~60 secs.
So If you are sensitive to metadata operations, I suggest another approach...
On Wed, Nov 9, 2016 at 1:05 PM, Frank Rothenstein
<f.rothenstein at bodden-kliniken.de> wrote:> As you said you want to have 3 or 4 replicas, so i would use the zfs
> knowledge and build 1 zpool per node with whatever config you know is
> fastest on this kind of hardware and as safe as you need (stripe,
> mirror, raidz1..3 - resilvering zfs is faster than healing gluster, I
> think) . 1 node -> 1 brick (per gluster volume).
>
> Frank
> Am Dienstag, den 08.11.2016, 19:19 +0000 schrieb Thomas Wakefield:
>> We haven?t decided how the JBODS would be configured. They would
>> likely be SAS attached without a raid controller for improved
>> performance. I run large ZFS arrays this way, but only in single
>> server NFS setups right now.
>> Mounting each hard drive as it?s own brick would probably give the
>> most usable space, but would need scripting to manage building all
>> the bricks. But does Gluster handle 1000?s of small bricks?
>>
>>
>>
>> > On Nov 8, 2016, at 9:18 AM, Frank Rothenstein <f.rothenstein at
bodden
>> > -kliniken.de> wrote:
>> >
>> > Hi Thomas,
>> >
>> > thats a huge storage.
>> > What I can say from my usecase - dont use Gluster directly if the
>> > files
>> > are small. I dont know, if the file count matters, but if the
files
>> > are
>> > small (few KiB), Gluster takes ages to remove for example. Doing
>> > the
>> > same in a VM with e.g. ext4 disk on the very same Gluster gives a
>> > big
>> > speedup.
>> > There are many options for a new Gluster volume, like Lindsay
>> > mentioned.
>> > And there are other options, like Ceph, OrangeFS.
>> > How do you want to use the JBODs? I dont think you would use every
>> > single drive as a brick... How are these connected to the servers?
>> >
>> > Im only dealing with about 10TiB Gluster volumes, so by far not at
>> > your
>> > planned level, but I really would like to see some results, if you
>> > go
>> > for Gluster!
>> >
>> > Frank
>> >
>> >
>> > Am Dienstag, den 08.11.2016, 13:49 +0000 schrieb Thomas Wakefield:
>> > > I think we are leaning towards erasure coding with 3 or 4
>> > > copies. But open to suggestions.
>> > >
>> > >
>> > > > On Nov 8, 2016, at 8:43 AM, Lindsay Mathieson
<lindsay.mathieso
>> > > > n at gm
>> > > > ail.com> wrote:
>> > > >
>> > > > On 8/11/2016 11:38 PM, Thomas Wakefield wrote:
>> > > > > High Performance Computing, we have a small cluster
on campus
>> > > > > of
>> > > > > about 50 linux compute servers.
>> > > > >
>> > > >
>> > > > D'oh! I should have thought of that.
>> > > >
>> > > >
>> > > > Are you looking at replication (2 or 3)/disperse or pure
>> > > > disperse?
>> > > >
>> > > > --
>> > > > Lindsay Mathieson
>> > > >
>> > >
>> > > _______________________________________________
>> > > Gluster-users mailing list
>> > > Gluster-users at gluster.org
>> > > http://www.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>> >
>> >
>> >
>> >
___________________________________________________________________
>> > ___________
>> > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
>> > Sandhufe 2
>> > 18311 Ribnitz-Damgarten
>> >
>> > Telefon: 03821-700-0
>> > Fax: 03821-700-240
>> >
>> > E-Mail: info at bodden-kliniken.de Internet:
http://www.bodden-klini
>> > ken.de
>> >
>> > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-
>> > Nr.: 079/133/40188
>> > Aufsichtsratsvorsitzende: Carmen Schr?ter, Gesch?ftsf?hrer: Dr.
>> > Falko Milski
>> >
>> > Der Inhalt dieser E-Mail ist ausschlie?lich f?r den bezeichneten
>> > Adressaten bestimmt. Wenn Sie nicht der vorge-
>> > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten,
>> > beachten Sie bitte, dass jede Form der Ver?f-
>> > fentlichung, Vervielf?ltigung oder Weitergabe des Inhalts dieser
E-
>> > Mail unzul?ssig ist. Wir bitten Sie, sofort den
>> > Absender zu informieren und die E-Mail zu l?schen.
>> >
>> >
>> > Bodden-Kliniken Ribnitz-Damgarten GmbH 2016
>> > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
>> >
>>
>>
>
>
>
>
>
>
______________________________________________________________________________
> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
> Sandhufe 2
> 18311 Ribnitz-Damgarten
>
> Telefon: 03821-700-0
> Fax: 03821-700-240
>
> E-Mail: info at bodden-kliniken.de Internet:
http://www.bodden-kliniken.de
>
> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.:
079/133/40188
> Aufsichtsratsvorsitzende: Carmen Schr?ter, Gesch?ftsf?hrer: Dr. Falko
Milski
>
> Der Inhalt dieser E-Mail ist ausschlie?lich f?r den bezeichneten Adressaten
bestimmt. Wenn Sie nicht der vorge-
> sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten
Sie bitte, dass jede Form der Ver?f-
> fentlichung, Vervielf?ltigung oder Weitergabe des Inhalts dieser E-Mail
unzul?ssig ist. Wir bitten Sie, sofort den
> Absender zu informieren und die E-Mail zu l?schen.
>
>
> Bodden-Kliniken Ribnitz-Damgarten GmbH 2016
> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users