Strahil
2019-Aug-21 20:32 UTC
[Gluster-users] Can I create a distributed dispersed volume withRAM drives?
Yeah, you are right... I forgot that subtle difference. You still need a DR volume in case data is precious . UPS is also nice to have. You should consider sharding and also NUMA pinning. For example if your node has 2 numa nodes and 128 GB per numa node - you should create 2 tmpfs (pinned per numa) of 128GB each.If you create a single 256 GB tmpfs perforamnce will be sporadic. You can google 'SAP HANA fast reboot' for an example. Have you thought about 'replicated dispersed' type of volumes ? What about network - RDMA ? Best Regards, Strahil NikolovOn Aug 21, 2019 13:47, Ewen Chan <alpha754293 at hotmail.com> wrote:> > Strahil: > > Thank you. > > I was doing some research in regards to the difference between ramfs and tmpfs and what others were saying about ramfs is that you can't actually set a ramfs mount point to have a fixed size, so it will keep filling up until you run out of RAM which will either cause the application to crash or it would case it to start swapping heavily. > > This is why I went with tmpfs instead because with tmpfs, you can limit it to a fixed size, so it won't grow indefinitely until you run out of RAM, which can cause the host system to hang. > > Thanks. > > Sincerely, > Ewen > > ________________________________ > From: Strahil <hunter86_bg at yahoo.com> > Sent: August 21, 2019 6:09 AM > To: Ewen <alpha754293 at hotmail.com>; gluster-users <gluster-users at gluster.org> > Subject: Re: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? > ? > In such setup you definately need? a remote DR volume... > > If you create your volumes as ramfs , you should not have issues, but use systemd.mount units - 1. For the ramfs , 2. For glusterd to require? the ramfs mount point. > > Best Regards, > Strahil NikolovOn Aug 20, 2019 06:37, Ewen Chan <alpha754293 at hotmail.com> wrote: > > > > To Whom It May Concern: > > > > I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. > > > > I have created a tmpfs mount point using half of the RAM. > > > > I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. > > > > Can I create a distributed dispersed volume with RAM drives? > > > > I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. > > > > Any help in regards to this matter would be greatly appreciated! > > > > Thank you. > > > > Sincerely, > > > > Ewen Chan > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190821/2778f783/attachment.html>
Ewen Chan
2019-Aug-21 20:43 UTC
[Gluster-users] Can I create a distributed dispersed volume withRAM drives?
Strahil: Good ideas. No, the data isn't really all that precious. It's temporary scratch data for HPC simulations. The systems are on a UPS. A straight dispersed volume results in data transfer rates that's 1/4 of just a distributed volume (~500 MB/s vs 2 GB/s). I'm also already using 4x EDR Infiniband (100 Gbps), with RDMA and direct-io enabled. Thank you. Sincerely, Ewen ________________________________ From: Strahil <hunter86_bg at yahoo.com> Sent: August 21, 2019 4:32 PM To: Ewen <alpha754293 at hotmail.com> Cc: gluster-users <gluster-users at gluster.org> Subject: Re: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? Yeah, you are right... I forgot that subtle difference. You still need a DR volume in case data is precious . UPS is also nice to have. You should consider sharding and also NUMA pinning. For example if your node has 2 numa nodes and 128 GB per numa node - you should create 2 tmpfs (pinned per numa) of 128GB each.If you create a single 256 GB tmpfs perforamnce will be sporadic. You can google 'SAP HANA fast reboot' for an example. Have you thought about 'replicated dispersed' type of volumes ? What about network - RDMA ? Best Regards, Strahil Nikolov On Aug 21, 2019 13:47, Ewen Chan <alpha754293 at hotmail.com> wrote: Strahil: Thank you. I was doing some research in regards to the difference between ramfs and tmpfs and what others were saying about ramfs is that you can't actually set a ramfs mount point to have a fixed size, so it will keep filling up until you run out of RAM which will either cause the application to crash or it would case it to start swapping heavily. This is why I went with tmpfs instead because with tmpfs, you can limit it to a fixed size, so it won't grow indefinitely until you run out of RAM, which can cause the host system to hang. Thanks. Sincerely, Ewen ________________________________ From: Strahil <hunter86_bg at yahoo.com> Sent: August 21, 2019 6:09 AM To: Ewen <alpha754293 at hotmail.com>; gluster-users <gluster-users at gluster.org> Subject: Re: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? In such setup you definately need a remote DR volume... If you create your volumes as ramfs , you should not have issues, but use systemd.mount units - 1. For the ramfs , 2. For glusterd to require the ramfs mount point. Best Regards, Strahil NikolovOn Aug 20, 2019 06:37, Ewen Chan <alpha754293 at hotmail.com> wrote:> > To Whom It May Concern: > > I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. > > I have created a tmpfs mount point using half of the RAM. > > I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. > > Can I create a distributed dispersed volume with RAM drives? > > I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. > > Any help in regards to this matter would be greatly appreciated! > > Thank you. > > Sincerely, > > Ewen Chan > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190821/5f03a30a/attachment.html>