On 09/04/18 19:02, Vincent Royer wrote:> Thanks, > > I suppose what I'm trying to gain is some clarity on what choice is > best for a given application.? How do I know if it's better for me to > use a raid card or not, to include flash-cache on it or not, to use > ZFS or not, when combined with a small number of SSDs in Replica 3. > >How few is "small number" - most importantly number per server? Replica 3 is a start, as it already tells us you can lose at one entire server and carry on as normal. If you lose two, your GlusterFS is down. What is your resilience goal? You should really be starting with requirements, not speccing out and buying servers and drives and then trying to force them to fit your expectations. -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180409/d31afd42/attachment.html>
Thanks, The 3 servers are new Lenovo units with redundant PS backed by two huge UPS units (on for each bank of power supplies). I think the chances of losing two nodes is incredibly slim, and in that case a Disaster Recovery from offsite backups would be reasonable. My requirements are about 2TB, highly available (so that I can reboot one of the 3 servers without taking down services). Beyond that my focus is high performance for small I/O. So I could do a single 2TB SSD per server, or two, or many more if that is "what is required". But I don't want to waste money... I like the idea of forgoing the RAID cards as they are quite expensive, especially the capacitor backed ones. The onboard controller can handle JBOD just fine, if Gluster is OK with it! *Vincent Royer* *778-825-1057* <http://www.epicenergy.ca/> *SUSTAINABLE MOBILE ENERGY SOLUTIONS* On Mon, Apr 9, 2018 at 11:25 AM, Alex Crow <acrow at integrafin.co.uk> wrote:> On 09/04/18 19:02, Vincent Royer wrote: > > Thanks, > > I suppose what I'm trying to gain is some clarity on what choice is best > for a given application. How do I know if it's better for me to use a raid > card or not, to include flash-cache on it or not, to use ZFS or not, when > combined with a small number of SSDs in Replica 3. > > > > How few is "small number" - most importantly number per server? Replica 3 > is a start, as it already tells us you can lose at one entire server and > carry on as normal. If you lose two, your GlusterFS is down. > > What is your resilience goal? You should really be starting with > requirements, not speccing out and buying servers and drives and then > trying to force them to fit your expectations. > > -- > This message is intended only for the addressee and may contain > confidential information. Unless you are that person, you may not > disclose its contents or use it in any way and are requested to delete > the message along with any attachments and notify us immediately. > This email is not intended to, nor should it be taken to, constitute advice. > The information provided is correct to our knowledge & belief and must not > be used as a substitute for obtaining tax, regulatory, investment, legal or > any other appropriate advice. > > "Transact" is operated by Integrated Financial Arrangements Ltd.29 Clement's Lane, London EC4N 7AE <https://maps.google.com/?q=29+Clement's+Lane,+London+EC4N+7AE&entry=gmail&source=g>. Tel: (020) 7608 4900 Fax: (020) 7608 5300. > (Registered office: as above; Registered in England and Wales under > number: 3727592). Authorised and regulated by the Financial Conduct > Authority (entered on the Financial Services Register; no. 190856). > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180409/1b672d5f/attachment.html>
On 09/04/18 22:15, Vincent Royer wrote:> Thanks, > > The 3 servers are new Lenovo units with redundant PS backed by two > huge UPS units (on for each bank of power supplies).? I think the > chances of losing two nodes is incredibly slim, and in that case a > Disaster Recovery from offsite backups would be reasonable. > > My requirements are about 2TB, highly available (so that I can reboot > one of the 3 servers without taking down services). > > Beyond that my focus is high performance for small I/O.This can be a difficult case for GlusterFS, if you mean "small files", as the metadata lookups are relatively costly (no separate MDS with in-memory or memory cached database). It's ideally placed for large files, and small I/O within those files should be OK. just speaking from experience - should be fine for VMs with such loads, especially if you shard.> > So I could do a single 2TB SSD per server, or two, or many more if > that is "what is required".? But I don't want to waste money...Resilience is never a waste. Skimping may well prove to be a waste of *your time* when you get woken up at 3am and have to fix a downed system. Your call entirely. I'm too old for that kind of thing, so I tend to push for both per-server and per-cluster redundancy. It usually gets approved after something "unexpected" happens the first time. Gluster and ZFS will be fine with onboard controllers. If you have enough ports you'll be just fine. If you need more buy HBA's to stick in your PCIe slots, M1015s and M1115s on ebay perform very well and are still dirt cheap. So are you using ZFS to get compression and checksumming down to the disk platter level? ZFS will give some gains in performance with compressible data and corruption protection, but, don't bother with dedup, I've tried it on 3 distributed filesystems and it bought less than 3% capacity and slammed performance. If you don't need either feature just stick with XFS for single-disk or software-RAIDed mirrors per brick. My personal opinion would be do a ZFS mirror of two SSDs per server, per brick, ie in your initial case, 2x2TB SSD per box in ZFS mirror. You can add more mirror sets later to add additional bricks.> > I like the idea of forgoing the RAID cards as they are quite > expensive, especially the capacitor backed ones.? The onboard > controller can handle JBOD just fine, if Gluster is OK with it!As I also said, if said expensive card dies, and you don't have another one in stock, you will effectively have lost everything on that server until you can source a new one (or even /if/ you can). Use the power of the software to get where you need to be, the tools are there... Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).