Γιώργος Βασιλόπουλος
2018-Jul-25 09:11 UTC
[Gluster-users] trying to figure out the best solution for vm and email storage
Hello I am trying to lay down my options regarding storage with glusterfs, vm storage and email storage Hardware in my disposal is specific : I have 9 servers for running vm's under ovirt 4.2 and 3 servers for storage The 3 storage machines are similar and each have 2xE5-2640v3 cpus 128GB RAM and 2x10G ethernet each storage server has inside 2x300gb 10k drives which I intent to use as os install and maybe a litle volume for isos on nfs also present are 6x200GB SSD drives which I think of using as tiering volume(s) And the main storage is on an external JBOD box with 12x4TB drives connected via SAS to the server with RAID controller capable of varius raid levels. So what I'm thinking is that I will create 3 replica 3 arbiter 1 volumes (high availability is the no 1 requirement) in a cyclic fashion, 2 data bricks and one arbiter on each storage server I will implement raid6 with one spare drive on each server (we are in an island and getting a disk replacement can take days occasionaly) which will give me about 36T of usable storage per server. So regarding the vm storage I am thinking that 2 volumes with 17TB each and an arbiter of 1TB. What messes things up is that I was required to put the email storage in this installation. Our email is pretty big and busy with about 50000 users curently at 13T of storage. Currently it runs on a few vm's and uses storage from nfs given from another vm. It is runs postfix/dovecot and right now a single big vm does? mailbox delivery but this reaches it's limits. Mail storage now is on a EMC VNX5500 But it will be moved to glusterfs for various reasons. I would like some advise regarding the email storage. I think my options are 1a. use a VM as NFS give it a huge disk (raw image on gluster vm optimized) and be done with it 1b use a VM as NFS give it a 2 or 3 disks unified under lvm vg->lv (raw images on gluster vm optimized) and maybe take some advantage of using 2-3 io-threads in o virt to write to 2-3 disks simultaneously. Will this give extra performance ? 2.Give gluster as nfs straight to dovecot but I wonder if this will have performance drawback since it will be fuse mounted. I am also worried about the arbiter volume since there will be thousands of small files, practically arbiter will probably have to be as large as the data bricks or half that size 3. Give gluster as glusterfs mount point which I think will have about the same issues as 2. I have read about problems with dovecot indexes and glusterfs. Is this still an issue? or is it a problem that only shows when there is no dovecot director. Personaly I am inclined on using solution 1 because I think that arbiter volumes will be smaller (Am I right?) though it may have some overhead regarding nfs on the vm. On the other hand this solution will use libgfapi which might balance things a bit. Will it help if in such a case use small (16mb) shard size and tiering ? I'm afraid I have it a bit mixed up in my mind and I could really use some help. -- ???????????? ??????? ???????????? ????????? ?.?. ???????????? ????. ?????????? ???????????? ?????? ?.?.?.?.?.?. ????? ???????????? ??? ??????? ?????? ????????? 70013 ??? : 2810393310 email : g.vasilopoulos at uoc.gr http://www.ucnet.uoc.gr
Vlad Kopylov
2018-Jul-25 23:00 UTC
[Gluster-users] trying to figure out the best solution for vm and email storage
Just create one 3 replica volume with 1 brick on each of 3 storage servers. Raid5 for servers will be more then enough - it is already replica 3. Use ovirt to mount glusterfs to VM from hosts (as it uses libgfapi) rather then fuse mount from VM itself. libgfapi is supposedly faster. Might depend on which mail storage type you use - if maildir, libgfapi to VM on he host should be better. Also mid that you might need compression for mail storage and deduplication (of attachments) On Wed, Jul 25, 2018 at 5:11 AM, ??????? ???????????? <g.vasilopoulos at uoc.gr> wrote:> Hello > I am trying to lay down my options regarding storage with glusterfs, vm > storage and email storage > > Hardware in my disposal is specific : I have 9 servers for running vm's > under ovirt 4.2 and 3 servers for storage > > The 3 storage machines are similar and each have 2xE5-2640v3 cpus 128GB > RAM and 2x10G ethernet > each storage server has inside 2x300gb 10k drives which I intent to use as > os install and maybe a litle volume for isos on nfs > also present are 6x200GB SSD drives which I think of using as tiering > volume(s) > And the main storage is on an external JBOD box with 12x4TB drives > connected via SAS to the server with RAID controller capable of varius raid > levels. > > So what I'm thinking is that I will create 3 replica 3 arbiter 1 volumes > (high availability is the no 1 requirement) in a cyclic fashion, 2 data > bricks and one arbiter on each storage server > I will implement raid6 with one spare drive on each server (we are in an > island and getting a disk replacement can take days occasionaly) which will > give me about 36T of usable storage per server. > So regarding the vm storage I am thinking that 2 volumes with 17TB each > and an arbiter of 1TB. > What messes things up is that I was required to put the email storage in > this installation. Our email is pretty big and busy with about 50000 users > curently at 13T of storage. > Currently it runs on a few vm's and uses storage from nfs given from > another vm. It is runs postfix/dovecot and right now a single big vm does > mailbox delivery but this reaches it's limits. Mail storage now is on a EMC > VNX5500 > But it will be moved to glusterfs for various reasons. > > I would like some advise regarding the email storage. I think my options > are > > 1a. use a VM as NFS give it a huge disk (raw image on gluster vm > optimized) and be done with it > 1b use a VM as NFS give it a 2 or 3 disks unified under lvm vg->lv (raw > images on gluster vm optimized) and maybe take some advantage of using 2-3 > io-threads in o virt to write to 2-3 disks simultaneously. Will this give > extra performance ? > > 2.Give gluster as nfs straight to dovecot but I wonder if this will have > performance drawback since it will be fuse mounted. I am also worried about > the arbiter volume since there will be thousands of small files, > practically arbiter > will probably have to be as large as the data bricks or half that size > > 3. Give gluster as glusterfs mount point which I think will have about the > same issues as 2. > > I have read about problems with dovecot indexes and glusterfs. Is this > still an issue? or is it a problem that only shows when there is no dovecot > director. > Personaly I am inclined on using solution 1 because I think that arbiter > volumes will be smaller (Am I right?) though it may have some overhead > regarding nfs on the vm. On the other hand this solution will use libgfapi > which might balance things a bit. > Will it help if in such a case use small (16mb) shard size and tiering ? > > I'm afraid I have it a bit mixed up in my mind and I could really use some > help. > > > > -- > ???????????? ??????? > ???????????? ????????? ?.?. > ???????????? ????. ?????????? > > ???????????? ?????? > ?.?.?.?.?.?. > ????? ???????????? ??? ??????? > ?????? ????????? 70013 > ??? : 2810393310 > email : g.vasilopoulos at uoc.gr > http://www.ucnet.uoc.gr > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180725/1254ff8e/attachment.html>