Aliet Santiesteban Sifontes
2011-May-05 23:55 UTC
[Dovecot] Dovecot imaptest on RHEL4/GFS1, RHEL6/GFS2, NFS and local storage results
We have done some benchmarking tests using dovecot 2.0.12 to find the best
shared filesystem for hosting many users, here I share with you the results,
notice the bad perfomance of all the shared filesystems against the local
storage.
Is there any specific optimization/tunning on dovecot for use GFS2 on
rhel6??, we have configured the director to make the user mailbox persistent
in a node, we will thank's any help from you.
we are interested in using GFS2 or NFS, we believe the problem is the locks,
how can we improve this??
best regards, Aliet
The results
rhel 4.8 x86_64/GFS1 two nodes, shared FC lun on a SAN
Totals:
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo
100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100%
30% 5%
1- 2608 1321 1311 2608 2508 3545 547 2001 2493 2702 5282
2- 2810 1440 1430 2810 2688 3835 403 2154 2679 2925 5706
3- 2913 1457 1441 2908 2778 3913 417 2253 2773 3034 5924
4- 2814 1448 1412 2812 2695 3910 401 2186 2686 2929 5712
5- 2789 1464 1432 2787 2652 3774 427 2112 2649 2879 5676
6- 2843 1460 1444 2839 2722 3948 422 2164 2713 2957 5778
rhel6 x86_64/GFS2 two nodes, shared FC lun on a SAN(Used RDM in VMWare
vSphere for GFS2 lun)
Tunned cluster suite cluster.conf + <dlm plock_ownership="1"
plock_rate_limit="0"/>
<gfs_controld plock_rate_limit="0"/>
Totals:
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo
100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100%
30% 5%
1- 2730 1340 1356 2704 2644 3748 522 2125 2643 2662 5422
2- 3309 1618 1659 3294 3223 4658 531 2563 3221 3239 6596
3- 2131 1046 1017 2055 2025 2911 381 1608 2024 2052 4256
4- 2176 1055 1039 2082 2058 2947 377 1671 2058 2078 4344
5- 1859 928 931 1859 1800 2626 304 1454 1799 1801 3706
6- 2672 1322 1329 2672 2607 3758 464 2097 2606 2615 5326
rhel6 x86_64/GFS2 two nodes, shared FC lun on a SAN(Used RDM in VMWare
vSphere for GFS2 lun)
Cluster suite defaults configs for plocks
Totals:
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo
100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100%
30% 5%
1- 1417 644 676 1325 1305 1872 308 1048 1302 1318 2824
2- 837 378 392 742 726 1050 117 588 722 734 1658
3- 803 363 347 752 745 1069 153 597 744 750 1658
4- 1682 802 811 1587 1569 2261 291 1299 1569 1585 3360
5- 1146 583 564 1146 1037 1500 213 811 1037 1049 2290
6- 838 403 366 744 734 1057 152 561 731 736 1664
rhel6 x86_64 two nodes used NFS(NAS Freenas 0.8, nfsvers 3)
Totals:
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo
100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100%
30% 5%
1- 1382 699 691 1357 1063 1500 224 861 1053 1313 2694
2- 1634 785 799 1610 1459 2120 311 1192 1451 1570 3204
2- 1635 826 806 1611 1463 2088 345 1159 1459 1568 3190
3- 1574 758 781 1537 1403 2060 324 1135 1396 1504 3090
4- 1685 842 807 1653 1506 2135 349 1215 1504 1634 3344
5- 1766 850 893 1737 1582 2289 335 1288 1579 1705 3480
6- 1597 797 769 1572 1423 2007 313 1133 1420 1536 3142
rhel6 x86_64 local storage
Totals:
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo
100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100%
30% 5%
1- 7798 3868 3889 7706 7566 10713 1080 6089 7559 7688 15562
2- 7806 3879 3874 7716 7585 10873 1114 6018 7578 7696 15572
3- 7866 3910 3855 7773 7748 11053 1076 6253 7747 7761 15710
4- 7893 3978 3931 7802 7772 10988 1117 6197 7767 7789 15760
5- 7775 3853 3809 7683 7654 10897 1081 6142 7651 7675 15534
6- 7877 3919 3872 7789 7758 10986 1085 6218 7755 7773 15720
Aliet Santiesteban Sifontes
2011-May-05 23:56 UTC
[Dovecot] Dovecot imaptest on RHEL4/GFS1, RHEL6/GFS2, NFS and local storage results
We have used sdbox as mailbox format, and all the user data is configured in LDAP Servers 2011/5/5 Aliet Santiesteban Sifontes <alietsantiesteban at gmail.com>> We have done some benchmarking tests using dovecot 2.0.12 to find the best > shared filesystem for hosting many users, here I share with you the results, > notice the bad perfomance of all the shared filesystems against the local > storage. > Is there any specific optimization/tunning on dovecot for use GFS2 on > rhel6??, we have configured the director to make the user mailbox persistent > in a node, we will thank's any help from you. > we are interested in using GFS2 or NFS, we believe the problem is the > locks, how can we improve this?? > > best regards, Aliet > > The results > > rhel 4.8 x86_64/GFS1 two nodes, shared FC lun on a SAN > > Totals: > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% > 30% 5% > 1- 2608 1321 1311 2608 2508 3545 547 2001 2493 2702 5282 > 2- 2810 1440 1430 2810 2688 3835 403 2154 2679 2925 5706 > 3- 2913 1457 1441 2908 2778 3913 417 2253 2773 3034 5924 > 4- 2814 1448 1412 2812 2695 3910 401 2186 2686 2929 5712 > 5- 2789 1464 1432 2787 2652 3774 427 2112 2649 2879 5676 > 6- 2843 1460 1444 2839 2722 3948 422 2164 2713 2957 5778 > > rhel6 x86_64/GFS2 two nodes, shared FC lun on a SAN(Used RDM in VMWare > vSphere for GFS2 lun) > Tunned cluster suite cluster.conf + <dlm plock_ownership="1" > plock_rate_limit="0"/> > <gfs_controld plock_rate_limit="0"/> > > Totals: > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% > 30% 5% > 1- 2730 1340 1356 2704 2644 3748 522 2125 2643 2662 5422 > 2- 3309 1618 1659 3294 3223 4658 531 2563 3221 3239 6596 > 3- 2131 1046 1017 2055 2025 2911 381 1608 2024 2052 4256 > 4- 2176 1055 1039 2082 2058 2947 377 1671 2058 2078 4344 > 5- 1859 928 931 1859 1800 2626 304 1454 1799 1801 3706 > 6- 2672 1322 1329 2672 2607 3758 464 2097 2606 2615 5326 > > > rhel6 x86_64/GFS2 two nodes, shared FC lun on a SAN(Used RDM in VMWare > vSphere for GFS2 lun) > Cluster suite defaults configs for plocks > > Totals: > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% > 30% 5% > 1- 1417 644 676 1325 1305 1872 308 1048 1302 1318 2824 > 2- 837 378 392 742 726 1050 117 588 722 734 1658 > 3- 803 363 347 752 745 1069 153 597 744 750 1658 > 4- 1682 802 811 1587 1569 2261 291 1299 1569 1585 3360 > 5- 1146 583 564 1146 1037 1500 213 811 1037 1049 2290 > 6- 838 403 366 744 734 1057 152 561 731 736 1664 > > rhel6 x86_64 two nodes used NFS(NAS Freenas 0.8, nfsvers 3) > > Totals: > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% > 30% 5% > 1- 1382 699 691 1357 1063 1500 224 861 1053 1313 2694 > 2- 1634 785 799 1610 1459 2120 311 1192 1451 1570 3204 > 2- 1635 826 806 1611 1463 2088 345 1159 1459 1568 3190 > 3- 1574 758 781 1537 1403 2060 324 1135 1396 1504 3090 > 4- 1685 842 807 1653 1506 2135 349 1215 1504 1634 3344 > 5- 1766 850 893 1737 1582 2289 335 1288 1579 1705 3480 > 6- 1597 797 769 1572 1423 2007 313 1133 1420 1536 3142 > > rhel6 x86_64 local storage > > Totals: > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% > 30% 5% > 1- 7798 3868 3889 7706 7566 10713 1080 6089 7559 7688 15562 > 2- 7806 3879 3874 7716 7585 10873 1114 6018 7578 7696 15572 > 3- 7866 3910 3855 7773 7748 11053 1076 6253 7747 7761 15710 > 4- 7893 3978 3931 7802 7772 10988 1117 6197 7767 7789 15760 > 5- 7775 3853 3809 7683 7654 10897 1081 6142 7651 7675 15534 > 6- 7877 3919 3872 7789 7758 10986 1085 6218 7755 7773 15720 >
Charles Marcus
2011-May-06 10:49 UTC
[Dovecot] Dovecot imaptest on RHEL4/GFS1, RHEL6/GFS2, NFS and local storage results
On 2011-05-05 7:56 PM, Aliet Santiesteban Sifontes wrote:> We have used sdbox as mailbox format, and all the user data is configured in > LDAP ServersIt might help Timo to provide some suggestions if you also provide dovecot -n output... ;) -- Best regards, Charles
Aliet Santiesteban Sifontes
2011-May-06 14:38 UTC
[Dovecot] Dovecot imaptest on RHEL4/GFS1, RHEL6/GFS2, NFS and local storage results
the configs:
[root at n02 ~]# dovecot -n
# 2.0.12: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-71.24.1.el6.x86_64 x86_64 Red Hat Enterprise Linux Server
release 6.0 (Santiago)
auth_cache_size = 15 M
auth_default_realm = test.com
auth_mechanisms = plain login
auth_worker_max_count = 60
disable_plaintext_auth = no
login_greeting = Server ready.
mail_fsync = never
mail_location = sdbox:~/sdbox:INDEX=/vmail/index/%n
mail_plugins = quota zlib
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date
mbox_write_locks = fcntl
mmap_disable = yes
namespace {
inbox = yes
location prefix separator = /
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = Sent
autocreate2 = Trash
autocreate3 = Drafts
autocreate4 = Junk
autocreate5 = Archives
autocreate6 = Templates
autosubscribe = Sent
autosubscribe2 = Trash
autosubscribe3 = Drafts
autosubscribe4 = Junk
autosubscribe5 = Archives
autosubscribe6 = Templates
quota = dict:User quota::file:%h/sdbox/dovecot-quota
quota_rule = *:storage=250M
quota_rule2 = Trash:storage=+50M
quota_rule3 = Spam:storage=+25M
quota_rule4 = Sent:ignore
sieve = ~/.dovecot.sieve
sieve_before = /var/vmail/sievescripts/before.d
sieve_dir = ~/sieve
zlib_save = gz
zlib_save_level = 6
}
postmaster_address = postmaster at test.com
protocols = imap pop3 lmtp sieve
service auth {
unix_listener auth-userdb {
group = vmail
mode = 0660
user = root
}
}
service imap-login {
service_count = 0
}
best regards
2011/5/6 Charles Marcus <CMarcus at media-brokers.com>
> On 2011-05-05 7:56 PM, Aliet Santiesteban Sifontes wrote:
> > We have used sdbox as mailbox format, and all the user data is
configured
> in
> > LDAP Servers
>
> It might help Timo to provide some suggestions if you also provide
> dovecot -n output... ;)
>
> --
>
> Best regards,
>
> Charles
>
Aliet Santiesteban Sifontes
2011-May-07 03:01 UTC
[Dovecot] Dovecot imaptest on RHEL4/GFS1, RHEL6/GFS2, NFS and local storage results
New results, now with all plugins disabled:
os rhel6 x86_64, GFS2 Lun
Totals:
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo
100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100%
30% 5%
1- 4514 2189 2250 4514 4264 6163 709 3403 4260 4292 8726
2- 2827 1409 1389 2827 2765 3951 495 2168 2765 2777 5644
3- 2711 1409 1368 2711 2649 3833 512 2145 2647 2662 5396
4- 1799 912 890 1799 1720 2492 360 1370 1719 1735 3592
5- 3817 1869 1896 3760 3717 5313 575 3026 3715 3737 7616
6- 3296 1583 1628 3296 3215 4585 523 2600 3215 3238 6584
2011/5/6 Aliet Santiesteban Sifontes <alietsantiesteban at gmail.com>
> the configs:
>
> [root at n02 ~]# dovecot -n
> # 2.0.12: /etc/dovecot/dovecot.conf
> # OS: Linux 2.6.32-71.24.1.el6.x86_64 x86_64 Red Hat Enterprise Linux
> Server release 6.0 (Santiago)
> auth_cache_size = 15 M
> auth_default_realm = test.com
> auth_mechanisms = plain login
> auth_worker_max_count = 60
> disable_plaintext_auth = no
> login_greeting = Server ready.
> mail_fsync = never
> mail_location = sdbox:~/sdbox:INDEX=/vmail/index/%n
> mail_plugins = quota zlib
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character
> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
> copy include variables body enotify environment mailbox date
> mbox_write_locks = fcntl
> mmap_disable = yes
> namespace {
> inbox = yes
> location > prefix > separator = /
> }
> passdb {
> args = /etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> }
> plugin {
> autocreate = Sent
> autocreate2 = Trash
> autocreate3 = Drafts
> autocreate4 = Junk
> autocreate5 = Archives
> autocreate6 = Templates
> autosubscribe = Sent
> autosubscribe2 = Trash
> autosubscribe3 = Drafts
> autosubscribe4 = Junk
> autosubscribe5 = Archives
> autosubscribe6 = Templates
> quota = dict:User quota::file:%h/sdbox/dovecot-quota
> quota_rule = *:storage=250M
> quota_rule2 = Trash:storage=+50M
> quota_rule3 = Spam:storage=+25M
> quota_rule4 = Sent:ignore
> sieve = ~/.dovecot.sieve
> sieve_before = /var/vmail/sievescripts/before.d
> sieve_dir = ~/sieve
> zlib_save = gz
> zlib_save_level = 6
> }
> postmaster_address = postmaster at test.com
> protocols = imap pop3 lmtp sieve
> service auth {
> unix_listener auth-userdb {
> group = vmail
> mode = 0660
> user = root
> }
> }
> service imap-login {
> service_count = 0
> }
>
> best regards
>
>
> 2011/5/6 Charles Marcus <CMarcus at media-brokers.com>
>
>> On 2011-05-05 7:56 PM, Aliet Santiesteban Sifontes wrote:
>> > We have used sdbox as mailbox format, and all the user data is
>> configured in
>> > LDAP Servers
>>
>> It might help Timo to provide some suggestions if you also provide
>> dovecot -n output... ;)
>>
>> --
>>
>> Best regards,
>>
>> Charles
>>
>
>
Aliet Santiesteban Sifontes
2011-May-26 16:00 UTC
[Dovecot] Dovecot imaptest on RHEL4/GFS1, RHEL6/GFS2, NFS and local storage results
Thanks Ed, right now we are finishing the setup, next week we will continue the tests and will let you know the results... best regards 2011/5/23 Ed W <lists at wildgooses.com>> On 11/05/2011 00:00, Aliet Santiesteban Sifontes wrote: > > Using local storage(local hard driver ext4 filesystems) > > > > > > Totals: > > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo > > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% > > 30% 5% > > 7798 3868 3889 7706 7566 10713 1080 6089 7559 7688 15562 > > 7806 3879 3874 7716 7585 10873 1114 6018 7578 7696 15572 > > 7866 3910 3855 7773 7748 11053 1076 6253 7747 7761 15710 > > 7893 3978 3931 7802 7772 10988 1117 6197 7767 7789 15760 > > 7775 3853 3809 7683 7654 10897 1081 6142 7651 7675 15534 > > 7877 3919 3872 7789 7758 10986 1085 6218 7755 7773 15720 > > > > GFS2-mdbox, (no plugins) > > > > Totals: > > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo > > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% > > 30% 5% > > 7547 3739 3749 7455 7421 10605 1053 5931 7417 7443 15074 > > 7480 3702 3724 7387 7367 10558 1064 5874 7366 7378 14946 > > 7523 3759 3711 7428 7394 10560 1126 5898 7390 7412 15014 > > 7455 3736 3621 7364 7326 10561 1088 5854 7324 7349 14880 > > 7431 3712 3686 7337 7312 10406 1017 5882 7311 7328 14844 > > 7426 3704 3671 7334 7296 10364 1076 5791 7296 7325 14834 > > 7517 3673 3782 7425 7406 10554 1103 5913 7404 7414 15008 > > Hi, this performance seems excellent! > > There is no reason at all why you might try this, but as someone on > lower end hardware I would be fascinated to learn how the performance > changes is: > > - Switch FC to gig ethernet? (expecting substantial performance hit?) > - Reverting to maildir (suspecting much less of a hit based on your > numbers above?) > - OCFS vs GFS (although probably not sensible in your architecture since > you have a support contract for GFS, some have suggested OCFS can be > faster?) > > Please do post any other performance results - seems like you have found > an excellent cluster setup? > > Ed W >