> * Kevin Laurie <superinterstellar at gmail.com> 2015.01.24 19:41: > > > Currently the time it takes to search 25,000mails is 4mins. If indexed > how > > much faster are we looking at? > > With a current version of Dovecot a search is pretty fast _without_ using > external indexes. I have a view defined (virtual plugin) with around 22.000 > messages in it, and searching the full view only takes 2.5 seconds: > >hmmm, could you please tell me more about your setup? What storage format do you use? Maildir or? What is the cumulative size of your messages? My results without FTS on ZFS FS with SSD L2ARC are not so good: a4 select INBOX._OLD-OUTLOOK * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk JunkRecorded $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. ** 7748 EXISTS* * 0 RECENT * OK [UIDVALIDITY 1421198037] UIDs valid * OK [UIDNEXT 11509] Predicted next UID * OK [HIGHESTMODSEQ 12204] Highest a4 OK [READ-WRITE] Select completed (0.001 secs). a5 search charset utf-8 body "mall" * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240 249 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 389 390 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 532 533 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 677 692 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 833 855 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010 1011 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137 1140 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282 1286 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471 1494 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606 1611 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735 1781 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002 2006 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208 2210 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377 2378 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539 2551 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754 2780 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936 2938 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161 3178 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386 3428 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801 3808 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953 3954 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059 4063 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145 4146 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258 4259 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502 4538 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594 4604 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787 4788 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901 4905 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129 5131 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276 5277 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391 5392 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642 5647 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866 5867 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958 5990 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233 6271 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482 6492 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852 6859 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042 7108 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317 7322 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417 7418 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502 7503 7558 7588 7589 7628 7685 7695 7699 7703 7723 *a5 OK Search completed (7.846 secs).* Searching against "subject" is pretty fast, few miliseconds ...
Dear Peter, My inbox is MDA_external Storage: 17GB of 24GB Subject / From / To is fast but FTS(Full Text Search) for body is horrible. I suppose this is where we need Apache Solr. Do you think my mail storage format is bad? Do I need to change for better performance? Please advise Kevin On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur at gmail.com> wrote:> > * Kevin Laurie <superinterstellar at gmail.com> 2015.01.24 19:41: > > > > > Currently the time it takes to search 25,000mails is 4mins. If indexed > > how > > > much faster are we looking at? > > > > With a current version of Dovecot a search is pretty fast _without_ using > > external indexes. I have a view defined (virtual plugin) with around > 22.000 > > messages in it, and searching the full view only takes 2.5 seconds: > > > > > hmmm, could you please tell me more about your setup? What storage format > do you use? Maildir or? What is the cumulative size of your messages? > > My results without FTS on ZFS FS with SSD L2ARC are not so good: > > a4 select INBOX._OLD-OUTLOOK > * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk > JunkRecorded $Forwarded) > * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk > $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. > ** 7748 EXISTS* > * 0 RECENT > * OK [UIDVALIDITY 1421198037] UIDs valid > * OK [UIDNEXT 11509] Predicted next UID > * OK [HIGHESTMODSEQ 12204] Highest > a4 OK [READ-WRITE] Select completed (0.001 secs). > > a5 search charset utf-8 body "mall" > * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240 249 > 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 389 390 > 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 532 533 > 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 677 692 > 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 833 855 > 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010 1011 > 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137 1140 > 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282 1286 > 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471 1494 > 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606 1611 > 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735 1781 > 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002 2006 > 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208 2210 > 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377 2378 > 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539 2551 > 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754 2780 > 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936 2938 > 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161 3178 > 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386 3428 > 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801 3808 > 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953 3954 > 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059 4063 > 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145 4146 > 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258 4259 > 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502 4538 > 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594 4604 > 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787 4788 > 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901 4905 > 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129 5131 > 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276 5277 > 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391 5392 > 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642 5647 > 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866 5867 > 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958 5990 > 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233 6271 > 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482 6492 > 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852 6859 > 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042 7108 > 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317 7322 > 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417 7418 > 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502 7503 > 7558 7588 7589 7628 7685 7695 7699 7703 7723 > *a5 OK Search completed (7.846 secs).* > > > Searching against "subject" is pretty fast, few miliseconds ... >
Hi Peter,
Sorry,
I think its maildir
Output of my dovecot -n is listed below:-
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS
auth_master_user_separator = *
auth_mechanisms = PLAIN LOGIN
dict {
acl = mysql:/etc/dovecot/dovecot-
share-folder.conf
quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
first_valid_uid = 2000
last_valid_uid = 2000
listen = *
log_path = /var/log/dovecot.log
mail_debug = yes
mail_gid = 2000
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_plugins = quota fts
mail_uid = 2000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
namespace {
inbox = yes
location mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = no
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix separator = /
type = private
}
namespace {
list = children
location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
prefix = Shared/%%u/
separator = /
subscriptions = yes
type = shared
}
passdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
passdb {
args = /etc/dovecot/dovecot-master-users-password
driver = passwd-file
master = yes
}
plugin {
acl = vfile
acl_shared_dict = proxy::acl
auth_socket_path = /var/run/dovecot/auth-master
quota = dict:user::proxy::quotadict
quota_rule = *:storage=1G
quota_warning = storage=85%% quota-warning 85 %u
quota_warning2 = storage=90%% quota-warning 90 %u
quota_warning3 = storage=95%% quota-warning 95 %u
sieve = /%Lh/sieve/dovecot.sieve
sieve_default = /var/vmail/sieve/dovecot.sieve
sieve_dir = /%Lh/sieve
sieve_global_dir = /var/vmail/sieve
}
protocols = pop3 imap sieve lmtp
service auth {
unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0666
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0666
user = vmail
}
unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
}
}
service dict {
unix_listener dict {
group = vmail
mode = 0660
user = vmail
}
}
service imap-login {
process_limit = 500
service_count = 1
}
service lmtp {
executable = lmtp -L
inet_listener lmtp {
port = 24
}
process_min_avail = 5
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
user = vmail
}
service pop3-login {
service_count = 1
}
service quota-warning {
executable = script /usr/local/bin/dovecot-quota-warning.sh
unix_listener quota-warning {
group = vmail
mode = 0660
user = vmail
}
}
ssl = required
ssl_cert = </etc/ssl/certs/sicl.net.crt
ssl_key = </etc/ssl/private/sicl.net.key
userdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
lda_mailbox_autocreate = yes
log_path = /var/log/dovecot-sieve.log
mail_plugins = quota fts sieve
postmaster_address = root
}
protocol lmtp {
info_log_path = /var/log/dovecot-lmtp.log
lmtp_save_to_detail_mailbox = yes
mail_plugins = quota sieve
postmaster_address = postmaster
recipient_delimiter = +
}
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep
mail_plugins = quota fts imap_quota
}
protocol pop3 {
mail_plugins = quota fts
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xu%08Xv
}
On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie <superinterstellar at
gmail.com>
wrote:
> Dear Peter,
>
> My inbox is MDA_external
> Storage: 17GB of 24GB
>
> Subject / From / To is fast but FTS(Full Text Search) for body is
> horrible. I suppose this is where we need Apache Solr.
>
> Do you think my mail storage format is bad? Do I need to change for better
> performance?
> Please advise
> Kevin
>
>
>
> On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur at
gmail.com> wrote:
>
>> > * Kevin Laurie <superinterstellar at gmail.com> 2015.01.24
19:41:
>> >
>> > > Currently the time it takes to search 25,000mails is 4mins.
If indexed
>> > how
>> > > much faster are we looking at?
>> >
>> > With a current version of Dovecot a search is pretty fast
_without_
>> using
>> > external indexes. I have a view defined (virtual plugin) with
around
>> 22.000
>> > messages in it, and searching the full view only takes 2.5
seconds:
>> >
>> >
>> hmmm, could you please tell me more about your setup? What storage
format
>> do you use? Maildir or? What is the cumulative size of your messages?
>>
>> My results without FTS on ZFS FS with SSD L2ARC are not so good:
>>
>> a4 select INBOX._OLD-OUTLOOK
>> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk
>> JunkRecorded $Forwarded)
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk
>> $NotJunk JunkRecorded $Forwarded \*)] Flags permitted.
>> ** 7748 EXISTS*
>> * 0 RECENT
>> * OK [UIDVALIDITY 1421198037] UIDs valid
>> * OK [UIDNEXT 11509] Predicted next UID
>> * OK [HIGHESTMODSEQ 12204] Highest
>> a4 OK [READ-WRITE] Select completed (0.001 secs).
>>
>> a5 search charset utf-8 body "mall"
>> * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240
249
>> 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 389
>> 390
>> 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 532
>> 533
>> 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 677
>> 692
>> 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 833
>> 855
>> 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010
1011
>> 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137
1140
>> 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282
1286
>> 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471
1494
>> 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606
1611
>> 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735
1781
>> 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002
2006
>> 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208
2210
>> 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377
2378
>> 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539
2551
>> 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754
2780
>> 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936
2938
>> 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161
3178
>> 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386
3428
>> 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801
3808
>> 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953
3954
>> 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059
4063
>> 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145
4146
>> 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258
4259
>> 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502
4538
>> 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594
4604
>> 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787
4788
>> 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901
4905
>> 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129
5131
>> 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276
5277
>> 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391
5392
>> 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642
5647
>> 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866
5867
>> 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958
5990
>> 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233
6271
>> 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482
6492
>> 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852
6859
>> 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042
7108
>> 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317
7322
>> 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417
7418
>> 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502
7503
>> 7558 7588 7589 7628 7685 7695 7699 7703 7723
>> *a5 OK Search completed (7.846 secs).*
>>
>>
>> Searching against "subject" is pretty fast, few miliseconds
...
>>
>
>
Dear Peter, Noted. Thanks for your input. Appreciate it. At this point my urgent most priority is to get FTS. Waiting 3 mins for a body search email is bad(but of course my email box is large) I need to have this sorted out by today as I have been putting this off for too long mostly because of lack of troubleshooting documentation online(if i do figure this out will create some for guidance) Regards Kevin On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur <petehodur at gmail.com> wrote:> > > On Thursday, January 29, 2015, Kevin Laurie <superinterstellar at gmail.com> > wrote: > >> Dear Peter, >> Oh. Sorry(didnt know you were addressing someone else) my apologies. But >> as you can see , I am desperately trying to address this issue. >> > > No problem ;))) i wrote because my search result is good but not so > good as someone wrote here couple of days ago. > > > >> I have 1 disk with 200GB running on a VPS. >> The file system is ext4. >> > > Im not dovecot expert, but i think this is your problem. You need more > iops. If you need performance, VPS may be (if connected to fast SAN) but > basicly is not a good choice. > > >> I think I will need to implement FTS to fix this as I will need body >> searches. >> > > FTS could be a solution. You are right. > > >> Have you tried FTS before? >> > > Im sorry, never :( > > But may be someone more skilled than me could answer the main question, > how much dovecot index helps in case imap search agains body. > > Im not sure, but my opinion is that it helps little (may be not at all) > > Ps: may be you use it, but do not forget to setup delivering from mail > server via LDA or LMTP instead of direct to maildirs. This is better > because dovecot updates your indexes at the time of message arrival. Later > accesses should be faster. > > Peter > > > >> >> >> Regards >> Kevin >> > >> >> On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur <petehodur at gmail.com> >> wrote: >> >>> Kevin, >>> >>> My message was not addressed to you ;) because my results are NOT so >>> good as someone wrote it here. I can fulltext cca 8k messages in 7-8 >>> seconds. >>> >>> Someone wrote that he has results cca 22k messages in 4 seconds :( >>> >>> Im not sure, but the answer is may be in the index and disk subsystem. >>> >>> If and only if dovecot index does not have saved in the index keywords >>> from BODY of messages the problem is your/my disk subsystem. >>> >>> Generally, messages from mailing list like this are pretty small. The >>> problem is that dovecot must read all messages. If you have maildir its >>> mean open and read many files. >>> >>> But my test search was agains archive of my personal inbox - so not onoy >>> small messages like this but also messages with big attachments etc. >>> >>> If dovecot index only headers, than it in case of full text search read >>> whole messages - mime parts allows that plain text body can be after >>> attachment etc. >>> >>> And here is it all about iops and throughput. >>> >>> How many disks you have? And what setup? >>> >>> Generally, the only good setup is many smaller disks in RAID 10 >>> (stripped mirrors) and if your fs allows it, with added read cache - in >>> case of zfs: l2arc on ssd. >>> >>> >>> >>> Pete >>> >>> >>> >>> On Thursday, January 29, 2015, Kevin Laurie <superinterstellar at gmail.com> >>> wrote: >>> >>>> Hi Peter, >>>> Sorry, >>>> >>>> I think its maildir >>>> Output of my dovecot -n is listed below:- >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> # 2.2.9: /etc/dovecot/dovecot.conf >>>> # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS >>>> auth_master_user_separator = * >>>> auth_mechanisms = PLAIN LOGIN >>>> dict { >>>> acl = mysql:/etc/dovecot/dovecot- >>>> share-folder.conf >>>> quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf >>>> } >>>> first_valid_uid = 2000 >>>> last_valid_uid = 2000 >>>> listen = * >>>> log_path = /var/log/dovecot.log >>>> mail_debug = yes >>>> mail_gid = 2000 >>>> mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ >>>> mail_plugins = quota fts >>>> mail_uid = 2000 >>>> managesieve_notify_capability = mailto >>>> managesieve_sieve_capability = fileinto reject envelope >>>> encoded-character vacation subaddress comparator-i;ascii-numeric relational >>>> regex imap4flags copy include variables body enotify environment mailbox >>>> date ihave >>>> namespace { >>>> inbox = yes >>>> location >>>> mailbox Drafts { >>>> auto = subscribe >>>> special_use = \Drafts >>>> } >>>> mailbox Junk { >>>> auto = subscribe >>>> special_use = \Junk >>>> } >>>> mailbox Sent { >>>> auto = subscribe >>>> special_use = \Sent >>>> } >>>> mailbox "Sent Messages" { >>>> auto = no >>>> special_use = \Sent >>>> } >>>> mailbox Spam { >>>> auto = no >>>> special_use = \Junk >>>> } >>>> mailbox Trash { >>>> auto = subscribe >>>> special_use = \Trash >>>> } >>>> prefix >>>> separator = / >>>> type = private >>>> } >>>> namespace { >>>> list = children >>>> location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u >>>> prefix = Shared/%%u/ >>>> separator = / >>>> subscriptions = yes >>>> type = shared >>>> } >>>> passdb { >>>> args = /etc/dovecot/dovecot-mysql.conf >>>> driver = sql >>>> } >>>> passdb { >>>> args = /etc/dovecot/dovecot-master-users-password >>>> driver = passwd-file >>>> master = yes >>>> } >>>> plugin { >>>> acl = vfile >>>> acl_shared_dict = proxy::acl >>>> auth_socket_path = /var/run/dovecot/auth-master >>>> quota = dict:user::proxy::quotadict >>>> quota_rule = *:storage=1G >>>> quota_warning = storage=85%% quota-warning 85 %u >>>> quota_warning2 = storage=90%% quota-warning 90 %u >>>> quota_warning3 = storage=95%% quota-warning 95 %u >>>> sieve = /%Lh/sieve/dovecot.sieve >>>> sieve_default = /var/vmail/sieve/dovecot.sieve >>>> sieve_dir = /%Lh/sieve >>>> sieve_global_dir = /var/vmail/sieve >>>> } >>>> protocols = pop3 imap sieve lmtp >>>> service auth { >>>> unix_listener /var/spool/postfix/private/dovecot-auth { >>>> group = postfix >>>> mode = 0666 >>>> user = postfix >>>> } >>>> unix_listener auth-master { >>>> group = vmail >>>> mode = 0666 >>>> user = vmail >>>> } >>>> unix_listener auth-userdb { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> service dict { >>>> unix_listener dict { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> service imap-login { >>>> process_limit = 500 >>>> service_count = 1 >>>> } >>>> service lmtp { >>>> executable = lmtp -L >>>> inet_listener lmtp { >>>> port = 24 >>>> } >>>> process_min_avail = 5 >>>> unix_listener /var/spool/postfix/private/dovecot-lmtp { >>>> group = postfix >>>> mode = 0600 >>>> user = postfix >>>> } >>>> user = vmail >>>> } >>>> service pop3-login { >>>> service_count = 1 >>>> } >>>> service quota-warning { >>>> executable = script /usr/local/bin/dovecot-quota-warning.sh >>>> unix_listener quota-warning { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> ssl = required >>>> ssl_cert = </etc/ssl/certs/sicl.net.crt >>>> ssl_key = </etc/ssl/private/sicl.net.key >>>> userdb { >>>> args = /etc/dovecot/dovecot-mysql.conf >>>> driver = sql >>>> } >>>> protocol lda { >>>> auth_socket_path = /var/run/dovecot/auth-master >>>> lda_mailbox_autocreate = yes >>>> log_path = /var/log/dovecot-sieve.log >>>> mail_plugins = quota fts sieve >>>> postmaster_address = root >>>> } >>>> protocol lmtp { >>>> info_log_path = /var/log/dovecot-lmtp.log >>>> lmtp_save_to_detail_mailbox = yes >>>> mail_plugins = quota sieve >>>> postmaster_address = postmaster >>>> recipient_delimiter = + >>>> } >>>> protocol imap { >>>> imap_client_workarounds = tb-extra-mailbox-sep >>>> mail_plugins = quota fts imap_quota >>>> } >>>> protocol pop3 { >>>> mail_plugins = quota fts >>>> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh >>>> pop3_uidl_format = %08Xu%08Xv >>>> } >>>> >>>> >>>> On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie < >>>> superinterstellar at gmail.com> wrote: >>>> >>>>> Dear Peter, >>>>> >>>>> My inbox is MDA_external >>>>> Storage: 17GB of 24GB >>>>> >>>>> Subject / From / To is fast but FTS(Full Text Search) for body is >>>>> horrible. I suppose this is where we need Apache Solr. >>>>> >>>>> Do you think my mail storage format is bad? Do I need to change for >>>>> better performance? >>>>> Please advise >>>>> Kevin >>>>> >>>>> >>>>> >>>>> On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur at gmail.com> >>>>> wrote: >>>>> >>>>>> > * Kevin Laurie <superinterstellar at gmail.com> 2015.01.24 19:41: >>>>>> > >>>>>> > > Currently the time it takes to search 25,000mails is 4mins. If >>>>>> indexed >>>>>> > how >>>>>> > > much faster are we looking at? >>>>>> > >>>>>> > With a current version of Dovecot a search is pretty fast _without_ >>>>>> using >>>>>> > external indexes. I have a view defined (virtual plugin) with >>>>>> around 22.000 >>>>>> > messages in it, and searching the full view only takes 2.5 seconds: >>>>>> > >>>>>> > >>>>>> hmmm, could you please tell me more about your setup? What storage >>>>>> format >>>>>> do you use? Maildir or? What is the cumulative size of your messages? >>>>>> >>>>>> My results without FTS on ZFS FS with SSD L2ARC are not so good: >>>>>> >>>>>> a4 select INBOX._OLD-OUTLOOK >>>>>> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk >>>>>> JunkRecorded $Forwarded) >>>>>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk >>>>>> $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. >>>>>> ** 7748 EXISTS* >>>>>> * 0 RECENT >>>>>> * OK [UIDVALIDITY 1421198037] UIDs valid >>>>>> * OK [UIDNEXT 11509] Predicted next UID >>>>>> * OK [HIGHESTMODSEQ 12204] Highest >>>>>> a4 OK [READ-WRITE] Select completed (0.001 secs). >>>>>> >>>>>> a5 search charset utf-8 body "mall" >>>>>> * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240 >>>>>> 249 >>>>>> 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 >>>>>> 389 390 >>>>>> 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 >>>>>> 532 533 >>>>>> 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 >>>>>> 677 692 >>>>>> 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 >>>>>> 833 855 >>>>>> 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010 >>>>>> 1011 >>>>>> 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137 >>>>>> 1140 >>>>>> 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282 >>>>>> 1286 >>>>>> 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471 >>>>>> 1494 >>>>>> 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606 >>>>>> 1611 >>>>>> 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735 >>>>>> 1781 >>>>>> 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002 >>>>>> 2006 >>>>>> 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208 >>>>>> 2210 >>>>>> 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377 >>>>>> 2378 >>>>>> 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539 >>>>>> 2551 >>>>>> 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754 >>>>>> 2780 >>>>>> 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936 >>>>>> 2938 >>>>>> 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161 >>>>>> 3178 >>>>>> 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386 >>>>>> 3428 >>>>>> 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801 >>>>>> 3808 >>>>>> 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953 >>>>>> 3954 >>>>>> 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059 >>>>>> 4063 >>>>>> 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145 >>>>>> 4146 >>>>>> 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258 >>>>>> 4259 >>>>>> 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502 >>>>>> 4538 >>>>>> 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594 >>>>>> 4604 >>>>>> 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787 >>>>>> 4788 >>>>>> 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901 >>>>>> 4905 >>>>>> 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129 >>>>>> 5131 >>>>>> 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276 >>>>>> 5277 >>>>>> 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391 >>>>>> 5392 >>>>>> 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642 >>>>>> 5647 >>>>>> 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866 >>>>>> 5867 >>>>>> 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958 >>>>>> 5990 >>>>>> 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233 >>>>>> 6271 >>>>>> 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482 >>>>>> 6492 >>>>>> 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852 >>>>>> 6859 >>>>>> 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042 >>>>>> 7108 >>>>>> 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317 >>>>>> 7322 >>>>>> 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417 >>>>>> 7418 >>>>>> 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502 >>>>>> 7503 >>>>>> 7558 7588 7589 7628 7685 7695 7699 7703 7723 >>>>>> *a5 OK Search completed (7.846 secs).* >>>>>> >>>>>> >>>>>> Searching against "subject" is pretty fast, few miliseconds ... >>>>>> >>>>> >>>>> >>>> >>
* Peter Hodur <petehodur at gmail.com> 2015.01.29 05:25:> > With a current version of Dovecot a search is pretty fast _without_ using > > external indexes. I have a view defined (virtual plugin) with around 22.000 > > messages in it, and searching the full view only takes 2.5 seconds: > > > > > hmmm, could you please tell me more about your setup? What storage format > do you use? Maildir or? What is the cumulative size of your messages?Although the question was not formatted/quoted correctly I guess it is more towards myself as it explicitely speaks of a non-FTS setup: $ dmesg | grep smpboot [ 0.000000] smpboot: Allowing 64 CPUs, 56 hotplug CPUs [ 0.148600] smpboot: CPU0: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz (fam: 06, model: 3c, stepping: 03) [ 0.300030] smpboot: Total of 8 processors activated (52677.72 BogoMIPS) The volume providing the mail root is using ext4 on an MD mirror (with CRYPT and LVM overhead): $ lsblk [...] ??sda3 8:3 0 ... 0 part ??md2 9:2 0 ... 0 raid1 ??crypt1 (dm-0) 253:0 0 ... 0 crypt ??vg0-vmail (dm-11) 253:11 0 ... 0 lvm /var/vmail $ doveconf -n | head -3 # 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 # OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8 I'm using unspectacular 7,200rpm SATA drives and mdbox. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20150129/14268a47/attachment.sig>
Dear Thomas, You do seem to have some computing power. I think that does help. Thanks Kevin A. On Thu, Jan 29, 2015 at 4:20 PM, Thomas Leuxner <tlx at leuxner.net> wrote:> * Peter Hodur <petehodur at gmail.com> 2015.01.29 05:25: > > > > With a current version of Dovecot a search is pretty fast _without_ > using > > > external indexes. I have a view defined (virtual plugin) with around > 22.000 > > > messages in it, and searching the full view only takes 2.5 seconds: > > > > > > > > hmmm, could you please tell me more about your setup? What storage format > > do you use? Maildir or? What is the cumulative size of your messages? > > > Although the question was not formatted/quoted correctly I guess it is > more towards myself as it explicitely speaks of a non-FTS setup: > > $ dmesg | grep smpboot > [ 0.000000] smpboot: Allowing 64 CPUs, 56 hotplug CPUs > [ 0.148600] smpboot: CPU0: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz > (fam: 06, model: 3c, stepping: 03) > [ 0.300030] smpboot: Total of 8 processors activated (52677.72 BogoMIPS) > > The volume providing the mail root is using ext4 on an MD mirror (with > CRYPT and LVM overhead): > > > $ lsblk > [...] > ??sda3 8:3 0 ... 0 part > ??md2 9:2 0 ... 0 raid1 > ??crypt1 (dm-0) 253:0 0 ... 0 crypt > ??vg0-vmail (dm-11) 253:11 0 ... 0 lvm /var/vmail > > $ doveconf -n | head -3 > # 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.4.6 > # OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8 > > I'm using unspectacular 7,200rpm SATA drives and mdbox. >
Hi Micheal, Here is how it should look:- ~# telnet localhost imap Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. . login user at domain.net password . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY QUOTA] Logged in . select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded \*)] Flags permitted. * 49319 EXISTS * 12 RECENT * OK [UNSEEN 46791] First unseen. * OK [UIDVALIDITY 1414214135] UIDs valid * OK [UIDNEXT 106554] Predicted next UID * OK [NOMODSEQ] No permanent modsequences . OK [READ-WRITE] Select completed (0.128 secs). . search text hello * OK Searched 6% of the mailbox, ETA 2:34 Finally at the end you logout by . logout On Thu, Jan 29, 2015 at 4:39 PM, Kevin Laurie <superinterstellar at gmail.com> wrote:> Dear Michael, > You will need to telnet locally to imap. So run the following commands in > your server > > 1. telnet localhost imap > 2. . login email at domain.com password > 3. . select inbox > 4. . search text "passport" (This command here will search your inbox and > give you the time needed) > 5. . logout > > Notice the period(.) You need to use it . > > Hope this is clear. > Regards > Kevin > > > On Thu, Jan 29, 2015 at 4:32 PM, M. Koehler <michael.koehler at makomi.de> > wrote: > >> Hi Kevin, >> >> I want to test it with my dovecot but I don?t know how to measure it. >> Could you please provide my the configuration/terminal commands to get an >> reliable output? I just have mailboxes up to 4GB in VM on a proxmox server >> (Debian Wheezy at the moment). >> >> regards, >> Michael >> >> >> >> > Am 29.01.2015 um 06:29 schrieb Kevin Laurie < >> superinterstellar at gmail.com>: >> > >> > Dear Peter, >> > Noted. Thanks for your input. Appreciate it. >> > At this point my urgent most priority is to get FTS. Waiting 3 mins for >> a >> > body search email is bad(but of course my email box is large) >> > I need to have this sorted out by today as I have been putting this off >> for >> > too long mostly because of lack of troubleshooting documentation >> online(if >> > i do figure this out will create some for guidance) >> > Regards >> > Kevin >> > >> > On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur <petehodur at gmail.com> >> wrote: >> > >> >> >> >> >> >> On Thursday, January 29, 2015, Kevin Laurie < >> superinterstellar at gmail.com> >> >> wrote: >> >> >> >>> Dear Peter, >> >>> Oh. Sorry(didnt know you were addressing someone else) my apologies. >> But >> >>> as you can see , I am desperately trying to address this issue. >> >>> >> >> >> >> No problem ;))) i wrote because my search result is good but not so >> >> good as someone wrote here couple of days ago. >> >> >> >> >> >> >> >>> I have 1 disk with 200GB running on a VPS. >> >>> The file system is ext4. >> >>> >> >> >> >> Im not dovecot expert, but i think this is your problem. You need more >> >> iops. If you need performance, VPS may be (if connected to fast SAN) >> but >> >> basicly is not a good choice. >> >> >> >> >> >>> I think I will need to implement FTS to fix this as I will need body >> >>> searches. >> >>> >> >> >> >> FTS could be a solution. You are right. >> >> >> >> >> >>> Have you tried FTS before? >> >>> >> >> >> >> Im sorry, never :( >> >> >> >> But may be someone more skilled than me could answer the main question, >> >> how much dovecot index helps in case imap search agains body. >> >> >> >> Im not sure, but my opinion is that it helps little (may be not at all) >> >> >> >> Ps: may be you use it, but do not forget to setup delivering from mail >> >> server via LDA or LMTP instead of direct to maildirs. This is better >> >> because dovecot updates your indexes at the time of message arrival. >> Later >> >> accesses should be faster. >> >> >> >> Peter >> >> >> >> >> >> >> >>> >> >>> >> >>> Regards >> >>> Kevin >> >>> >> >> >> >>> >> >>> On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur <petehodur at gmail.com> >> >>> wrote: >> >>> >> >>>> Kevin, >> >>>> >> >>>> My message was not addressed to you ;) because my results are NOT so >> >>>> good as someone wrote it here. I can fulltext cca 8k messages in 7-8 >> >>>> seconds. >> >>>> >> >>>> Someone wrote that he has results cca 22k messages in 4 seconds :( >> >>>> >> >>>> Im not sure, but the answer is may be in the index and disk >> subsystem. >> >>>> >> >>>> If and only if dovecot index does not have saved in the index >> keywords >> >>>> from BODY of messages the problem is your/my disk subsystem. >> >>>> >> >>>> Generally, messages from mailing list like this are pretty small. The >> >>>> problem is that dovecot must read all messages. If you have maildir >> its >> >>>> mean open and read many files. >> >>>> >> >>>> But my test search was agains archive of my personal inbox - so not >> onoy >> >>>> small messages like this but also messages with big attachments etc. >> >>>> >> >>>> If dovecot index only headers, than it in case of full text search >> read >> >>>> whole messages - mime parts allows that plain text body can be after >> >>>> attachment etc. >> >>>> >> >>>> And here is it all about iops and throughput. >> >>>> >> >>>> How many disks you have? And what setup? >> >>>> >> >>>> Generally, the only good setup is many smaller disks in RAID 10 >> >>>> (stripped mirrors) and if your fs allows it, with added read cache - >> in >> >>>> case of zfs: l2arc on ssd. >> >>>> >> >>>> >> >>>> >> >>>> Pete >> >>>> >> >>>> >> >>>> >> >>>> On Thursday, January 29, 2015, Kevin Laurie < >> superinterstellar at gmail.com> >> >>>> wrote: >> >>>> >> >>>>> Hi Peter, >> >>>>> Sorry, >> >>>>> >> >>>>> I think its maildir >> >>>>> Output of my dovecot -n is listed below:- >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> # 2.2.9: /etc/dovecot/dovecot.conf >> >>>>> # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS >> >>>>> auth_master_user_separator = * >> >>>>> auth_mechanisms = PLAIN LOGIN >> >>>>> dict { >> >>>>> acl = mysql:/etc/dovecot/dovecot- >> >>>>> share-folder.conf >> >>>>> quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf >> >>>>> } >> >>>>> first_valid_uid = 2000 >> >>>>> last_valid_uid = 2000 >> >>>>> listen = * >> >>>>> log_path = /var/log/dovecot.log >> >>>>> mail_debug = yes >> >>>>> mail_gid = 2000 >> >>>>> mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ >> >>>>> mail_plugins = quota fts >> >>>>> mail_uid = 2000 >> >>>>> managesieve_notify_capability = mailto >> >>>>> managesieve_sieve_capability = fileinto reject envelope >> >>>>> encoded-character vacation subaddress comparator-i;ascii-numeric >> relational >> >>>>> regex imap4flags copy include variables body enotify environment >> mailbox >> >>>>> date ihave >> >>>>> namespace { >> >>>>> inbox = yes >> >>>>> location >> >>>>> mailbox Drafts { >> >>>>> auto = subscribe >> >>>>> special_use = \Drafts >> >>>>> } >> >>>>> mailbox Junk { >> >>>>> auto = subscribe >> >>>>> special_use = \Junk >> >>>>> } >> >>>>> mailbox Sent { >> >>>>> auto = subscribe >> >>>>> special_use = \Sent >> >>>>> } >> >>>>> mailbox "Sent Messages" { >> >>>>> auto = no >> >>>>> special_use = \Sent >> >>>>> } >> >>>>> mailbox Spam { >> >>>>> auto = no >> >>>>> special_use = \Junk >> >>>>> } >> >>>>> mailbox Trash { >> >>>>> auto = subscribe >> >>>>> special_use = \Trash >> >>>>> } >> >>>>> prefix >> >>>>> separator = / >> >>>>> type = private >> >>>>> } >> >>>>> namespace { >> >>>>> list = children >> >>>>> location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u >> >>>>> prefix = Shared/%%u/ >> >>>>> separator = / >> >>>>> subscriptions = yes >> >>>>> type = shared >> >>>>> } >> >>>>> passdb { >> >>>>> args = /etc/dovecot/dovecot-mysql.conf >> >>>>> driver = sql >> >>>>> } >> >>>>> passdb { >> >>>>> args = /etc/dovecot/dovecot-master-users-password >> >>>>> driver = passwd-file >> >>>>> master = yes >> >>>>> } >> >>>>> plugin { >> >>>>> acl = vfile >> >>>>> acl_shared_dict = proxy::acl >> >>>>> auth_socket_path = /var/run/dovecot/auth-master >> >>>>> quota = dict:user::proxy::quotadict >> >>>>> quota_rule = *:storage=1G >> >>>>> quota_warning = storage=85%% quota-warning 85 %u >> >>>>> quota_warning2 = storage=90%% quota-warning 90 %u >> >>>>> quota_warning3 = storage=95%% quota-warning 95 %u >> >>>>> sieve = /%Lh/sieve/dovecot.sieve >> >>>>> sieve_default = /var/vmail/sieve/dovecot.sieve >> >>>>> sieve_dir = /%Lh/sieve >> >>>>> sieve_global_dir = /var/vmail/sieve >> >>>>> } >> >>>>> protocols = pop3 imap sieve lmtp >> >>>>> service auth { >> >>>>> unix_listener /var/spool/postfix/private/dovecot-auth { >> >>>>> group = postfix >> >>>>> mode = 0666 >> >>>>> user = postfix >> >>>>> } >> >>>>> unix_listener auth-master { >> >>>>> group = vmail >> >>>>> mode = 0666 >> >>>>> user = vmail >> >>>>> } >> >>>>> unix_listener auth-userdb { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> service dict { >> >>>>> unix_listener dict { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> service imap-login { >> >>>>> process_limit = 500 >> >>>>> service_count = 1 >> >>>>> } >> >>>>> service lmtp { >> >>>>> executable = lmtp -L >> >>>>> inet_listener lmtp { >> >>>>> port = 24 >> >>>>> } >> >>>>> process_min_avail = 5 >> >>>>> unix_listener /var/spool/postfix/private/dovecot-lmtp { >> >>>>> group = postfix >> >>>>> mode = 0600 >> >>>>> user = postfix >> >>>>> } >> >>>>> user = vmail >> >>>>> } >> >>>>> service pop3-login { >> >>>>> service_count = 1 >> >>>>> } >> >>>>> service quota-warning { >> >>>>> executable = script /usr/local/bin/dovecot-quota-warning.sh >> >>>>> unix_listener quota-warning { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> ssl = required >> >>>>> ssl_cert = </etc/ssl/certs/sicl.net.crt >> >>>>> ssl_key = </etc/ssl/private/sicl.net.key >> >>>>> userdb { >> >>>>> args = /etc/dovecot/dovecot-mysql.conf >> >>>>> driver = sql >> >>>>> } >> >>>>> protocol lda { >> >>>>> auth_socket_path = /var/run/dovecot/auth-master >> >>>>> lda_mailbox_autocreate = yes >> >>>>> log_path = /var/log/dovecot-sieve.log >> >>>>> mail_plugins = quota fts sieve >> >>>>> postmaster_address = root >> >>>>> } >> >>>>> protocol lmtp { >> >>>>> info_log_path = /var/log/dovecot-lmtp.log >> >>>>> lmtp_save_to_detail_mailbox = yes >> >>>>> mail_plugins = quota sieve >> >>>>> postmaster_address = postmaster >> >>>>> recipient_delimiter = + >> >>>>> } >> >>>>> protocol imap { >> >>>>> imap_client_workarounds = tb-extra-mailbox-sep >> >>>>> mail_plugins = quota fts imap_quota >> >>>>> } >> >>>>> protocol pop3 { >> >>>>> mail_plugins = quota fts >> >>>>> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh >> >>>>> pop3_uidl_format = %08Xu%08Xv >> >>>>> } >> >>>>> >> >>>>> >> >>>>> On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie < >> >>>>> superinterstellar at gmail.com> wrote: >> >>>>> >> >>>>>> Dear Peter, >> >>>>>> >> >>>>>> My inbox is MDA_external >> >>>>>> Storage: 17GB of 24GB >> >>>>>> >> >>>>>> Subject / From / To is fast but FTS(Full Text Search) for body is >> >>>>>> horrible. I suppose this is where we need Apache Solr. >> >>>>>> >> >>>>>> Do you think my mail storage format is bad? Do I need to change for >> >>>>>> better performance? >> >>>>>> Please advise >> >>>>>> Kevin >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur at gmail.com >> > >> >>>>>> wrote: >> >>>>>> >> >>>>>>>> * Kevin Laurie <superinterstellar at gmail.com> 2015.01.24 19:41: >> >>>>>>>> >> >>>>>>>>> Currently the time it takes to search 25,000mails is 4mins. If >> >>>>>>> indexed >> >>>>>>>> how >> >>>>>>>>> much faster are we looking at? >> >>>>>>>> >> >>>>>>>> With a current version of Dovecot a search is pretty fast >> _without_ >> >>>>>>> using >> >>>>>>>> external indexes. I have a view defined (virtual plugin) with >> >>>>>>> around 22.000 >> >>>>>>>> messages in it, and searching the full view only takes 2.5 >> seconds: >> >>>>>>>> >> >>>>>>>> >> >>>>>>> hmmm, could you please tell me more about your setup? What storage >> >>>>>>> format >> >>>>>>> do you use? Maildir or? What is the cumulative size of your >> messages? >> >>>>>>> >> >>>>>>> My results without FTS on ZFS FS with SSD L2ARC are not so good: >> >>>>>>> >> >>>>>>> a4 select INBOX._OLD-OUTLOOK >> >>>>>>> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk >> >>>>>>> JunkRecorded $Forwarded) >> >>>>>>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft >> $Junk >> >>>>>>> $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. >> >>>>>>> ** 7748 EXISTS* >> >>>>>>> * 0 RECENT >> >>>>>>> * OK [UIDVALIDITY 1421198037] UIDs valid >> >>>>>>> * OK [UIDNEXT 11509] Predicted next UID >> >>>>>>> * OK [HIGHESTMODSEQ 12204] Highest >> >>>>>>> a4 OK [READ-WRITE] Select completed (0.001 secs). >> >>>>>>> >> >>>>>>> a5 search charset utf-8 body "mall" >> >>>>>>> * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 >> 240 >> >>>>>>> 249 >> >>>>>>> 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 >> 385 >> >>>>>>> 389 390 >> >>>>>>> 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 >> 531 >> >>>>>>> 532 533 >> >>>>>>> 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 >> 672 >> >>>>>>> 677 692 >> >>>>>>> 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 >> 821 >> >>>>>>> 833 855 >> >>>>>>> 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 >> 1010 >> >>>>>>> 1011 >> >>>>>>> 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 >> 1137 >> >>>>>>> 1140 >> >>>>>>> 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 >> 1282 >> >>>>>>> 1286 >> >>>>>>> 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 >> 1471 >> >>>>>>> 1494 >> >>>>>>> 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 >> 1606 >> >>>>>>> 1611 >> >>>>>>> 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 >> 1735 >> >>>>>>> 1781 >> >>>>>>> 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 >> 2002 >> >>>>>>> 2006 >> >>>>>>> 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 >> 2208 >> >>>>>>> 2210 >> >>>>>>> 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 >> 2377 >> >>>>>>> 2378 >> >>>>>>> 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 >> 2539 >> >>>>>>> 2551 >> >>>>>>> 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 >> 2754 >> >>>>>>> 2780 >> >>>>>>> 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 >> 2936 >> >>>>>>> 2938 >> >>>>>>> 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 >> 3161 >> >>>>>>> 3178 >> >>>>>>> 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 >> 3386 >> >>>>>>> 3428 >> >>>>>>> 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 >> 3801 >> >>>>>>> 3808 >> >>>>>>> 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 >> 3953 >> >>>>>>> 3954 >> >>>>>>> 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 >> 4059 >> >>>>>>> 4063 >> >>>>>>> 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 >> 4145 >> >>>>>>> 4146 >> >>>>>>> 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 >> 4258 >> >>>>>>> 4259 >> >>>>>>> 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 >> 4502 >> >>>>>>> 4538 >> >>>>>>> 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 >> 4594 >> >>>>>>> 4604 >> >>>>>>> 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 >> 4787 >> >>>>>>> 4788 >> >>>>>>> 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 >> 4901 >> >>>>>>> 4905 >> >>>>>>> 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 >> 5129 >> >>>>>>> 5131 >> >>>>>>> 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 >> 5276 >> >>>>>>> 5277 >> >>>>>>> 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 >> 5391 >> >>>>>>> 5392 >> >>>>>>> 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 >> 5642 >> >>>>>>> 5647 >> >>>>>>> 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 >> 5866 >> >>>>>>> 5867 >> >>>>>>> 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 >> 5958 >> >>>>>>> 5990 >> >>>>>>> 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 >> 6233 >> >>>>>>> 6271 >> >>>>>>> 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 >> 6482 >> >>>>>>> 6492 >> >>>>>>> 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 >> 6852 >> >>>>>>> 6859 >> >>>>>>> 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 >> 7042 >> >>>>>>> 7108 >> >>>>>>> 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 >> 7317 >> >>>>>>> 7322 >> >>>>>>> 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 >> 7417 >> >>>>>>> 7418 >> >>>>>>> 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 >> 7502 >> >>>>>>> 7503 >> >>>>>>> 7558 7588 7589 7628 7685 7695 7699 7703 7723 >> >>>>>>> *a5 OK Search completed (7.846 secs).* >> >>>>>>> >> >>>>>>> >> >>>>>>> Searching against "subject" is pretty fast, few miliseconds ... >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >