dovecot - Jan 2019 - Dovecot and FTS experiment

Hello,

I'm trying to experiment with Dovecot and Solr server.
I have >30k email addresses that I want to index to speed up searching 
and save IOPS on mail servers.
For now - I'm doing some experiments and I'm testing how it is working. 
I'm thinking about adding one additional server with Solr and
configure all mail servers to use that server.

I have some questions.
1. I have 15 mail servers. It will be good If I add new server with Solr 
and use it on all Dovecot servers? Or maybe I should install Solr on all 
mail servers?
2. I notice - I have mail account with 3GB of mail. Index files in mail 
dir has 5MB. After indexing mailbox in Solr - index files has 15MB. What 
changes in those files? FTS indexing adds something to that files - but 
what?
3. If I need to disconnect Dovecot from Solr server - simply Can I 
remove plugin configuration from dovecot.conf and everything would be 
working? Or maybe - I need to remove index files and recreate it? It is 
important for me because if some day I would decide to disconnect FTS 
server - I would have to recreate indexes for >30k mailboxes..
4. What if some day I want to change Solr server to new server with 
empty Solr database? I can simply reindex mailboxes? Or I need to do 
something more?

Thanks!

> On 29 January 2019 at 19:15 Tomasz Nowak <tomek at flup.pl> wrote:
> 
> 
> Hello,
> 
> I'm trying to experiment with Dovecot and Solr server.
> I have >30k email addresses that I want to index to speed up searching 
> and save IOPS on mail servers.
> For now - I'm doing some experiments and I'm testing how it is
working.
> I'm thinking about adding one additional server with Solr and
> configure all mail servers to use that server.
> 
> I have some questions.
> 1. I have 15 mail servers. It will be good If I add new server with Solr 
> and use it on all Dovecot servers? Or maybe I should install Solr on all 
> mail servers?
Possibly, it's hard to say. It depends more about how much mail you have in
total. You might need more than one server. One per backend is definetly too
much.
> 2. I notice - I have mail account with 3GB of mail. Index files in mail 
> dir has 5MB. After indexing mailbox in Solr - index files has 15MB. What 
> changes in those files? FTS indexing adds something to that files - but 
> what?
You can use doveadm dump /path/to/index/directory to view the contents on the
indexes.
> 3. If I need to disconnect Dovecot from Solr server - simply Can I 
> remove plugin configuration from dovecot.conf and everything would be 
> working? Or maybe - I need to remove index files and recreate it? It is 
> important for me because if some day I would decide to disconnect FTS 
> server - I would have to recreate indexes for >30k mailboxes..
You can just stop loading fts.
> 4. What if some day I want to change Solr server to new server with 
> empty Solr database? I can simply reindex mailboxes? Or I need to do 
> something more?
> 
You can run doveadm fts rescan -A and doveadm index -A "*" to perform
full FTS indexing.
> Thanks!
Aki

On 1/29/2019 9:15 AM, Tomasz Nowak wrote:
> Hello,
>
> I'm trying to experiment with Dovecot and Solr server.
> I have >30k email addresses that I want to index to speed up searching 
> and save IOPS on mail servers.
> For now - I'm doing some experiments and I'm testing how it is 
> working. I'm thinking about adding one additional server with Solr and
> configure all mail servers to use that server.
>
> I have some questions.
> 1. I have 15 mail servers. It will be good If I add new server with 
> Solr and use it on all Dovecot servers? Or maybe I should install Solr 
> on all mail servers?
You need to start somewhere. If you've never played with Solr before I 
suggest you start with one and get it working before you explore 
"sharding". When you're ready for that you should consult the solr
mailing list. The importance of enough RAM for Solr cannot be overstated.

> 2. I notice - I have mail account with 3GB of mail. Index files in 
> mail dir has 5MB. After indexing mailbox in Solr - index files has 
> 15MB. What changes in those files? FTS indexing adds something to that 
> files - but what?
What mail storage format are you using?? dbox?

Thinking...I believe that Dovecot records which mails have been reported 
to the FTS.? That may help account for the increased size.


-- 
Daniel

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190129/2d228a35/attachment-0001.html>

W dniu 2019-01-29 19:37, Aki Tuomi napisa?(a):
> Possibly, it's hard to say. It depends more about how much mail you
> have in total. You might need more than one server. One per backend is
> definetly too much.
Great, thank you. I have more than 15TB of mails and I know that one 
Solr server could be not enough.
But it is good that I not need to install Solr od every mail server. On 
that amount of data - I think
that Solr server need more optimalization and I don't want to change 
configuration of mail servers.

> You can use doveadm dump /path/to/index/directory to view the contents
> on the indexes.
Great command, thank you! I didn't know about it.
> You can just stop loading fts.
Super. I like it :)
> You can run doveadm fts rescan -A and doveadm index -A "*" to
perform
> full FTS indexing.
Ok, thank you. I was afraid that there might be more dependencies.

W dniu 2019-01-29 23:48, Daniel Miller via dovecot napisa?(a):
> You need to start somewhere. If you've never played with Solr before I
> suggest you start with one and get it working before you explore
> "sharding". When you're ready for that you should consult the
solr
> mailing list. The importance of enough RAM for Solr cannot be
> overstated.
Ok. Thank you. It's good to know that everything will work good with 
external Solr server.
Most of the howto's that I read - describes local installation of every 
component, so
I need to ensure that there wont be problems.
>> 2. I notice - I have mail account with 3GB of mail. Index files in
>> mail dir has 5MB. After indexing mailbox in Solr - index files has
>> 15MB. What changes in those files? FTS indexing adds something to
>> that files - but what?
> 
> What mail storage format are you using?  dbox?
> 
> Thinking...I believe that Dovecot records which mails have been
> reported to the FTS.  That may help account for the increased size.
Today - maildir. But I have dbox migration in TODO. After another 
answear I know "doveadm dump" command, so I
can check what are in indexes now.