On 15 Jul 2019, at 23.17, Ralf Hildebrandt via dovecot <dovecot at dovecot.org> wrote:> > * Ralf Hildebrandt via dovecot <dovecot at dovecot.org>: >> We're using dovecot (via LMTP) as a backup for all incoming mail. >> >> I upgraded from 2.3.6 to 2.3.7 on the 12th: >> 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 2:2.3.7-8~bionic >> >> and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup >> IMAP box is suddenly taking quite some time to get delivered and is piling up >> in the queue. > > And alas, I had a look at the monitoring and found that disk IO has > increase A LOT after the update: https://www.arschkrebs.de/images/dovecot-disk-io-increase.png > > I zoomed in and found that the sudden increace in IO coincides with > the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png <https://www.arschkrebs.de/images/dovecot-io-2.png>Do the different disks have different kinds of data? Like you seem to be using external attachments and fts? Also your doveconf -n doesn't show what fts backend you're using. Or is it just half-configured and not really used? -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190716/f4f1f48e/attachment.html>
* Timo Sirainen via dovecot <dovecot at dovecot.org>:> > And alas, I had a look at the monitoring and found that disk IO has > > increase A LOT after the update: https://www.arschkrebs.de/images/dovecot-disk-io-increase.png > > > > I zoomed in and found that the sudden increace in IO coincides with > > the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png <https://www.arschkrebs.de/images/dovecot-io-2.png> >> Do the different disks have different kinds of data?All of the data is an /dev/sda1> Like you seem to be using external attachments and fts?Jup, using that, but on one disk> Also your doveconf -n > doesn't show what fts backend you're using. Or is it just > half-configured and not really used?I used to use lucene, but then I had issues building it and commented it out. Looking now, it turns out that your packages come with lucene support... Reenabling: mail_plugins = zlib fts fts_lucene ... plugin { fts = lucene fts_autoindex = yes fts_languages = de,en fts_lucene = whitespace_chars=@. zlib_save = gz zlib_save_level = 5 } -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | https://www.charite.de
On 16 Jul 2019, at 11.15, Ralf Hildebrandt via dovecot <dovecot at dovecot.org> wrote:> > * Timo Sirainen via dovecot <dovecot at dovecot.org>: > >>> And alas, I had a look at the monitoring and found that disk IO has >>> increase A LOT after the update: https://www.arschkrebs.de/images/dovecot-disk-io-increase.png >>> >>> I zoomed in and found that the sudden increace in IO coincides with >>> the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png <https://www.arschkrebs.de/images/dovecot-io-2.png> >> > >> Do the different disks have different kinds of data? > > All of the data is an /dev/sda1What filesystem is this? I did a bunch of testing, and after initially thinking I saw an increase it was just due to bad testing because the new TCP_NODELAY changes allowed imaptest to do more work. So I can't see that there is any disk IO difference. There are a few intentional changes to lib-index and also mdbox code. One of those is that dovecot.index.cache files are being recreated more often now. In some cases they were wasting a lot of disk space because expunged mails hadn't been removed from them. I guess it's a possibility that all the disk IO was because in your system Dovecot started recreating all those files at the same time. Maybe you could try upgrading again during slower hours and see if the disk IO normalizes after a while?