Duc Anh Do
2021-Apr-22 07:02 UTC
Size of transaction log file is increasing continuously if dsync is failed
Hi Dovecot community, I would like to share a problem I saw a few days ago. Any comment is highly appreciated. *Steps to reproduce* 0. Version of Dovecot I use is v2.3.11.3. 1. Prepare 2 Dovecot backends: - One backend is used as the source of dsync command. Other backend is used as the destination of the dsync command. - Mailbox format is Maildir. Mail dir of two test accounts is empty. 2. Prepare the test account in the source of dsync: - Send a test message to the test account. For example, I use swaks: swaks --server 127.0.0.1 --port 24 --protocol LMTP --to user1 at local - Use doveadm expunge command to clean mail dir of the test account: doveadm expunge -u user1 at local mailbox INBOX all 3. Prepare the test account in the destination of dsync: - Log in test account via IMAP then select INBOX first: telnet 127.0.0.1 imap ... s login user1 at local 1 ... s select INBOX ... s logout - Then perform dsync: doveadm -o imapc_host=backend2.local -o imapc_port=143 -o imapc_user=user1 at local -o imapc_password=1 backup -s '' -R -u user1 at local imapc: - Confirm error occurs when using dsync: dsync(user1 at local): Info: imapc(backend2.local:143): Connected to 192.168.100.2:143 (local 192.168.100.27:58850) dsync(user1 at local): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted. AQAAAHm4+Jk4. Log in the test account in dsync destination then select INBOX. Repeat many times. I can see the size of the transaction log file is increasing continuously. *Before*: -rw-------. 1 vmail vmail 456 Apr 22 10:54 dovecot.index.log *After*: -rw-------. 1 vmail vmail 544 Apr 22 10:59 dovecot.index.log -rw-------. 1 vmail vmail 632 Apr 22 11:00 dovecot.index.log -rw-------. 1 vmail vmail 720 Apr 22 11:00 dovecot.index.log *Problems* - Size of the transaction log file is increasing continuously. Do you think this is an issue of Dovecot? - The value of *mail_index_log_rotate_max_size* is 1M by default. However the transaction log file dovecot.index.log is not rotated when its size exceeds limit. - Because the size of the transaction log file is too big and it is not rotated, I think "out of memory" error will occur when I try to use IMAP SELECT or POP3 STAT command (if the virtual size limit is small enough). - If "size of the transaction log file is increasing continuously" is an issue, what should I do to avoid it or is there any workaround? I understand I should fix the error *Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted* first before doing anything next. I think removing the transaction log file is an understandable workaround. *Investigaton* - About the error *Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted*, I can understand the cause after reading this thread https://dovecot.org/pipermail/dovecot/2014-April/095588.html 2) Maildir + INBOX + backup/sync/replicate> In the test scenarios where the INBOX on one side was to be completely > removed, e.g. doveadm backup -R the dsync failed and > nothing was synced to the target. This is because before moving the source > mails to the mailbox, dsync cleans out the old ones ( > -R preserves nothing) and in Maildir the INBOX can not be removed. This is > a feature/not easily solvable, because in Maildir > INBOX is different from other folders. >- The transaction log file is rotated whenever there's any change in index file. I can confirm by sending a test message to the test account in dsync destination. I can see that logic in *src/lib-index/mail-index-sync.c* and *src/lib-index/mail-index-write.c*. After that, the size of the new transaction log file is not increasing anymore. - When the error *Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted* occurred, I tried to dump the transaction log file and I saw these records: record: offset=432, type=index-deleted (ext), size=12 record: offset=444, type=index-undeleted (ext), size=12 - Do you think these records relate to the issue that the size of the transaction log file is increasing continuously? - Did the error *Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted* make content of transaction log file incomplete/incorrect? Regards, Anh Do -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20210422/ae5c4d65/attachment-0001.html>