(Note: I am not subscribed to this ML, so please be sure to CC me in replies.) Hello, I see a strange bug when running dovecot under btrfs with a mailbox in mdbox format. At some point, which can take a few minutes or more than a day, dovecot starts logging pread() failures, like so: Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- Mails/dovecot.index.log: Input/output error These would trigger when getmail runs (which in turn is configured to use dovecot-lda). It's only ever the *.index.log files, and it would usually happen to the same one many times in a row. To demonstrate, this is the full list of unique pread() failures on one host (my desktop PC): # journalctl -u dovecot -p 3 -o cat | sort -u | grep pread imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/BTRFS/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/Gentoo/User/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/Newsletters/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/SCons/User/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/storage/dovecot.map.index.log: Input/output error and on the current host (a home server): # journalctl -u dovecot -p 3 -o cat | sort -u | grep pread imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/Privat/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/storage/dovecot.map.index.log: Input/output error Rebooting the system would make it go away (without loss of mails), though remounting the file system should have worked, too. I tried a variety of options, but the only one that helped was to convert the mailbox from mdbox to maildir. I have not seen any pread() failures in the two weeks since. I originally wrote to the btrfs ML, see the thread at [0], in particular my messages [1] and [2], for additional details, including references to what little I could find on the web. However, nothing much came of that thread, so I am now posting here. My dovecot configuration: # doveconf -n # 2.2.19: /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.9 (357ac0a0e68b+) # OS: Linux 4.4.8-gentoo x86_64 Gentoo Base System release 2.2 auth_mechanisms = plain login mail_location = maildir:~/.maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { args = * driver = pam } plugin { sieve = file:~/.sieve;active=~/.dovecot.sieve } postmaster_address = root at diefledermaus protocols = imap sieve service imap-login { inet_listener imap { port = 10087 } inet_listener imaps { port = 10887 } } ssl = required ssl_cert = </etc/ssl/dovecot/dovecot.pem ssl_key = </etc/ssl/dovecot/dovecot.key userdb { driver = passwd } protocol lda { auth_socket_path = /var/run/dovecot/auth-master mail_plugins = sieve } protocol sieve { managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date } I used to use claws-mail, then switched to KMail in September 2015. So the errors happened with both clients. [0] http://thread.gmane.org/gmane.comp.file-systems.btrfs/49637 [1] http://article.gmane.org/gmane.comp.file-systems.btrfs/49647 [2] http://article.gmane.org/gmane.comp.file-systems.btrfs/55552 Greetings -- Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20160505/491893a7/attachment-0001.sig>
On 05 May 2016, at 16:21, Marc Joliet <marcec at gmx.de> wrote:> > (Note: I am not subscribed to this ML, so please be sure to CC me in replies.) > > Hello, > > I see a strange bug when running dovecot under btrfs with a mailbox in mdbox > format. At some point, which can take a few minutes or more than a day, > dovecot starts logging pread() failures, like so: > > Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed with > file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- > Mails/dovecot.index.log: Input/output error > > These would trigger when getmail runs (which in turn is configured to use > dovecot-lda). It's only ever the *.index.log files, and it would usually > happen to the same one many times in a row. To demonstrate, this is the full > list of unique pread() failures on one host (my desktop PC):*.index.log files are always appended to using O_APPEND flag. Maybe this is relevant. Also when a new .log file is created it's opened without the O_APPEND flag and the O_APPEND is added later. This was causing a bug recently in unionfs, which ignored the flag change and caused log file corruption.> Rebooting the system would make it go away (without loss of mails), though > remounting the file system should have worked, too. I tried a variety of > options, but the only one that helped was to convert the mailbox from mdbox to > maildir. I have not seen any pread() failures in the two weeks since.I can't really think of why Maildir format would get rid of the problem, since the dovecot.index.log files still exist and they're accessed the same way. Maybe with Maildir there are enough locks that it prevents some race condition that triggers this. You could try stress testing with imaptest to see if it's easy to reproduce: http://imapwiki.org/ImapTest
On Friday 06 May 2016 01:51:34 Timo Sirainen wrote:>On 05 May 2016, at 16:21, Marc Joliet <marcec at gmx.de> wrote: >> (Note: I am not subscribed to this ML, so please be sure to CC me in >> replies.) >> >> Hello, >> >> I see a strange bug when running dovecot under btrfs with a mailbox in >> mdbox >> format. At some point, which can take a few minutes or more than a day, >> dovecot starts logging pread() failures, like so: >> >> Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed >> with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- >> Mails/dovecot.index.log: Input/output error >> >> These would trigger when getmail runs (which in turn is configured to use >> dovecot-lda). It's only ever the *.index.log files, and it would usually >> happen to the same one many times in a row. To demonstrate, this is the >> full >> list of unique pread() failures on one host (my desktop PC): >*.index.log files are always appended to using O_APPEND flag. Maybe this is >relevant. > >Also when a new .log file is created it's opened without the O_APPEND flag >and the O_APPEND is added later. This was causing a bug recently in unionfs, >which ignored the flag change and caused log file corruption.I will mention that in the btrfs ML thread. Who knows, maybe it will lead to something.>> Rebooting the system would make it go away (without loss of mails), though >> remounting the file system should have worked, too. I tried a variety of >> options, but the only one that helped was to convert the mailbox from mdbox >> to maildir. I have not seen any pread() failures in the two weeks since. >I can't really think of why Maildir format would get rid of the problem, >since the dovecot.index.log files still exist and they're accessed the same >way. Maybe with Maildir there are enough locks that it prevents some race >condition that triggers this. > >You could try stress testing with imaptest to see if it's easy to reproduce: >http://imapwiki.org/ImapTestI will experiment with that when I have some more time. Thanks for the suggestions! Greetings -- Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20160507/4e7cec2c/attachment-0001.sig>
Am Friday 27 May 2016 schrieb Timo Sirainen <tss at iki.fi>>On 05 May 2016, at 16:21, Marc Joliet <marcec at gmx.de> wrote:[...]>> Rebooting the system would make it go away (without loss of mails), though >> remounting the file system should have worked, too. I tried a variety of >> options, but the only one that helped was to convert the mailbox from mdbox >> to maildir. I have not seen any pread() failures in the two weeks since. >I can't really think of why Maildir format would get rid of the problem, >since the dovecot.index.log files still exist and they're accessed the same >way. Maybe with Maildir there are enough locks that it prevents some race >condition that triggers this.Yeah, I was somewhat surprised even without knowing that. And what do you know: two days ago I discovered pread() failures in my system log again, but this time they went away by themselves without me noticing that they even occurred: Mai 23 18:03:45 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:03:45 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:13:57 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:13:57 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error There were also a whole bunch affecting the same file on May 20th, but not before. I probably didn't notice those because they started shortly before I rebooted for a kernel upgrade (from gentoo-sources 4.4.10 to 4.5.5).>You could try stress testing with imaptest to see if it's easy to reproduce: >http://imapwiki.org/ImapTestI still haven't had time to do this, but haven't forgotten! Greetings -- Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20160528/65eade36/attachment.sig>
Seemingly Similar Threads
- Re: [PATCH nbdkit v2 1/2] ocaml: Change pread method to avoid leaking heap memory.
- Fwd: [PATCH (geoip)] use IO.pread from the io-extra lib if possible
- [PATCH] python: Support buffer protocol for pread() result
- 1.2.6 Spurious pread()/stream errors/index corruption
- Re: [PATCH nbdkit v2 2/2] server: Use a thread-local pread/pwrite buffer to avoid leaking heap data.