(Note: I am not subscribed to this ML, so please be sure to CC me in replies.)
Hello,
I see a strange bug when running dovecot under btrfs with a mailbox in mdbox
format. At some point, which can take a few minutes or more than a day,
dovecot starts logging pread() failures, like so:
Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed with
file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox-
Mails/dovecot.index.log: Input/output error
These would trigger when getmail runs (which in turn is configured to use
dovecot-lda). It's only ever the *.index.log files, and it would usually
happen to the same one many times in a row. To demonstrate, this is the full
list of unique pread() failures on one host (my desktop PC):
# journalctl -u dovecot -p 3 -o cat | sort -u | grep pread
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/mailboxes/BTRFS/dbox-Mails/dovecot.index.log: Input/output
error
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/mailboxes/Gentoo/User/dbox-Mails/dovecot.index.log:
Input/output error
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: Input/output
error
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox-Mails/dovecot.index.log:
Input/output error
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/mailboxes/Newsletters/dbox-Mails/dovecot.index.log:
Input/output error
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/mailboxes/SCons/User/dbox-Mails/dovecot.index.log:
Input/output error
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/storage/dovecot.map.index.log: Input/output error
and on the current host (a home server):
# journalctl -u dovecot -p 3 -o cat | sort -u | grep pread
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/mailboxes/Privat/dbox-Mails/dovecot.index.log:
Input/output error
imap(marcec): Error: pread() failed with file
/home/marcec/.mdbox/storage/dovecot.map.index.log: Input/output error
Rebooting the system would make it go away (without loss of mails), though
remounting the file system should have worked, too. I tried a variety of
options, but the only one that helped was to convert the mailbox from mdbox to
maildir. I have not seen any pread() failures in the two weeks since.
I originally wrote to the btrfs ML, see the thread at [0], in particular my
messages [1] and [2], for additional details, including references to what
little I could find on the web. However, nothing much came of that thread, so
I am now posting here.
My dovecot configuration:
# doveconf -n
# 2.2.19: /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.9 (357ac0a0e68b+)
# OS: Linux 4.4.8-gentoo x86_64 Gentoo Base System release 2.2
auth_mechanisms = plain login
mail_location = maildir:~/.maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
separator = /
}
passdb {
args = *
driver = pam
}
plugin {
sieve = file:~/.sieve;active=~/.dovecot.sieve
}
postmaster_address = root at diefledermaus
protocols = imap sieve
service imap-login {
inet_listener imap {
port = 10087
}
inet_listener imaps {
port = 10887
}
}
ssl = required
ssl_cert = </etc/ssl/dovecot/dovecot.pem
ssl_key = </etc/ssl/dovecot/dovecot.key
userdb {
driver = passwd
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
mail_plugins = sieve
}
protocol sieve {
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date
}
I used to use claws-mail, then switched to KMail in September 2015. So the
errors happened with both clients.
[0] http://thread.gmane.org/gmane.comp.file-systems.btrfs/49637
[1] http://article.gmane.org/gmane.comp.file-systems.btrfs/49647
[2] http://article.gmane.org/gmane.comp.file-systems.btrfs/55552
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20160505/491893a7/attachment-0001.sig>
On 05 May 2016, at 16:21, Marc Joliet <marcec at gmx.de> wrote:> > (Note: I am not subscribed to this ML, so please be sure to CC me in replies.) > > Hello, > > I see a strange bug when running dovecot under btrfs with a mailbox in mdbox > format. At some point, which can take a few minutes or more than a day, > dovecot starts logging pread() failures, like so: > > Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed with > file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- > Mails/dovecot.index.log: Input/output error > > These would trigger when getmail runs (which in turn is configured to use > dovecot-lda). It's only ever the *.index.log files, and it would usually > happen to the same one many times in a row. To demonstrate, this is the full > list of unique pread() failures on one host (my desktop PC):*.index.log files are always appended to using O_APPEND flag. Maybe this is relevant. Also when a new .log file is created it's opened without the O_APPEND flag and the O_APPEND is added later. This was causing a bug recently in unionfs, which ignored the flag change and caused log file corruption.> Rebooting the system would make it go away (without loss of mails), though > remounting the file system should have worked, too. I tried a variety of > options, but the only one that helped was to convert the mailbox from mdbox to > maildir. I have not seen any pread() failures in the two weeks since.I can't really think of why Maildir format would get rid of the problem, since the dovecot.index.log files still exist and they're accessed the same way. Maybe with Maildir there are enough locks that it prevents some race condition that triggers this. You could try stress testing with imaptest to see if it's easy to reproduce: http://imapwiki.org/ImapTest
On Friday 06 May 2016 01:51:34 Timo Sirainen wrote:>On 05 May 2016, at 16:21, Marc Joliet <marcec at gmx.de> wrote: >> (Note: I am not subscribed to this ML, so please be sure to CC me in >> replies.) >> >> Hello, >> >> I see a strange bug when running dovecot under btrfs with a mailbox in >> mdbox >> format. At some point, which can take a few minutes or more than a day, >> dovecot starts logging pread() failures, like so: >> >> Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed >> with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- >> Mails/dovecot.index.log: Input/output error >> >> These would trigger when getmail runs (which in turn is configured to use >> dovecot-lda). It's only ever the *.index.log files, and it would usually >> happen to the same one many times in a row. To demonstrate, this is the >> full >> list of unique pread() failures on one host (my desktop PC): >*.index.log files are always appended to using O_APPEND flag. Maybe this is >relevant. > >Also when a new .log file is created it's opened without the O_APPEND flag >and the O_APPEND is added later. This was causing a bug recently in unionfs, >which ignored the flag change and caused log file corruption.I will mention that in the btrfs ML thread. Who knows, maybe it will lead to something.>> Rebooting the system would make it go away (without loss of mails), though >> remounting the file system should have worked, too. I tried a variety of >> options, but the only one that helped was to convert the mailbox from mdbox >> to maildir. I have not seen any pread() failures in the two weeks since. >I can't really think of why Maildir format would get rid of the problem, >since the dovecot.index.log files still exist and they're accessed the same >way. Maybe with Maildir there are enough locks that it prevents some race >condition that triggers this. > >You could try stress testing with imaptest to see if it's easy to reproduce: >http://imapwiki.org/ImapTestI will experiment with that when I have some more time. Thanks for the suggestions! Greetings -- Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20160507/4e7cec2c/attachment-0001.sig>
Am Friday 27 May 2016 schrieb Timo Sirainen <tss at iki.fi>>On 05 May 2016, at 16:21, Marc Joliet <marcec at gmx.de> wrote:[...]>> Rebooting the system would make it go away (without loss of mails), though >> remounting the file system should have worked, too. I tried a variety of >> options, but the only one that helped was to convert the mailbox from mdbox >> to maildir. I have not seen any pread() failures in the two weeks since. >I can't really think of why Maildir format would get rid of the problem, >since the dovecot.index.log files still exist and they're accessed the same >way. Maybe with Maildir there are enough locks that it prevents some race >condition that triggers this.Yeah, I was somewhat surprised even without knowing that. And what do you know: two days ago I discovered pread() failures in my system log again, but this time they went away by themselves without me noticing that they even occurred: Mai 23 18:03:45 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:03:45 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:13:57 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:13:57 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error There were also a whole bunch affecting the same file on May 20th, but not before. I probably didn't notice those because they started shortly before I rebooted for a kernel upgrade (from gentoo-sources 4.4.10 to 4.5.5).>You could try stress testing with imaptest to see if it's easy to reproduce: >http://imapwiki.org/ImapTestI still haven't had time to do this, but haven't forgotten! Greetings -- Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20160528/65eade36/attachment.sig>
Possibly Parallel Threads
- Re: [PATCH nbdkit v2 1/2] ocaml: Change pread method to avoid leaking heap memory.
- Fwd: [PATCH (geoip)] use IO.pread from the io-extra lib if possible
- [PATCH] python: Support buffer protocol for pread() result
- 1.2.6 Spurious pread()/stream errors/index corruption
- Re: [PATCH nbdkit v2 2/2] server: Use a thread-local pread/pwrite buffer to avoid leaking heap data.