dovecot - Aug 2018 - [trees-plugin] - Dovecot index gets corrupted, when using maildir and recievend and accessing mail at the same time

Hello dovecot team,

at first I want to thank you the awesome dovecot software.

I'm currently researching and developing the trees plugin
(https://0xacab.org/riseuplabs/trees). In the future I would love to
package this software for debian. I run into a problem, which I can not
fix by myself, that's why I'm reaching out to you.

The plugin is producing a reproducible error, when using dovecots
mailbox format maildir, and receiving and accessing mail at the same
time. I'm searching for additional information to fix that.

I know that it's may not the scope of the mailing list to support
external plugins, but since you are the experts, I thought I should ask
at least.

Trees Plugin:
==================
The plugin adds individually encrypted mail storage to the Dovecot IMAP
server. It encrypts all (incoming and outgoing) mail on the mail server
with a key, which can only be accessed with the users password. So the
server operator is not able to see any mail content, if it's saved on
the hard disk.


Bug Description:
==================
I'm using a postfix dovecot setup with the enabled trees plugin with the
mailboxformat maildir.
When I'm receiving and accessing mail at the same time, the dovecot
index gets corrupted. For accessing those mail I use Thunderbird.

Example corruption:
> ug  4 09:55:15 trees dovecot: imap-login: Login: user=<treesenabled at
trees.testing>, method=PLAIN, rip=192.168.22.1, lip=192.168.22.77, mpid=4643,
TLS, session=<w0OCCplyhLLAqBYB>
> Aug  4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Debug:
trees plugin initialized
> Aug  4 09:55:15 trees dovecot: imap(treesenabled at trees.testing):
Connection closed in=506 out=495
> Aug  4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Error:
Maildir filename has wrong S value, renamed the file from
/var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2,
to
/var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=783:2,
> Aug  4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Error:
Corrupted index cache file
/var/vmail/trees.testing/treesenabled/Maildir/dovecot.index.cache: Broken
physical size for mail UID 1 in mailbox INBOX:
read(/var/vmail/trees.testing/treesenabled/Maildir/cur/1533376515.M379290P4641.trees,S=1456,W=748:2,)
failed: Cached message size larger than expected (1456 > 728, box=INBOX,
UID=1, cached Message-Id=<4ae8ea2c-17ac-37d5-9b9e-b32c6edc777a at
trees.testing>)
> Aug  4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Panic:
file istream.c: line 175 (i_stream_read): assertion failed: (old_size <=
_stream->pos - _stream->skip)
> Aug  4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Error:
Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7fa88e2afe92] ->
/usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7fa88e2aff8d] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fa88e245a91] ->
/usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x290) [0x7fa88e2baf20] ->
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7fa88e2bb72d] ->
/usr/lib/dovecot/libdovecot.so.0(message_get_body_size+0xfd) [0x7fa88e29ab6d]
-> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x220)
[0x7fa88e5eb3d0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66fc0)
[0x7fa88e5a2fc0] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x5d)
[0x7fa88e572e1d] ->
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x536)
[0x7fa88e6264f6] -> dovecot-trees.testing/imap(+0x1ebbc) [0x559b794f1bbc]
-> dovecot-trees.testing/imap(+0x1cfb6) [0x559b794effb6] ->
dovecot-trees.testing/imap(imap_fetch_more+0x39) [0x559b794f10e9] ->
dovecot-trees.testing/imap(cmd_fetch+0x33b) [0x559b794e2d9b] ->
dovecot-trees.testing/imap(command_exec+0xa5) [0x559b794ee735] ->
dovecot-trees.testing/imap(+0x199c2) [0x559b794ec9c2] ->
dovecot-trees.testing/imap(+0x19a4c) [0x559b794eca4c] ->
dovecot-trees.testing/imap(client_handle_input+0x1b5) [0x559b794ece55] ->
dovecot-trees.testing/imap(client_input+0x86) [0x559b794ed3c6] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7fa88e2c49f2] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109)
[0x7fa88e2c6029] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7fa88e2c4a8c]
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fa88e2c4c38] ->
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fa88e24bfd3] ->
dovecot-trees.testing/imap(main+0x328) [0x559b794dfe68] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fa88de9b2e1] ->
dovecot-trees.testing/imap(_start+0x2a) [0x559b794dffea]
> Aug  4 09:55:15 trees dovecot: imap(treesenabled at trees.testing): Fatal:
master: service(imap): child 4633 killed with signal 6 (core dumps disabled)

The behavior does not occur, when I'm using maildir and access my mail
after receiving it. It also does not occur with other mailbox formats
such as mbox, sdbox and mdbox

I opened a bug report at the trees repository, which is more detailed (
https://0xacab.org/riseuplabs/trees/issues/24 ). There I described how
it's possible to reproduce this error.

In the trees repo there is also a vagrant box, with which it's possible
to spin up the exact same setup.

Questions:
==================
1.) Do you know what could possibly lead to the the larger cached
message size, when accessing this mail via imap?

2.) Why does these sizes differ from each other? Mail size from the log
is: 'S=1456,W=748:2' or 'renamed to ...S=783:2'. But I do not
see where
the size of '728' come from?

3.) Do you have a hint what could lead to this behavior and how could I
solve this error?

Unrelated but also interesting for trees usage:
4.) Is there a tool, with which you can export and import mail to the
same inbox?

Background: After activation for an existing mail user, only new
outgoing and incoming mail is encrypted. So it would be necessary to
"re-save" the existing mail again in this mailbox, to pipe all mails
through the encryption process.

I know that there is dsync, imapsync and several different tools, but
I'm not sure what is the best tool for the job, or if I should use dsync
to do a 3 way sync?
> existing_mailbox at server.com  > tmp_mailbox at server.com >
existing_mailbox at server.com



Thank you:
==================
Thank you for time reading this mail :)


Dovecot Config:
==================
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4 ext4
auth_mechanisms = plain login
instance_name = trees.testing
login_greeting = trees.testing ready.
mail_gid = 5000
mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir
mail_plugins = " quota trees"
mail_uid = 5000
mailbox_list_index = yes
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Junk {
    auto = subscribe
    autoexpunge = 30 days
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    auto = subscribe
    autoexpunge = 30 days
    special_use = \Trash
  }
  prefix }
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
protocols = " imap pop3"
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0660
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 4
}
service imap {
  executable = imap
}
service pop3 {
  executable = pop3
}
ssl = required
ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
ssl_cipher_list = ALL:!LOW:!RC4:!EXP:!aNULL
ssl_dh_parameters_length = 2048
ssl_key =  # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
userdb {
  driver = prefetch
}
userdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180809/522f9cb9/attachment.sig>

Follow up:

cmouse from #dovecot pointed out, that this was a dovecot bug which is
already fixed in newer dovecot versions. Thank you!

With dovecot 2.2.34 (current debian backports version), I can not
reproduce the error.

So my last, "bug unrelated" question remains:
> 4.) Is there a tool, with which you can export and import mail to the
> same inbox?
> 
> Background: After activation for an existing mail user, only new
> outgoing and incoming mail is encrypted. So it would be necessary to
> "re-save" the existing mail again in this mailbox, to pipe all
mails
> through the encryption process.
> 
> I know that there is dsync, imapsync and several different tools, but
> I'm not sure what is the best tool for the job, or if I should use
dsync
> to do a 3 way sync?
> 
>> existing_mailbox at server.com  > tmp_mailbox at server.com >
existing_mailbox at server.com
> 
Thanks again to cmouse.

cheers neutron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180809/22559161/attachment.sig>

Op 09/08/2018 om 19:29 schreef neutron:
> Follow up:
>
> cmouse from #dovecot pointed out, that this was a dovecot bug which is
> already fixed in newer dovecot versions. Thank you!
>
> With dovecot 2.2.34 (current debian backports version), I can not
> reproduce the error.
>
> So my last, "bug unrelated" question remains:
>
>> 4.) Is there a tool, with which you can export and import mail to the
>> same inbox?
>>
>> Background: After activation for an existing mail user, only new
>> outgoing and incoming mail is encrypted. So it would be necessary to
>> "re-save" the existing mail again in this mailbox, to pipe
all mails
>> through the encryption process.
>>
>> I know that there is dsync, imapsync and several different tools, but
>> I'm not sure what is the best tool for the job, or if I should use
dsync
>> to do a 3 way sync?
Maybe this could help:

https://wiki1.dovecot.org/Plugins/Convert

No idea whether it would work with that plugin though.

Regards,

Stephan.

On Thu, 9 Aug 2018, neutron wrote:
> The plugin adds individually encrypted mail storage to the Dovecot IMAP
> server. It encrypts all (incoming and outgoing) mail on the mail server
> with a key, which can only be accessed with the users password. So the
> server operator is not able to see any mail content, if it's saved on
> the hard disk.
Another privacy plugin that assumes the server operator is unmotivated or
respects your privacy anyways, and won't just skim your password right off
the top to look at your mail.  A vault with steel walls and a dirt floor.

Joseph Tam <jtam.home at gmail.com>

Quoting Joseph Tam <jtam.home at gmail.com>:
> Another privacy plugin that assumes the server operator is unmotivated or
> respects your privacy anyways, and won't just skim your password right
off
> the top to look at your mail.  A vault with steel walls and a dirt floor.
*SIGH* As usual, you're right on the money, Joseph.

I used to let things like this "slide", but somewhat recently I've
had some
clients badgering me to implement something like this. It takes longer than it
should to explain how pointless the exercise is.

Given that:

1) Email transactions, from submission, to delivery, to final reception by a
MUA, are done with plaintext contents. Those who want security, will undergo
the additional steps and hassles with using PGP to encrypt the contents,
providing the only demonstrably secure (against "Evil SysAdmins")
means of
cloaking your content. The submission, delivery, and final reception is still
performed as "plaintext", albeit with an attachment that is encrypted,
a
process done (and undone) by the ultimate endpoint clients.

2) Even if the "Evil SysAdmin" doesn't scribble all of the
users' passphrases
into a log, it's trivial for various tools, many of which were hastily
cobbled
together during the fad of implementing Sarbanes-Oxley Act (SOX) compliance on
mail servers. Tools like "milter-bcc" and friends which automatically
clone
all email submitted to or arriving through SMTP, etc. It doesn't matter if
your SMTP software implements 65,536 Jiggabyte Key Quantum-Computing-Resistant
crypto, when it has the decrypted contents in its spool.

I imagine this is an exercise in buzzword collection, and to be seen to be
"doing something" to improve security and/or privacy.

If privacy is desired, there are only end-to-end encryption/signature schemes
to ensure anything at all, and even there we're at the mercy of mathematical
gods greater than we.

Looking to a "magical" oracle on your server to do it for you, whilst
keeping
all of the leaky, plaintext, and promiscuous protocols (DSN, bounces,
intermediate MXer hosts that eruct contents to various envelope addresses,
etc) that will betray you behind your back without a moment's notice is a
Fool's Errand.

Think it over.

=M=