dovecot - Oct 2013 - Plugin issue with update from 2.0.19 to 2.1.17

Hello. Probably only Timo can help-me with this.

I have a self-made plugin based on the zlib plugin that i use to cryptograph the
messages at inbox.

As a side-effect of the cryptography, my plugin changes the size of the message,
but until 2.0.19 this works well with dovecot index and the W/S flags.

But now, i'm going to upgrade to 2.1.17 and now i have these messages on log
at my test ambiance:

Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error: Cached
message size smaller than expected (367 < 529)
Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error: Maildir
filename has wrong S value, renamed the file from
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S
to
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S
Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error: Corrupted
index cache file
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache:
Broken physical size for mail UID 6

I understood with these messages that the dovecot is arguing to get the real
size of the file now, overriding the return of size from my crypto plugin. But i
don't understood if this is a consequence from the changes to correct the
index issue (the dovecot.index.cache issue).

So... I want know if the correct way to fix this would be change my plugin to
return the real size of the file (that will be larger than the effective message
that the plugin returne after the de-cryptography) or if i need use some new 
function to the plugin replace the expected size based on the real size of the
message, not of the file.

Thanks

Reinaldo

On Oct 15, 2013, at 4:33 PM, Reinaldo Matukuma wrote:
> Hello. Probably only Timo can help-me with this.
> 
> I have a self-made plugin based on the zlib plugin that i use to
cryptograph the messages at inbox.
> 
> As a side-effect of the cryptography, my plugin changes the size of the
message, but until 2.0.19 this works well with dovecot index and the W/S flags.
> 
> But now, i'm going to upgrade to 2.1.17 and now i have these messages
on log at my test ambiance:
> 
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Cached message size smaller than expected (367 < 529)
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Maildir filename has wrong S value, renamed the file from
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S
to
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Corrupted index cache file
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache:
Broken physical size for mail UID 6
I think there's something else up, this looks quite similar to what I
reported the other day:

http://dovecot.org/list/dovecot/2013-October/092917.html

Do you get further messages after the "broken physical size"
indicating that dovecot is then trying to open the file based on the original
filename?

Regardless, setting this should turn off the file size/name corrections, perhaps
that will help:

maildir_broken_filename_sizes=yes

Charles
> 
> I understood with these messages that the dovecot is arguing to get the
real size of the file now, overriding the return of size from my crypto plugin.
But i don't understood if this is a consequence from the changes to correct
the index issue (the dovecot.index.cache issue).
> 
> So... I want know if the correct way to fix this would be change my plugin
to return the real size of the file (that will be larger than the effective
message that the plugin returne after the de-cryptography) or if i need use some
new  function to the plugin replace the expected size based on the real size of
the message, not of the file.
> 
> Thanks
> 
> Reinaldo
>

Humm... Looking deeper on my plugin i saw that have a call to the
i_stream_create_header_filter:

full_input[0] = i_stream_create_header_filter(input, HEADER_FILTER_EXCLUDE |
HEADER_FILTER_NO_CR, exclude_headers, 3, filter_callback,
&zuser->hash_value);

I need this because i really have 3 lines on the header that i need to hide from
the client...

And i have a call to i_stream_create_concat too:

imail->data.stream = i_stream_create_concat(full_input);

Because i have a header and body separation too just for the fact that the
header could be searched via dovecot cache once it remains un-cryptographed.

So... The size of S flag before dovecot change it is exactly the header size
minus 3 lines of the header plus the un-cryptographed body content.

I notice too two things:

- that dovecot does the rename of the file and can open the message (including
the un-cryptographed body part) but removes the W flag on the filename.
Isn't the W flag used anymore?

- that the size used by dovecot when it renames the file is only the size of the
header part, excluding the body part... I did a test with a larger message and
got this at the log:


Oct 17 17:24:27 test dovecot: imap(reinaldo at exemplo.com.br): Error: Cached
message size larger than expected (3472 > 1010)
Oct 17 17:24:27 test dovecot: imap(reinaldo at exemplo.com.br): Error: Maildir
filename has wrong S value, renamed the file from
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/cur/1382041442.M605987P2439V2051I475262.test,S=3472,W=3538:2,
to
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/cur/1382041442.M605987P2439V2051I475262.test,S=1010:2,
Oct 17 17:24:27 test dovecot: imap(reinaldo at exemplo.com.br): Error: Corrupted
index cache file
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/dovecot.index.cache:
Broken physical size for mail UID 18
Oct 17 17:24:27 test dovecot: imap(reinaldo at exemplo.com.br): Error: read()
failed: Input/output error (FETCH for mailbox INBOX UID 18)
Oct 17 17:24:27 test dovecot: imap(reinaldo at exemplo.com.br): Disconnected:
Internal error occurred. Refer to server log for more information. [2013-10-17
17:24:27] in=339 out=2915

So... I'm assuming that on dovecot 2.0.19 the index.cache was indexing all
the header and un-cryptographed body, really using the information on S and W
flags.

Does the dovecot 2.1.17 index work in another way, needing to index only the
header part of the message?


> From: ramatukuma at hotmail.com
> To: dovecot at dovecot.org
> Date: Tue, 15 Oct 2013 17:33:44 -0300
> Subject: [Dovecot] Plugin issue with update from 2.0.19 to 2.1.17
> 
> Hello. Probably only Timo can help-me with this.
> 
> I have a self-made plugin based on the zlib plugin that i use to
cryptograph the messages at inbox.
> 
> As a side-effect of the cryptography, my plugin changes the size of the
message, but until 2.0.19 this works well with dovecot index and the W/S flags.
> 
> But now, i'm going to upgrade to 2.1.17 and now i have these messages
on log at my test ambiance:
> 
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Cached message size smaller than expected (367 < 529)
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Maildir filename has wrong S value, renamed the file from
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S
to
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Corrupted index cache file
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache:
Broken physical size for mail UID 6
> 
> I understood with these messages that the dovecot is arguing to get the
real size of the file now, overriding the return of size from my crypto plugin.
But i don't understood if this is a consequence from the changes to correct
the index issue (the dovecot.index.cache issue).
> 
> So... I want know if the correct way to fix this would be change my plugin
to return the real size of the file (that will be larger than the effective
message that the plugin returne after the de-cryptography) or if i need use some
new  function to the plugin replace the expected size based on the real size of
the message, not of the file.
> 
> Thanks
> 
> Reinaldo
>

On 15.10.2013, at 23.33, Reinaldo Matukuma <ramatukuma at hotmail.com>
wrote:
> Hello. Probably only Timo can help-me with this.
> 
> I have a self-made plugin based on the zlib plugin that i use to
cryptograph the messages at inbox.
> 
> As a side-effect of the cryptography, my plugin changes the size of the
message, but until 2.0.19 this works well with dovecot index and the W/S flags.
> 
> But now, i'm going to upgrade to 2.1.17 and now i have these messages
on log at my test ambiance:
> 
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Cached message size smaller than expected (367 < 529)
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Maildir filename has wrong S value, renamed the file from
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S
to
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S
> Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error:
Corrupted index cache file
/storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache:
Broken physical size for mail UID 6
Is it intended that when decrypting messages you?ll get exactly the original
message back? Or are you also modifying the message? Assuming there is no
modification, you could take a look at how
http://dovecot.org/patches/2.2/mail-filter.tar.gz does that without problems.
Message modifications should also be possible with that code with small
modifications. I?m not sure if that code works for v2.1.