dovecot - Dec 2013 - Plugin development - ostream and index cache size

Hi,

I'm new to this list, so first thanks for the dovecot tool! So far, it 
has been great to work with.

Currently, I'm developing a scrambler plugin for dovecot, that should 
stores the mails encrypted. To do that, I've created the 
scrambler_ostream and scrambler_istream that does the encryption and 
that I hook in the mail storage process.

The scrambler_ostream is stacked on top of the output stream of the 
mail_save_context (mail_save_context->data.output) and the 
scrambler_istream is attached to the input stream when the virtual 
function istream_opened in the mail_private struct is called.

Basically, the encrypt/decrypt works, but during the process the data 
size changes slightly. And when I'm doing a FETCH BINARY[] via imap, it 
results in the error...

Cached message size smaller than expected (204 < 208)
Error: Corrupted index cache file 
/home/.../mail/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken 
physical size for mail UID 1

My guess is, that the FETCH function tries to deliver the unencrypted 
cache version of the mail (204 bytes) and not the stored version (208 
bytes).

How can I make dovecot cache the output of my scrambler_ostream or at 
least disable this caching for testing?

I've read already a lot of code, but I've found a hint yet. How does - 
for example - the zlib plugin handle this issue? I have read the code a 
couple of times, but still have no clue.

Best regards,
Philipp

On 11.12.2013, at 18.14, Philipp Br?ll <bruell at simiatech.com> wrote:
> I'm new to this list, so first thanks for the dovecot tool! So far, it
has been great to work with.
> 
> Currently, I'm developing a scrambler plugin for dovecot, that should
stores the mails encrypted. To do that, I've created the scrambler_ostream
and scrambler_istream that does the encryption and that I hook in the mail
storage process.
> 
> The scrambler_ostream is stacked on top of the output stream of the
mail_save_context (mail_save_context->data.output) and the scrambler_istream
is attached to the input stream when the virtual function istream_opened in the
mail_private struct is called.
> 
> Basically, the encrypt/decrypt works, but during the process the data size
changes slightly. And when I'm doing a FETCH BINARY[] via imap, it results
in the error...
> 
> Cached message size smaller than expected (204 < 208)
> Error: Corrupted index cache file
/home/.../mail/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken physical
size for mail UID 1
> 
> My guess is, that the FETCH function tries to deliver the unencrypted cache
version of the mail (204 bytes) and not the stored version (208 bytes).
> 
> How can I make dovecot cache the output of my scrambler_ostream or at least
disable this caching for testing?
> 
> I've read already a lot of code, but I've found a hint yet. How
does - for example - the zlib plugin handle this issue? I have read the code a
couple of times, but still have no clue.
Well, it shoud work exactly the same way as with zlib plugin .. difficult to say
what you?re doing differently without seeing the code. After decryption the mail
will look exactly the same as before encryption, right? No added/removed pieces
of data? That?s a requirement when using zlib-like code for this. BTW. In v2.2.9
there?s also mail-filter plugin where you can implement this
encryption/decryption using an external script.