dovecot - May 2012 - [Plugin dev] First-time-only cache lock failure

Hi

This is a question related to plugin developement on Dovecot 2.1.4.

I'm currently implementing a dovecot plugin that retrieves mails from a
corporate server through an internal protocol.
The plugin is very similar to the pop3c storage (lib-storage/index/pop3c).
The plugin has to map original server UIDs to IMAP UIDs, using
MAIL_CACHE_POP3_UIDL cache field the same way pop3c does.

So, I've got a function similar to pop3c_sync_messages(), performing several
mail_index_append() and mail_cache_add() calls.

The problem is that on the first sync, when the index and cache files don't
exist yet, there's a failure writing data in cache.
The call to mail_index_sync_commit()\mail_cache_transaction_commit() fails
because of mail_cache_transaction_lock().

The index and cache files are properly created, but the cache doesn't
contain the MAIL_CACHE_POP3_UIDL values.
It works properly on the second sync, the MAIL_CACHE_POP3_UIDL field being
properly saved.

Would you have some ideas on what could be wrong here?

Thanks,

John

On Mon, 2012-05-14 at 17:17 +0100, John G wrote:
> Hi
> 
> This is a question related to plugin developement on Dovecot 2.1.4.
> 
> I'm currently implementing a dovecot plugin that retrieves mails from a
corporate server through an internal protocol.
> The plugin is very similar to the pop3c storage (lib-storage/index/pop3c).
> The plugin has to map original server UIDs to IMAP UIDs, using
MAIL_CACHE_POP3_UIDL cache field the same way pop3c does.
> 
> So, I've got a function similar to pop3c_sync_messages(), performing
several mail_index_append() and mail_cache_add() calls.
> 
> The problem is that on the first sync, when the index and cache files
don't exist yet, there's a failure writing data in cache.
> The call to mail_index_sync_commit()\mail_cache_transaction_commit() fails
because of mail_cache_transaction_lock().
> 
> The index and cache files are properly created, but the cache doesn't
contain the MAIL_CACHE_POP3_UIDL values.
> It works properly on the second sync, the MAIL_CACHE_POP3_UIDL field being
properly saved.
> 
> Would you have some ideas on what could be wrong here?
I'm not entirely sure what the problem is with cache file, but you
shouldn't use cache file anyway for this purpose if you want it to work
reliably. Cache file's contents can disappear in some situations. pop3c
shouldn't do it either, but it doesn't have to be very reliable :)

If the original server UIDs are of fixed length (or have a reasonable
maximum length), add the UIDs to dovecot.index file itself by
registering an "extension". For example see how mdbox's
"mdbox"
extension that creates struct mdbox_mail_index_record for each mail.

If the original server UIDs are variable length, you can create a
separate mapping file similar to Maildir's dovecot-uidlist.