dovecot - Aug 2007 - processing Dovecot maildirs with shell scripts

formail

  darix

Hello,

I have a postfix server which uses procmail as its delivery agent to
deliver incoming messages to several maildirs. For several reasons,
not really relevant here, I need to process/refilter/sort again the
content of these mailboxes before delivering the messages to the
_other_ maildirs which Dovecot sees and serves to local or remote imap
users. Basically, I need to write some shell scripts that, from cron
jobs, regularly do this:

foreach file in some_incoming_mailbox/new/
 do
 #process the file and/or set some variable according to its content
 delete the file or
 mv it to some_other_maildir/new/

where the actual action and value of some_other_maildir is set every
time by the "process the file" part of the script.

The part that I still miss is guidelines to do these things safely,
that is without losing or corrupting messages or doing anything else
that would confuse dovecot.

For example, under which conditions it is sure to move or delete (the
Unix way, with mv or rm) a message from mailbox1/new/ to mailbox2/new,
when the latter is served via dovecot to remote users?

What is advisable and most efficient? Use locks (how?), only work on
file which are at least one minute old? Should I move the messages to
tmp instead of new? What else?

Tips or pointers about this are very welcome.

Ciao,
	Marco

On Mon, Aug 27, 2007 21:21:31 PM +0200, Marcus Rueckert (darix at opensu.se)
wrote:
> formail
No, sorry.

I just come from a thread in the procmail mailing list whose result is
that formail is either unusable or unnecessary in cases like this:

http://www.xray.mpe.mpg.de/mailing-lists/procmail/2007-08/msg00117.html

-- 
Help *everybody* love Free Standards and Free Software!
http://digifreedom.net

On Mon, 2007-08-27 at 21:23 +0200, M. Fioretti wrote:
> Hello,
> 
> I have a postfix server which uses procmail as its delivery agent to
> deliver incoming messages to several maildirs. For several reasons,
> not really relevant here, I need to process/refilter/sort again the
> content of these mailboxes before delivering the messages to the
> _other_ maildirs which Dovecot sees and serves to local or remote imap
> users. Basically, I need to write some shell scripts that, from cron
> jobs, regularly do this:
> 
> foreach file in some_incoming_mailbox/new/
>  do
>  #process the file and/or set some variable according to its content
>  delete the file or
>  mv it to some_other_maildir/new/
So what happens if Dovecot sees the mail and moves it to cur/ before you
had a chance to do that? If you really need to see all the new mails,
then you'll need to figure out some way to handle that, such as add a
new private flag for all processed messages.

Or you could deliver mails to some private maildir that Dovecot doesn't
see, and process the messages from there to maildirs that Dovecot sees.
> The part that I still miss is guidelines to do these things safely,
> that is without losing or corrupting messages or doing anything else
> that would confuse dovecot.
> 
> For example, under which conditions it is sure to move or delete (the
> Unix way, with mv or rm) a message from mailbox1/new/ to mailbox2/new,
> when the latter is served via dovecot to remote users?
mv, rm and ln are always safe. Just don't ever write to new mail files
in new/ or cur/ directory, because then Dovecot might read partially
written files. That means that "cp file new/" isn't safe, but
"ln file
new/" is.

Also make sure the filename is always unique.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070828/ded882b1/attachment-0002.bin>

On Tue, Aug 28, 2007 01:03:13 AM +0300, Timo Sirainen (tss at iki.fi)
wrote:
> > Basically, I need to write some shell scripts that, from cron
> > jobs, regularly do this:
> > 
> > foreach file in some_incoming_mailbox/new/
> >  do
> >  #process the file and/or set some variable according to its content
> >  delete the file or
> >  mv it to some_other_maildir/new/
> 
> So what happens if Dovecot sees the mail and moves it to cur/ before you
> had a chance to do that? If you really need to see all the new mails,
> then you'll need to figure out some way to handle that, such as add a
> new private flag for all processed messages.
> 
> Or you could deliver mails to some private maildir that Dovecot doesn't
> see, and process the messages from there to maildirs that Dovecot sees.
> 
Yes, this is exactly what I want to do, sorry if it wasn't clear from
the beginning. I am not going to _create_ messages, or _remove_ them
under Dovecot's nose.

I have procmail delivering to:

aux_maildir_1/
aux_maildir_2/
...
aux_maildir_N/
INBOX

these scripts must only delete or move (not create or rename!)
files/messages, with certain criteria and at certain times, from the
aux_maildirs only to INBOX and INBOX.one, INBOX.two, INBOX.number

Dovecot only sees the INBOX* folders. Under these constraints I'm
safe, am I not?  Also, since all the file names are created by
procmail (there is nobody else injecting email from the outside) I can
count on them being always unique, right?
> mv, rm and ln are always safe. Just don't ever write to new mail files
> in new/ or cur/ directory, because then Dovecot might read partially
> written files. That means that "cp file new/" isn't safe, but
"ln file
> new/" is.
Er.. sorry, what do you mean exactly here? where is "file" where I
link to new? What is the complete, exact sequence of Unix commands to
safely go from:

aux_maildir_1/new/some_file

to, say,

INBOX.one/new/some_file

TIA,
	Marco

-- 
Help *everybody* love Free Standards and Free Software
http://digifreedom.net/