dovecot - Jul 2006 - Dovecot deliver logging problem and procmail

Hello!

Currently I'm trying to integrate dovecot's deliver program into
procmail.
So basically I'd write a patch for procmail to deliver not directly into 
the mbox file but delivering with the dovecot deliver program.

Do you think this is a good concept?

For testing I use the commands discussed below. Currently I'm having a 
problem that deliver doesn't recognize the logging configuration from the
environment (as non root user):
setenv INFO_LOG_PATH $HOME/deliver.log
setenv LOG_PATH $HOME/deliver.log
setenv LOG_TIMESTAMP "%b %d %H:%M:%S "

cat testmail.txt | /usr/libexec/dovecot/deliver -m testbox
cat testmail.txt | strace /usr/libexec/dovecot/deliver -d gerhard -m
Mail/testbox
cat testmail.txt | strace /usr/libexec/dovecot/deliver -m Mail/testbox

Mail is delivered correctly, but no logging at all.

Any ideas?

Config (currently not security optimized, local testing only):
auth default {
  #Some stuff deleted
   socket listen {
     master {
       path = /var/run/dovecot-auth-master
       # WARNING: Giving untrusted users access to master socket may be a
       # security risk, don't give too wide permissions to it!
#      mode = 0600
       mode = 0660
       # Default user/group is the one who started dovecot-auth (root)
       #user        group = users
     }
   }
}

protocol lda {
   # Support for dynamically loadable plugins. mail_plugins is a space 
separated
   # list of plugins to load. For example quota plugin is used to enforce 
quota.
   #mail_plugins    #mail_plugin_dir = /usr/lib/dovecot/imap

   # Address from LDA should send MDNs like out of quota
   postmaster_address = postmaster at wiesinger.com

   # Hostname to show in mail headers. Default is the system's
   # real hostname.
   #hostname 
   # Binary to use for sending mails.
   #sendmail_path = /usr/lib/sendmail

   # If there is no user-specific Sieve-script, global Sieve script is
   # executed if set.
   #global_script_path 
   # UNIX socket path to master authentication server to find users.
   auth_socket_path = /var/run/dovecot-auth-master
}

Thank you for your help.

Ciao,
Gerhard

On Jul 2, 2006, at 12:31 PM, Gerhard Wiesinger wrote:
> For testing I use the commands discussed below. Currently I'm  
> having a problem that deliver doesn't recognize the logging  
> configuration from the environment (as non root user):
> setenv INFO_LOG_PATH $HOME/deliver.log
> setenv LOG_PATH $HOME/deliver.log
> setenv LOG_TIMESTAMP "%b %d %H:%M:%S "
>
> cat testmail.txt | /usr/libexec/dovecot/deliver -m testbox
The environment variables are all just cleared before reading  
dovecot.conf. You could try putting them inside protocol lda section.  
I'm not sure if %h/deliver.log would work as a path, plain  
"deliver.log" might do the same.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20060702/47353286/attachment.bin>

Ok, I got it finally to work with the following config:
protocol lda {
   auth_socket_path = /var/run/dovecot-auth-master
   postmaster_address = postmaster at domain
   log_path = /home/user/deliver.log
   info_log_path = /home/user/deliver.log
   log_timestamp=%Y.%m.%d %H:%M:%S%t
   default_mail_env = mbox:~/Mail:INBOX=/var/mail/%u
}

I've a first version with procmail now running:
When it delivers to a mbox folder it doesn't write directly to the 
mbox but it delivers through dovecots deliver agent.
(BTW: procmail is really ugly code, I'll release the code change for 
review when I have cleaned up a little bit).

As far as I saw dovecot's deliver process updates some header fields and 
the index files:
Status:
X-Keywords: (with a lot of spaces, why?)
Content-Length: 6

After access with dovecot the following header fields change:
Status: O
X-IMAPbase: 115195111 0000000009
X-UID: 2

So basically this isn't an advantage over the procmail or any other 
delivering process top mbox. The problem is that after the first access 
the X-UID and X-IMAPbase header must be added, so on large mbox files this 
might take a long time.

So basically dovecot's deliver could add these two lines, correct?
This would speedup mboxes a lot with new content.

Ciao,
Gerhard

On Sun, 2 Jul 2006, Timo Sirainen wrote:
> On Jul 2, 2006, at 12:31 PM, Gerhard Wiesinger wrote:
>
>> For testing I use the commands discussed below. Currently I'm
having a
>> problem that deliver doesn't recognize the logging configuration
from the
>> environment (as non root user):
>> setenv INFO_LOG_PATH $HOME/deliver.log
>> setenv LOG_PATH $HOME/deliver.log
>> setenv LOG_TIMESTAMP "%b %d %H:%M:%S "
>> 
>> cat testmail.txt | /usr/libexec/dovecot/deliver -m testbox
>
> The environment variables are all just cleared before reading dovecot.conf.
> You could try putting them inside protocol lda section. I'm not sure if
> %h/deliver.log would work as a path, plain "deliver.log" might do
the same.
>
>

Hello Timo!

Any ideas why it may take so long time to open big mbox files with indexes and 
delivered through dovecot deliver?

So it looks like that something is wrong with the indexes or the algorithm.

For details have a look below.

BTW: procmail filtering through deliver works well now for me in production 
environment. I'll make some code cleanups and configuration options. Maybe I
can release the patch today.

Ciao,
Gerhard

On Mon, 3 Jul 2006, Timo Sirainen wrote:
> On Jul 3, 2006, at 11:07 PM, Gerhard Wiesinger wrote:
> 
>> As far as I saw dovecot's deliver process updates some header
fields and the
>> index files:
>> Status:
>> X-Keywords: (with a lot of spaces, why?)
>> Content-Length: 6
> 
> The extra spaces are there so that if Dovecot needs to update some headers
it
> can take the needed space from there so it doesn't have to move the
whole
> mbox.
>
Yes, I saw it after I wrote the mail ...
>> After access with dovecot the following header fields change:
>> Status: O
>> X-IMAPbase: 115195111 0000000009
>> X-UID: 2
>> 
>> So basically this isn't an advantage over the procmail or any other
>> delivering process top mbox. The problem is that after the first access
the
>> X-UID and X-IMAPbase header must be added, so on large mbox files this
might
>> take a long time.
> 
> Well, first of all Dovecot should take the space it needs from the
X-Keywords
> spaces, so it doesn't need to write all that much data.
> 
> Second, the X-UID etc. headers are added by deliver if the mbox file is
fully
> synced against the indexes. So after you had opened it once with IMAP,
after
> that deliver should write the X-UID headers also as long as you don't
use
> something else than Dovecot to modify the mbox.
Ok, that worked well. I tried to deliver some large mails (~50MB) which worked 
well. After sending around 500MB (10 Mails) I opened the mailbox with pine over 
imap. It still took a lot of time (>20s) to open the mbox. Indexes should be 
up2date, so why does it take so long time?

After opening a second time everything is fine.

Any ideas?

Ciao,
Gerhard