dovecot - Nov 2017 - dovecot-lda without starting dovecot?

On Tue, 7 Nov 2017 21:40:55 +0200
Timo Sirainen <tss at iki.fi> wrote:
> On 7 Nov 2017, at 21.33, Stephan von Krawczynski <skraw.ml at
ithnet.com> wrote:
> > 
> > 
> > Me again have to stress that our former implementation of the lda
process
> > did do exactly nothing to all the dovecot files, and everything worked
> > pretty well. We had no problems in years. So I really wonder if it
> > wouldn't be the best way to simply cut away all the heavy dovecot
magic
> > from the lda as an option. All it really needs to do is pipe the mail
into
> > a temp file, do the sieve stuff and learn from it where to rename this
> > temp file to, basically. On the other hand you have lots of config
> > parameters in dovecot dealing with different approaches to lock files
> > (from fcntl to dotlock). I would expect at least one of them to work
over
> > NFS.  
> 
> I suppose you could create a unique new per-delivery temporary directory
> where dovecot-lda writes the mail, so it knows nothing about the user's
real
> home/mail directory. Then move it from there to Maildir/tmp/ -> rename
to
> Maildir/new/. I suppose you'd need to copy the Sieve script to the temp
> directory. Not sure if something would have to be done
> with .dovecot.lda-dupes.
> 
> > And we do see stuff like:
> > 
> > 2017-11-07T00:46:59.102961+01:00 mail-a05 dovecot:
> > lda(somebody at somedomain.com): Warning: Locking transaction log
> > file <nfs-path>/dovecot.index.log took 40 seconds (appending)
> > 
> > So some locking seems to work.  
> 
> The problem isn't locking, but caching of data. Especially because
Dovecot
> doesn't lock when reading files.
The problem with data caching would not be existant if dovecot used O_DIRECT
like proposed in the nfs (linux) docs.

---- From nfs(5) manpage ----

The NFS protocol is not designed to support true cluster file system cache
coherence without some type of application serialization. If absolute cache
coherence among clients is required, applications should use file locking.
Alternatively, applications can also open their files with the O_DIRECT flag
to disable data caching entirely. 

----- end ----

AFAIK no complex director stuff would be needed then, right?
The second sentence implies that using file locking should also be enough,
which dovecot does.


-- 
Regards,
Stephan

> On 20 Nov 2017, at 10.50, Stephan von Krawczynski <skraw.ml at
ithnet.com> wrote:
> 
> AFAIK no complex director stuff would be needed then, right?
> The second sentence implies that using file locking should also be enough,
> which dovecot does.
You are building such a complex system and then you think that creating director
layer
would be complex?

Just out of curiosity may I ask what size of an environment is this? As it's
very hard for
me to believe that this would scale.

Sami

On Mon, 20 Nov 2017 12:05:46 +0200
Sami Ketola <sami.ketola at dovecot.fi> wrote:
> > On 20 Nov 2017, at 10.50, Stephan von Krawczynski <skraw.ml at
ithnet.com>
> > wrote:
> > 
> > AFAIK no complex director stuff would be needed then, right?
> > The second sentence implies that using file locking should also be
enough,
> > which dovecot does.  
> 
> You are building such a complex system and then you think that creating
> director layer would be complex?
> 
> Just out of curiosity may I ask what size of an environment is this? As
it's
> very hard for me to believe that this would scale.
> 
> Sami
If I had lets say 10 nodes handling incoming SMTP and delivery to local
maildirs in parallel and independantly on a network storage and imapping this
from another independant 4 nodes, then building a not needed bottleneck by
shifting over the local delivery via director and LMTP to fewer nodes is just
braindead creation of high loads on few boxes.

And btw it scales perfectly because all that is needed if load increases is
up'ing additional nodes for SMTP/delivery or IMAP _which are all the same
(of
two types)_.

The loadbalancer needed for this can be equally used for e.g. web, ftp and
other services. So there is absolutely no extra stuff needed for the mail
setup.

-- 
Regards,
Stephan