dovecot - Jan 2019 - download Dovecot emails over rsync (done!), possible alternatives

Hello,

Apologies for the extra long email. It's a situation with a solved
problem and I might need to solve it again, I'd like to have some
feedback.

Two years ago, in a trip around the Antarctic with a very limited
communication I deployed a mail system based on Postfix+Dovecot.  I need
to do something similar again and I used a "hack" that I thought that
I
might avoid (that time, let's say, everything was last minute... or
later than last minute!).

I installed Dovecot on the internet server and the server on the ship.

On the ship I had a fetchmail downloading "small emails" (up to 100 KB
if I remember correctly) and delivering to the local server. For bigger
emails fetchmail had a "size limit" and fetchmail was not downloading
them. I wrote a Python script that was connecting from the ship, to the
internet server, and was listing the emails that were "too big to be
downloaded" (this was for each user. I could have done something
different with procmailrc I think on the receiving time).

The Python script was fetching, if I remember correctly, the email ID,
date, subject and size. The Python script was sending an email to the
ship server's user saying "there is an email for you with this size,
from, date and subject and this is the email ID" (and inserting this
into a database to avoid sending the same notification to the users
again).

Then the scientists sometimes were ignoring the email, were asking to
the sender to send a smaller version, or asking me to download it when
possible.

If they wanted me to download the email:

I had a script that was connecting to the internet server, opening
/home/$USER/Maildir/dovecot-uidlist, finding the file for that email
UUID (this was a simple script on the internet server and I was
executing it using ssh and getting the output). Then it was creating a
new IMAP folder named "Downloaded" in the ship's server (it was
just
creating the folder in the file system and adding it in the
subscriptions file, all using file system operations from Python). Then
using rsync it was downloading the email file from the internet's server
to the ship's server temporary file (taking whatever it would take, with
many retries, etc.) and then moving it to the Downloaded folder.

(note: it was using rsync and not IMAP because the connection was slow
and very unstable. With rsync and the correct options it was continuing
the download and retrying until success).

For the next expedition I'll have more time to prepare for a similar
setup. I've considered if I should stop relying on dovecot-uidlist,
local file system's, subscriptions, etc. and if I should write a plugin
for Dovecot to expose part of this information using... a JSON API? IMAP
commands?

The local ship's server bit: could I inject a SMTP mail file
(downloaded) using Dovecot admin commands? some API? Any advantage?

To discover which file to download using rsync: instead of opening the
dovecot-uidlist any better method?

Any other different approach that I might want to consider?

Sorry again for the long message. I thought that this might need some
context. Also, thanks to the design of Dovecot I could do what I did
in a very simple way. Actually Dovecot design was as I would have
designed for this use case (the index file, subscription from the
mailing list, etc.).

Thank you very much again! and thanks for the great Dovecot as well!

-- 
Carles Pina i Estany
	Web: http://pinux.info || Blog: http://pintant.cat
	GPG Key 0x8CD5C157

Hi,

Actually for the injecting the downloaded email into the ship's system:

On Jan/27/2019, Carles Pina i Estany wrote:
> executing it using ssh and getting the output). Then it was creating a
> new IMAP folder named "Downloaded" in the ship's server (it
was just
> creating the folder in the file system and adding it in the
> subscriptions file, all using file system operations from Python). Then
> using rsync it was downloading the email file from the internet's
server
> to the ship's server temporary file (taking whatever it would take,
with
> many retries, etc.) and then moving it to the Downloaded folder.
I could use Python+IMAP to create the folder in the ship's server,
subscribe and then perhaps using lmtp protocol inject the new email
(maybe with Python+lmtpd as well)

This way on the ship's server I wouldn't need to change Dovecot
directories.

On the internet server perhaps I could have a IMAP client that would
extract the email using IMAP and leaving it into a file: this way I
would avoid touching Dovecot's file system and assuming the structure of
dovecot-uidlist and how the files are saved.

That would be nicer than the last time...

Any other ideas: I'm happy to listen.

Also, if anyone wants to talk about this next week in FOSDEM I'll be
there. I see that past years there was someone from this mailing list
(Timo Sirainen)
(https://www.dovecot.org/list/dovecot/2012-January/133190.html)

Cheers,

-- 
Carles Pina i Estany
	Web: http://pinux.info || Blog: http://pintant.cat
	GPG Key 0x8CD5C157