thr3ads.net - dovecot - Version controlled (git) Maildir generated by Dovecot [Oct 2020]

If this information is useful, please help other people find it:
Share via:

Adam

2020-Oct-06 17:21 UTC

Version controlled (git) Maildir generated by Dovecot

Hi Everybody,

I'd like to start archiving e-mails by moving them to a server with running
Dovecot.

I installed "dovecot-core" and "dovecot-imapd" (version
2.3.4.1) on Debian 10.

One of a few configurations I made is to use Maildir:

# grep '^mail_location = ' /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Maildir

I successfully moved some e-mails (at this moment it's just a test) to
subfolders that describe a year of the backup.

I'd like to start using git on the whole Maildir. After going through the
Maildir directory and subdirectories I realized that Dovecot saves some extra
files that I might not need to backup (version control).

Based on my understanding and after reading https://wiki.dovecot.org/IndexFiles
I believe that the only files I need to archive are files in "cur/"
directory and these files should start with Unix time.

So I decided to write a ".gitignore" whitelist that will include only
files in "cur/" that start with a number:

# cat .gitignore
-----------------------------------
!.gitignore
!*/
!/Maildir/cur/[0-9]*
!/Maildir/*/cur/[0-9]*
-----------------------------------

and it's working:

# git add . --dry-run
-----------------------------------
add '.gitignore'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M110302P17113.mail1,S=8066,W=8276:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M134554P17113.mail1,S=8024,W=8234:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M157569P17113.mail1,S=8005,W=8215:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M180965P17113.mail1,S=8021,W=8231:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M205518P17113.mail1,S=8017,W=8227:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M355896P17113.mail1,S=8042,W=8252:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M52133P17113.mail1,S=8053,W=8263:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M555847P17113.mail1,S=8029,W=8239:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M581502P17113.mail1,S=8047,W=8257:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M606379P17113.mail1,S=8044,W=8254:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M628850P17113.mail1,S=8036,W=8246:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M657431P17113.mail1,S=8069,W=8279:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M681265P17113.mail1,S=8039,W=8249:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M706585P17113.mail1,S=8073,W=8283:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M730046P17113.mail1,S=8059,W=8269:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M753487P17113.mail1,S=8034,W=8244:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M776202P17113.mail1,S=8019,W=8229:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M799238P17113.mail1,S=8085,W=8295:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M822106P17113.mail1,S=8081,W=8291:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M844397P17113.mail1,S=8102,W=8312:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M867952P17113.mail1,S=7964,W=8174:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M889882P17113.mail1,S=7992,W=8202:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M912429P17113.mail1,S=8005,W=8215:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M935326P17113.mail1,S=8027,W=8237:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M959570P17113.mail1,S=8037,W=8247:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M987388P17113.mail1,S=8018,W=8228:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M112704P17113.mail1,S=8044,W=8254:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M13262P17113.mail1,S=8025,W=8235:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M321294P17113.mail1,S=8012,W=8222:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M347297P17113.mail1,S=8014,W=8224:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M36960P17113.mail1,S=8037,W=8247:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M371029P17113.mail1,S=8005,W=8215:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M395270P17113.mail1,S=8024,W=8234:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M418277P17113.mail1,S=8046,W=8256:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M441021P17113.mail1,S=8047,W=8257:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M466329P17113.mail1,S=8025,W=8235:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M491134P17113.mail1,S=8038,W=8248:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M515650P17113.mail1,S=8046,W=8256:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M539950P17113.mail1,S=8017,W=8227:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M563073P17113.mail1,S=7983,W=8193:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M587009P17113.mail1,S=7963,W=8173:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M611652P17113.mail1,S=7994,W=8204:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M65180P17113.mail1,S=8015,W=8225:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M90376P17113.mail1,S=8036,W=8246:2,S'
-----------------------------------

Would you be so kind and confirm that this is all I need to archive and that
Dovecot will/can re-create all files ignored by git?

For example based on my research the following files / directories would be
ignored:
- new/ (empty dir)
- tmp/ (empty dir)
- maildirfolder/ (dir with maildirfolder file)
- dovecot-uidlist
- dovecot-uidvalidity
- dovecot-uidvalidity.5f7c4f24
- dovecot.index
- dovecot.index.cache
- dovecot.index.log
- dovecot.index.log.2
- dovecot.list.index
- dovecot.list.index.log
- dovecot.list.index.log.2
- dovecot.mailbox.log


In case of my test:
# git status --ignored
-----------------------------------
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .gitignore
        Maildir/

Ignored files:
  (use "git add -f <file>..." to include in what will be
committed)

        Maildir/.INBOX.years.2020.family/dovecot-uidlist
        Maildir/.INBOX.years.2020.family/dovecot.index
        Maildir/.INBOX.years.2020.family/dovecot.index.cache
        Maildir/.INBOX.years.2020.family/dovecot.index.log
        Maildir/.INBOX.years.2020.family/dovecot.index.log.2
        Maildir/.INBOX.years.2020.family/maildirfolder
        Maildir/dovecot-uidlist
        Maildir/dovecot-uidvalidity
        Maildir/dovecot-uidvalidity.5f7c4f24
        Maildir/dovecot.index.log
        Maildir/dovecot.list.index
        Maildir/dovecot.list.index.log
        Maildir/dovecot.list.index.log.2
        Maildir/dovecot.mailbox.log
        Maildir/subscriptions

nothing added to commit but untracked files present (use "git add" to
track)
-----------------------------------

Thank you.

Adam

David Myers

2020-Oct-07 07:42 UTC

head link

Version controlled (git) Maildir generated by Dovecot

Hello Adam, and the dovecot list

Just a question, I hate to pollute the thread, so feel free to push these
questions into a new thread if deemed necessary. So as you can guess I'm a
bit of a newb here, so rather obvious questions are about to arrive....

As you are using GIT for your archive (which is a cool idea by the way) I'm
sure you are well aware that not all files types play nicely with version
control, my question therefore is : How do you plan to handle attachments ?

Also, although I appreciate the idea of using git, emails generally don't
'change', but I guess that also depends on how you are storing them
(single
email with links to previous / next ... etc, or as a single big file for
each specific thread). Although this is hitting my limits of understanding
for how dovecot works, so I probably need educating on this (a pointer to
the docs would be good).

You seem concerned regarding the files that you are ignoring that you will
need to 'recreate them', so why not do a complete git add . prior to
adding
them into the git ignore, then you have an initial state for those files
too.

Final thought, what advantage do you envisage by using git as opposed to
simply using a filter to select the files over a certain age, and place
them into a zipped TAR archive ? Although I guess you could eventually zip
the git archive too, and in the interim it would remain searchable by your
users mail clients whilt in git.

Thanks in advance, and apologies once again for polluting your question
with my own.

David











On Tue, Oct 6, 2020 at 7:22 PM Adam <adam.ranek at seznam.cz> wrote:
> Hi Everybody,
>
> I'd like to start archiving e-mails by moving them to a server with
> running Dovecot.
>
> I installed "dovecot-core" and "dovecot-imapd" (version
2.3.4.1) on Debian
> 10.
>
> One of a few configurations I made is to use Maildir:
>
> # grep '^mail_location = ' /etc/dovecot/conf.d/10-mail.conf
> mail_location = maildir:~/Maildir
>
> I successfully moved some e-mails (at this moment it's just a test) to
> subfolders that describe a year of the backup.
>
> I'd like to start using git on the whole Maildir. After going through
the
> Maildir directory and subdirectories I realized that Dovecot saves some
> extra files that I might not need to backup (version control).
>
> Based on my understanding and after reading
> https://wiki.dovecot.org/IndexFiles I believe that the only files I need
> to archive are files in "cur/" directory and these files should
start with
> Unix time.
>
> So I decided to write a ".gitignore" whitelist that will include
only
> files in "cur/" that start with a number:
>
> # cat .gitignore
> -----------------------------------
> !.gitignore
> !*/
> !/Maildir/cur/[0-9]*
> !/Maildir/*/cur/[0-9]*
> -----------------------------------
>
> and it's working:
>
> # git add . --dry-run
> -----------------------------------
> add '.gitignore'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M110302P17113.mail1,S=8066,W=8276:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M134554P17113.mail1,S=8024,W=8234:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M157569P17113.mail1,S=8005,W=8215:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M180965P17113.mail1,S=8021,W=8231:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M205518P17113.mail1,S=8017,W=8227:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M355896P17113.mail1,S=8042,W=8252:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M52133P17113.mail1,S=8053,W=8263:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M555847P17113.mail1,S=8029,W=8239:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M581502P17113.mail1,S=8047,W=8257:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M606379P17113.mail1,S=8044,W=8254:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M628850P17113.mail1,S=8036,W=8246:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M657431P17113.mail1,S=8069,W=8279:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M681265P17113.mail1,S=8039,W=8249:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M706585P17113.mail1,S=8073,W=8283:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M730046P17113.mail1,S=8059,W=8269:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M753487P17113.mail1,S=8034,W=8244:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M776202P17113.mail1,S=8019,W=8229:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M799238P17113.mail1,S=8085,W=8295:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M822106P17113.mail1,S=8081,W=8291:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M844397P17113.mail1,S=8102,W=8312:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M867952P17113.mail1,S=7964,W=8174:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M889882P17113.mail1,S=7992,W=8202:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M912429P17113.mail1,S=8005,W=8215:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M935326P17113.mail1,S=8027,W=8237:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M959570P17113.mail1,S=8037,W=8247:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986418.M987388P17113.mail1,S=8018,W=8228:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M112704P17113.mail1,S=8044,W=8254:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M13262P17113.mail1,S=8025,W=8235:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M321294P17113.mail1,S=8012,W=8222:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M347297P17113.mail1,S=8014,W=8224:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M36960P17113.mail1,S=8037,W=8247:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M371029P17113.mail1,S=8005,W=8215:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M395270P17113.mail1,S=8024,W=8234:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M418277P17113.mail1,S=8046,W=8256:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M441021P17113.mail1,S=8047,W=8257:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M466329P17113.mail1,S=8025,W=8235:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M491134P17113.mail1,S=8038,W=8248:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M515650P17113.mail1,S=8046,W=8256:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M539950P17113.mail1,S=8017,W=8227:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M563073P17113.mail1,S=7983,W=8193:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M587009P17113.mail1,S=7963,W=8173:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M611652P17113.mail1,S=7994,W=8204:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M65180P17113.mail1,S=8015,W=8225:2,S'
> add
>
'Maildir/.INBOX.years.2020.family/cur/1601986419.M90376P17113.mail1,S=8036,W=8246:2,S'
> -----------------------------------
>
> Would you be so kind and confirm that this is all I need to archive and
> that Dovecot will/can re-create all files ignored by git?
>
> For example based on my research the following files / directories would
> be ignored:
> - new/ (empty dir)
> - tmp/ (empty dir)
> - maildirfolder/ (dir with maildirfolder file)
> - dovecot-uidlist
> - dovecot-uidvalidity
> - dovecot-uidvalidity.5f7c4f24
> - dovecot.index
> - dovecot.index.cache
> - dovecot.index.log
> - dovecot.index.log.2
> - dovecot.list.index
> - dovecot.list.index.log
> - dovecot.list.index.log.2
> - dovecot.mailbox.log
>
>
> In case of my test:
> # git status --ignored
> -----------------------------------
> On branch master
>
> No commits yet
>
> Untracked files:
>   (use "git add <file>..." to include in what will be
committed)
>
>         .gitignore
>         Maildir/
>
> Ignored files:
>   (use "git add -f <file>..." to include in what will be
committed)
>
>         Maildir/.INBOX.years.2020.family/dovecot-uidlist
>         Maildir/.INBOX.years.2020.family/dovecot.index
>         Maildir/.INBOX.years.2020.family/dovecot.index.cache
>         Maildir/.INBOX.years.2020.family/dovecot.index.log
>         Maildir/.INBOX.years.2020.family/dovecot.index.log.2
>         Maildir/.INBOX.years.2020.family/maildirfolder
>         Maildir/dovecot-uidlist
>         Maildir/dovecot-uidvalidity
>         Maildir/dovecot-uidvalidity.5f7c4f24
>         Maildir/dovecot.index.log
>         Maildir/dovecot.list.index
>         Maildir/dovecot.list.index.log
>         Maildir/dovecot.list.index.log.2
>         Maildir/dovecot.mailbox.log
>         Maildir/subscriptions
>
> nothing added to commit but untracked files present (use "git
add" to
> track)
> -----------------------------------
>
> Thank you.
>
> Adam
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20201007/e05e436e/attachment-0001.html>

Adam

2020-Oct-07 08:41 UTC

head link

Version controlled (git) Maildir generated by Dovecot

Hi David, 

Please find answers bellow.

Kind regards,
Adam

---------- P?vodn? e-mail ----------
Od: David Myers <david.myers.24j74 at gmail.com>
Komu: Adam <adam.ranek at seznam.cz>, Dovecot Mailing List <dovecot at
dovecot.org>
Datum: 7. 10. 2020 8:44:28
P?edm?t: Re: Version controlled (git) Maildir generated by Dovecot
Hello Adam, and the dovecot list
> Just a question, I hate to pollute the thread, so feel free to push these
> questions into a new thread if deemed necessary. So as you can guess
I'm a bit
> of a newb here, so rather obvious questions are about to arrive....
> 
> As you are using GIT for your archive (which is a cool idea by the way)
I'm
> sure you are well aware that not all files types play nicely with version
> control, my question therefore is : How do you plan to handle attachments ?
I use git for everything including for example LibreOffice / Word documents. Git
works just fine with binary files. You can't use text tools like "git
diff" but... it works.
> Also, although I appreciate the idea of using git, emails generally
don't
> 'change', but I guess that also depends on how you are storing them
(single
> email with links to previous / next ... etc, or as a single big file for
each
> specific thread). Although this is hitting my limits of understanding for
how
> dovecot works, so I probably need educating on this (a pointer to the docs
would
> be good).
As I mentioned in the first e-mail, I configured Dovecot to use Maildir format
-> each e-mail is a single text file. Mail body + attachment(s) are in the
same file, attachment(s) are Base64 encoded.
> You seem concerned regarding the files that you are ignoring that you will
need
> to 'recreate them', so why not do a complete git add . prior to
adding them into
> the git ignore, then you have an initial state for those files too.
But I don't want to store files that can be regenerated. I don't want to
backup stuff, that doesn't have information value.
> Final thought, what advantage do you envisage by using git as opposed to
simply
> using a filter to select the files over a certain age, and place them into
a
> zipped TAR archive ? Although I guess you could eventually zip the git
archive
> too, and in the interim it would remain searchable by your users mail
clients
> whilt in git.
I like to use git ;-). Tar will work just fine.

In this use case the only real benefit of git is that it never forgets. Unless I
delete whole .git directory, I can make a mistake, delete some e-mails (files),
commit changes and rollback. I can't rollback if I delete tar archive.
> Thanks in advance, and apologies once again for polluting your question
with my
> own.
David

Apparently Analagous Threads

Search for more possibly parallel threads

dovecot - Oct 2020 - Version controlled (git) Maildir generated by Dovecot

Version controlled (git) Maildir generated by Dovecot

Version controlled (git) Maildir generated by Dovecot

Version controlled (git) Maildir generated by Dovecot

Apparently Analagous Threads