Hi Everybody,
I'd like to start archiving e-mails by moving them to a server with running
Dovecot.
I installed "dovecot-core" and "dovecot-imapd" (version
2.3.4.1) on Debian 10.
One of a few configurations I made is to use Maildir:
# grep '^mail_location = ' /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Maildir
I successfully moved some e-mails (at this moment it's just a test) to
subfolders that describe a year of the backup.
I'd like to start using git on the whole Maildir. After going through the
Maildir directory and subdirectories I realized that Dovecot saves some extra
files that I might not need to backup (version control).
Based on my understanding and after reading https://wiki.dovecot.org/IndexFiles
I believe that the only files I need to archive are files in "cur/"
directory and these files should start with Unix time.
So I decided to write a ".gitignore" whitelist that will include only
files in "cur/" that start with a number:
# cat .gitignore
-----------------------------------
!.gitignore
!*/
!/Maildir/cur/[0-9]*
!/Maildir/*/cur/[0-9]*
-----------------------------------
and it's working:
# git add . --dry-run
-----------------------------------
add '.gitignore'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M110302P17113.mail1,S=8066,W=8276:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M134554P17113.mail1,S=8024,W=8234:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M157569P17113.mail1,S=8005,W=8215:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M180965P17113.mail1,S=8021,W=8231:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M205518P17113.mail1,S=8017,W=8227:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M355896P17113.mail1,S=8042,W=8252:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M52133P17113.mail1,S=8053,W=8263:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M555847P17113.mail1,S=8029,W=8239:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M581502P17113.mail1,S=8047,W=8257:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M606379P17113.mail1,S=8044,W=8254:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M628850P17113.mail1,S=8036,W=8246:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M657431P17113.mail1,S=8069,W=8279:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M681265P17113.mail1,S=8039,W=8249:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M706585P17113.mail1,S=8073,W=8283:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M730046P17113.mail1,S=8059,W=8269:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M753487P17113.mail1,S=8034,W=8244:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M776202P17113.mail1,S=8019,W=8229:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M799238P17113.mail1,S=8085,W=8295:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M822106P17113.mail1,S=8081,W=8291:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M844397P17113.mail1,S=8102,W=8312:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M867952P17113.mail1,S=7964,W=8174:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M889882P17113.mail1,S=7992,W=8202:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M912429P17113.mail1,S=8005,W=8215:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M935326P17113.mail1,S=8027,W=8237:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M959570P17113.mail1,S=8037,W=8247:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986418.M987388P17113.mail1,S=8018,W=8228:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M112704P17113.mail1,S=8044,W=8254:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M13262P17113.mail1,S=8025,W=8235:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M321294P17113.mail1,S=8012,W=8222:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M347297P17113.mail1,S=8014,W=8224:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M36960P17113.mail1,S=8037,W=8247:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M371029P17113.mail1,S=8005,W=8215:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M395270P17113.mail1,S=8024,W=8234:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M418277P17113.mail1,S=8046,W=8256:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M441021P17113.mail1,S=8047,W=8257:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M466329P17113.mail1,S=8025,W=8235:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M491134P17113.mail1,S=8038,W=8248:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M515650P17113.mail1,S=8046,W=8256:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M539950P17113.mail1,S=8017,W=8227:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M563073P17113.mail1,S=7983,W=8193:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M587009P17113.mail1,S=7963,W=8173:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M611652P17113.mail1,S=7994,W=8204:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M65180P17113.mail1,S=8015,W=8225:2,S'
add
'Maildir/.INBOX.years.2020.family/cur/1601986419.M90376P17113.mail1,S=8036,W=8246:2,S'
-----------------------------------
Would you be so kind and confirm that this is all I need to archive and that
Dovecot will/can re-create all files ignored by git?
For example based on my research the following files / directories would be
ignored:
- new/ (empty dir)
- tmp/ (empty dir)
- maildirfolder/ (dir with maildirfolder file)
- dovecot-uidlist
- dovecot-uidvalidity
- dovecot-uidvalidity.5f7c4f24
- dovecot.index
- dovecot.index.cache
- dovecot.index.log
- dovecot.index.log.2
- dovecot.list.index
- dovecot.list.index.log
- dovecot.list.index.log.2
- dovecot.mailbox.log
In case of my test:
# git status --ignored
-----------------------------------
On branch master
No commits yet
Untracked files:
(use "git add <file>..." to include in what will be committed)
.gitignore
Maildir/
Ignored files:
(use "git add -f <file>..." to include in what will be
committed)
Maildir/.INBOX.years.2020.family/dovecot-uidlist
Maildir/.INBOX.years.2020.family/dovecot.index
Maildir/.INBOX.years.2020.family/dovecot.index.cache
Maildir/.INBOX.years.2020.family/dovecot.index.log
Maildir/.INBOX.years.2020.family/dovecot.index.log.2
Maildir/.INBOX.years.2020.family/maildirfolder
Maildir/dovecot-uidlist
Maildir/dovecot-uidvalidity
Maildir/dovecot-uidvalidity.5f7c4f24
Maildir/dovecot.index.log
Maildir/dovecot.list.index
Maildir/dovecot.list.index.log
Maildir/dovecot.list.index.log.2
Maildir/dovecot.mailbox.log
Maildir/subscriptions
nothing added to commit but untracked files present (use "git add" to
track)
-----------------------------------
Thank you.
Adam
Hello Adam, and the dovecot list Just a question, I hate to pollute the thread, so feel free to push these questions into a new thread if deemed necessary. So as you can guess I'm a bit of a newb here, so rather obvious questions are about to arrive.... As you are using GIT for your archive (which is a cool idea by the way) I'm sure you are well aware that not all files types play nicely with version control, my question therefore is : How do you plan to handle attachments ? Also, although I appreciate the idea of using git, emails generally don't 'change', but I guess that also depends on how you are storing them (single email with links to previous / next ... etc, or as a single big file for each specific thread). Although this is hitting my limits of understanding for how dovecot works, so I probably need educating on this (a pointer to the docs would be good). You seem concerned regarding the files that you are ignoring that you will need to 'recreate them', so why not do a complete git add . prior to adding them into the git ignore, then you have an initial state for those files too. Final thought, what advantage do you envisage by using git as opposed to simply using a filter to select the files over a certain age, and place them into a zipped TAR archive ? Although I guess you could eventually zip the git archive too, and in the interim it would remain searchable by your users mail clients whilt in git. Thanks in advance, and apologies once again for polluting your question with my own. David On Tue, Oct 6, 2020 at 7:22 PM Adam <adam.ranek at seznam.cz> wrote:> Hi Everybody, > > I'd like to start archiving e-mails by moving them to a server with > running Dovecot. > > I installed "dovecot-core" and "dovecot-imapd" (version 2.3.4.1) on Debian > 10. > > One of a few configurations I made is to use Maildir: > > # grep '^mail_location = ' /etc/dovecot/conf.d/10-mail.conf > mail_location = maildir:~/Maildir > > I successfully moved some e-mails (at this moment it's just a test) to > subfolders that describe a year of the backup. > > I'd like to start using git on the whole Maildir. After going through the > Maildir directory and subdirectories I realized that Dovecot saves some > extra files that I might not need to backup (version control). > > Based on my understanding and after reading > https://wiki.dovecot.org/IndexFiles I believe that the only files I need > to archive are files in "cur/" directory and these files should start with > Unix time. > > So I decided to write a ".gitignore" whitelist that will include only > files in "cur/" that start with a number: > > # cat .gitignore > ----------------------------------- > !.gitignore > !*/ > !/Maildir/cur/[0-9]* > !/Maildir/*/cur/[0-9]* > ----------------------------------- > > and it's working: > > # git add . --dry-run > ----------------------------------- > add '.gitignore' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M110302P17113.mail1,S=8066,W=8276:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M134554P17113.mail1,S=8024,W=8234:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M157569P17113.mail1,S=8005,W=8215:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M180965P17113.mail1,S=8021,W=8231:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M205518P17113.mail1,S=8017,W=8227:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M355896P17113.mail1,S=8042,W=8252:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M52133P17113.mail1,S=8053,W=8263:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M555847P17113.mail1,S=8029,W=8239:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M581502P17113.mail1,S=8047,W=8257:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M606379P17113.mail1,S=8044,W=8254:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M628850P17113.mail1,S=8036,W=8246:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M657431P17113.mail1,S=8069,W=8279:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M681265P17113.mail1,S=8039,W=8249:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M706585P17113.mail1,S=8073,W=8283:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M730046P17113.mail1,S=8059,W=8269:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M753487P17113.mail1,S=8034,W=8244:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M776202P17113.mail1,S=8019,W=8229:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M799238P17113.mail1,S=8085,W=8295:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M822106P17113.mail1,S=8081,W=8291:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M844397P17113.mail1,S=8102,W=8312:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M867952P17113.mail1,S=7964,W=8174:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M889882P17113.mail1,S=7992,W=8202:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M912429P17113.mail1,S=8005,W=8215:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M935326P17113.mail1,S=8027,W=8237:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M959570P17113.mail1,S=8037,W=8247:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986418.M987388P17113.mail1,S=8018,W=8228:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M112704P17113.mail1,S=8044,W=8254:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M13262P17113.mail1,S=8025,W=8235:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M321294P17113.mail1,S=8012,W=8222:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M347297P17113.mail1,S=8014,W=8224:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M36960P17113.mail1,S=8037,W=8247:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M371029P17113.mail1,S=8005,W=8215:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M395270P17113.mail1,S=8024,W=8234:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M418277P17113.mail1,S=8046,W=8256:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M441021P17113.mail1,S=8047,W=8257:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M466329P17113.mail1,S=8025,W=8235:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M491134P17113.mail1,S=8038,W=8248:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M515650P17113.mail1,S=8046,W=8256:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M539950P17113.mail1,S=8017,W=8227:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M563073P17113.mail1,S=7983,W=8193:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M587009P17113.mail1,S=7963,W=8173:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M611652P17113.mail1,S=7994,W=8204:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M65180P17113.mail1,S=8015,W=8225:2,S' > add > 'Maildir/.INBOX.years.2020.family/cur/1601986419.M90376P17113.mail1,S=8036,W=8246:2,S' > ----------------------------------- > > Would you be so kind and confirm that this is all I need to archive and > that Dovecot will/can re-create all files ignored by git? > > For example based on my research the following files / directories would > be ignored: > - new/ (empty dir) > - tmp/ (empty dir) > - maildirfolder/ (dir with maildirfolder file) > - dovecot-uidlist > - dovecot-uidvalidity > - dovecot-uidvalidity.5f7c4f24 > - dovecot.index > - dovecot.index.cache > - dovecot.index.log > - dovecot.index.log.2 > - dovecot.list.index > - dovecot.list.index.log > - dovecot.list.index.log.2 > - dovecot.mailbox.log > > > In case of my test: > # git status --ignored > ----------------------------------- > On branch master > > No commits yet > > Untracked files: > (use "git add <file>..." to include in what will be committed) > > .gitignore > Maildir/ > > Ignored files: > (use "git add -f <file>..." to include in what will be committed) > > Maildir/.INBOX.years.2020.family/dovecot-uidlist > Maildir/.INBOX.years.2020.family/dovecot.index > Maildir/.INBOX.years.2020.family/dovecot.index.cache > Maildir/.INBOX.years.2020.family/dovecot.index.log > Maildir/.INBOX.years.2020.family/dovecot.index.log.2 > Maildir/.INBOX.years.2020.family/maildirfolder > Maildir/dovecot-uidlist > Maildir/dovecot-uidvalidity > Maildir/dovecot-uidvalidity.5f7c4f24 > Maildir/dovecot.index.log > Maildir/dovecot.list.index > Maildir/dovecot.list.index.log > Maildir/dovecot.list.index.log.2 > Maildir/dovecot.mailbox.log > Maildir/subscriptions > > nothing added to commit but untracked files present (use "git add" to > track) > ----------------------------------- > > Thank you. > > Adam >-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20201007/e05e436e/attachment-0001.html>
Hi David, Please find answers bellow. Kind regards, Adam ---------- P?vodn? e-mail ---------- Od: David Myers <david.myers.24j74 at gmail.com> Komu: Adam <adam.ranek at seznam.cz>, Dovecot Mailing List <dovecot at dovecot.org> Datum: 7. 10. 2020 8:44:28 P?edm?t: Re: Version controlled (git) Maildir generated by Dovecot Hello Adam, and the dovecot list> Just a question, I hate to pollute the thread, so feel free to push these > questions into a new thread if deemed necessary. So as you can guess I'm a bit > of a newb here, so rather obvious questions are about to arrive.... > > As you are using GIT for your archive (which is a cool idea by the way) I'm > sure you are well aware that not all files types play nicely with version > control, my question therefore is : How do you plan to handle attachments ?I use git for everything including for example LibreOffice / Word documents. Git works just fine with binary files. You can't use text tools like "git diff" but... it works.> Also, although I appreciate the idea of using git, emails generally don't > 'change', but I guess that also depends on how you are storing them (single > email with links to previous / next ... etc, or as a single big file for each > specific thread). Although this is hitting my limits of understanding for how > dovecot works, so I probably need educating on this (a pointer to the docs would > be good).As I mentioned in the first e-mail, I configured Dovecot to use Maildir format -> each e-mail is a single text file. Mail body + attachment(s) are in the same file, attachment(s) are Base64 encoded.> You seem concerned regarding the files that you are ignoring that you will need > to 'recreate them', so why not do a complete git add . prior to adding them into > the git ignore, then you have an initial state for those files too.But I don't want to store files that can be regenerated. I don't want to backup stuff, that doesn't have information value.> Final thought, what advantage do you envisage by using git as opposed to simply > using a filter to select the files over a certain age, and place them into a > zipped TAR archive ? Although I guess you could eventually zip the git archive > too, and in the interim it would remain searchable by your users mail clients > whilt in git.I like to use git ;-). Tar will work just fine. In this use case the only real benefit of git is that it never forgets. Unless I delete whole .git directory, I can make a mistake, delete some e-mails (files), commit changes and rollback. I can't rollback if I delete tar archive.> Thanks in advance, and apologies once again for polluting your question with my > own.David
Reasonably Related Threads
- Version controlled (git) Maildir generated by Dovecot
- Dovecot dsync not replicating ".dovecot.sieve -> .sieve/managesieve.sieve" / setactive
- Dovecot dsync not replicating ".dovecot.sieve -> .sieve/managesieve.sieve" / setactive
- need help with public namespaces
- dsync not replicating ".dovecot.sieve -> .sieve/managesieve.sieve"