dovecot - Jun 2014 - Mailboxes are in Maildir format. Any good backup tips? Had success with version control?

I'm still pretty new to running a mail server, but one thing I've come 
to appreciate over the years is a good backup strategy. Since I have 
always run my own servers for practice and for personal use I don't have 
access to Enterprise backup solutions. Because of that I usually just 
fall back to scripts and tarballs and offload the content on a regular 
basis.

Right now I'm using LVM snapshots + tarballs for daily backups, but I'd 
like to get better coverage for incremental changes that occur 
throughout the day. The size of existing content is low, but (small) 
changes are frequent.

I went with Maildir format because based on my reading it is referred to 
as time tested and corruption resistant. Because individual emails are 
stored as separate files this also leads me to believe that a version 
control system (Git, SVN) would allow for easy point in time restores.

I'm also going to research the GNU tar utility's support for incremental
archives as that sounds promising.

Suggestions and warnings are most welcome.

Thanks!

On 1.7.2014 00:28, deoren wrote:
> I'm still pretty new to running a mail server, but one thing I've
come
> to appreciate over the years is a good backup strategy. Since I have
> always run my own servers for practice and for personal use I don't
have
> access to Enterprise backup solutions. Because of that I usually just
> fall back to scripts and tarballs and offload the content on a regular
> basis.
>
> Right now I'm using LVM snapshots + tarballs for daily backups, but
I'd
> like to get better coverage for incremental changes that occur
> throughout the day. The size of existing content is low, but (small)
> changes are frequent.
>
> I went with Maildir format because based on my reading it is referred to
> as time tested and corruption resistant. Because individual emails are
> stored as separate files this also leads me to believe that a version
> control system (Git, SVN) would allow for easy point in time restores.
>
> I'm also going to research the GNU tar utility's support for
incremental
> archives as that sounds promising.
>
> Suggestions and warnings are most welcome.
>
> Thanks!
Since you're using maildir, you might want to check rsync out as well, 
especially with --link-dest. In short, you call rsync on your backup 
machine like this:

rsync --link-dest=previous-backup-dir source new-backup-dir

This compares current files with their copies in previous backup. All 
files which didn't change are hardlinked, saving storage space.

On 2014-07-01 00:28, deoren wrote:
> I'm still pretty new to running a mail server, but one thing I've
come
> to appreciate over the years is a good backup strategy. Since I have
> always run my own servers for practice and for personal use I don't
> have access to Enterprise backup solutions. Because of that I usually
> just fall back to scripts and tarballs and offload the content on a
> regular basis.
>
> Suggestions and warnings are most welcome.
>
I suggest rdiff-backup: a great tool that keeps an actual sync
(rsync-based) of the data-directory and the metadata (delta) in a
seperate directory to restore data from any date.

Alternatively, you might want to take a look at bacula, which was faster
in most cases (development seems to have stalled, but there is a fork I
have not had time to take a look at: bareos). However, I liked the
rdiff-backup way because I can restore files via scp or rsync (most of
my requests were like "please restore from yesterday") or if I want to
restore data from a certain date I can use rdiff-backup from command
line (bacula is much more complex, and you need the admin tool to
restore files)

hth,
Robert

deoren <Dovecot-mailing-list at whyaskwhy.org>
wrote:
> 
> Right now I'm using LVM snapshots + tarballs for daily backups, but
> I'd like to get better coverage for incremental changes that occur
> throughout the day. The size of existing content is low, but (small)
> changes are frequent.
If you actually want to preserve those increments (as opposed to just keeping
an rsync mirror up-to-date), I like rdiff-backup.  It handles maildirs well
because of the one-message-per-file design.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

On 01.07.2014 00:28, deoren wrote:
> I'm still pretty new to running a mail server, but one thing I've
come
> to appreciate over the years is a good backup strategy. Since I have 
> always run my own servers for practice and for personal use I don't 
> have access to Enterprise backup solutions. Because of that I usually 
> just fall back to scripts and tarballs and offload the content on a 
> regular basis.
>
> Right now I'm using LVM snapshots + tarballs for daily backups, but 
> I'd like to get better coverage for incremental changes that occur 
> throughout the day. The size of existing content is low, but (small) 
> changes are frequent.
>
> I went with Maildir format because based on my reading it is referred 
> to as time tested and corruption resistant. Because individual emails 
> are stored as separate files this also leads me to believe that a 
> version control system (Git, SVN) would allow for easy point in time 
> restores.
>
> I'm also going to research the GNU tar utility's support for 
> incremental archives as that sounds promising.
Hello Deoren,

I can recommend dirvish (http://www.dirvish.org/).

Supports incremental backups with aging strategy and hardlinks and very 
efficient storage. Works for years for me. In the background rsync is 
used for syncing the backups but with a high level interface.

Ciao,
Gerhard

On 6/30/2014 6:28 PM, deoren wrote:
> I'm still pretty new to running a mail server, but one thing I've
come
> to appreciate over the years is a good backup strategy. Since I have
> always run my own servers for practice and for personal use I don't
have
> access to Enterprise backup solutions. Because of that I usually just
> fall back to scripts and tarballs and offload the content on a regular
> basis.
> 
LVM snapshots of the file system combined with rdiff-backup to a second
server.

The main advantages over tar/rsync:

- Deltas instead of entire file. Plus deltas are compressed.

- It handles lots and lots of files well.

- You can easily age off older deltas.  Not hard to keep 26W or 52W of
daily or a few-times-per-day backups.

- Metadata such as file permissions / owner / group are stored in
regular files in the rdiff-backup target directory, so the destination
file system where you store the rdiff-backup files does not matter much.

- Files are stored along with a SHA1 hash, making it possible to detect
bitrot in your backups.

Downsides:

- Restoring a file more then 10 or 20 deltas old requires a lot of disk
activity (putting /tmp on a SSD helps a lot)

Suggestions:

- Each mailbox folder should be its own rdiff-backup target.  That
allows you to only backup mailbox folders which have changed in last N
hours/days.  It also means that if things go pear-shaped with
rdiff-backup, only that one backup target is at risk.

- Put /tmp on a SSD, especially if you run rdiff-backup "verify" and
validate more then 1-2 revisions into the past.

- Once you have your rdiff-backup directories on another server it is
trivial to LVM snapshot that and then rsync to either a USB drive or
offsite server (or both).  The rdiff-backup directory structure is very
rsync-friendly.

- If you're going to do hourly backups, have (2) rdiff-backup locations.
 One that deals with the hourly backups and has a short retention cycle
of only 3-4 weeks.  Another location that deals with daily backups and
has a 55W retention cycle.  That way you can restore to an hour within
the past 3-4 weeks, or any day within the past 55W.

On 6/30/2014 5:28 PM, deoren wrote:
> I'm still pretty new to running a mail server, but one thing I've
come
> to appreciate over the years is a good backup strategy. Since I have 
> always run my own servers for practice and for personal use I don't
have
> access to Enterprise backup solutions. Because of that I usually just 
> fall back to scripts and tarballs and offload the content on a regular 
> basis.
> 
> Right now I'm using LVM snapshots + tarballs for daily backups, but
I'd
> like to get better coverage for incremental changes that occur 
> throughout the day. The size of existing content is low, but (small) 
> changes are frequent.
> 
> I went with Maildir format because based on my reading it is referred to 
> as time tested and corruption resistant. Because individual emails are 
> stored as separate files this also leads me to believe that a version 
> control system (Git, SVN) would allow for easy point in time restores.
> 
> I'm also going to research the GNU tar utility's support for
incremental
> archives as that sounds promising.
> 
> Suggestions and warnings are most welcome.
> 
> Thanks!
> 
Sorry for the late reply, and thanks to everyone who replied with
suggestions. I appreciate you taking the time to do that and you've
given me a lot of good ideas to look over.

Options are good!