We switched our mail server over from mbox to maildir a few months ago and couldn't be happier - the performance under normal load is incredible. However we now have a problem with backup.. Typically we would run tar on the mail server, sending its output to another server via ssh.. Like this: tar cfp - /home | gzip | ssh other.server (cd /backups && tar xfp -).. I have tried gzipping the stream on the source side, gzipping on the destination, and running without gzip, all three take an amazingly long time to complete (> 14 hours). Ours is a modest server with about 1300 users, about 300GB of mail total. The whole thing sits on hardware RAID-10, so I'm primarily guarding against a MAJOR hardware failure, a config blunder, or a hacker wiping out files. How do you guys back up your maildir? I've considered exporing the maildir with NFS and backing it up from another server.. Any opinions on that? Thanks, Shayne -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://dovecot.org/pipermail/dovecot/attachments/20060510/4ce5c184/attachment-0001.html>
On Wed, 2006-05-10 at 19:29 -0500, Shayne Hardesty wrote:> We switched our mail server over from mbox to maildir a few months > ago and couldn't be happier - the performance under normal load is > incredible. However we now have a problem with backup.. Typically we > would run tar on the mail server, sending its output to another server > via ssh.. Like this: tar cfp - /home | gzip | ssh other.server > (cd /backups && tar xfp -).. I have tried gzipping the stream on the > source side, gzipping on the destination, and running without gzip, > all three take an amazingly long time to complete (> 14 hours). Ours > is a modest server with about 1300 users, about 300GB of mail total. > The whole thing sits on hardware RAID-10, so I'm primarily guarding > against a MAJOR hardware failure, a config blunder, or a hacker wiping > out files. How do you guys back up your maildir? I've considered > exporing the maildir with NFS and backing it up from another server.. > Any opinions on that?I've had great success with rsync and hard link based backup. I've customised my own solution based on the scripts found at http://www.mikerubel.org/computers/rsync_snapshots/ -- Karl Latiss <karl.latiss at atvert.com.au> Atvert Systems
On Wed, May 10, 2006 at 07:29:02PM -0500, Shayne Hardesty wrote:> We switched our mail server over from mbox to maildir a few months ago and > couldn't be happier - the performance under normal load is incredible. > However we now have a problem with backup.. Typically we would run tar on > the mail server, sending its output to another server via ssh.. Like this: > tar cfp - /home | gzip | ssh other.server (cd /backups && tar xfp -).. I > have tried gzipping the stream on the source side, gzipping on the > destination, and running without gzip, all three take an amazingly long time > to complete (> 14 hours). Ours is a modest server with about 1300 users, > about 300GB of mail total. The whole thing sits on hardware RAID-10, so I'm > primarily guarding against a MAJOR hardware failure, a config blunder, or a > hacker wiping out files. How do you guys back up your maildir? I've > considered exporing the maildir with NFS and backing it up from another > server.. Any opinions on that?any maildir backup solution that involves reading files from the filesystem will be inherently slower because there are simply lots of files. a filesystem that supports snapshots is quite possibly the most sensible choice for filesystems that contain lots of small files. I am running maildir on ZFS, each user has their own ZFS file system, and users' mail is backed up using "zfs snapshot" and "zfs send". as these operate at the filesystem-block level, they are significantly faster than any other backup method which reads files from the filesystem. also, since you mention it, a mirror is not going to help you in the case of a config error, or a hacker trashing files, as you simply end up with two copies of the same (possibly useless) bits on disk. grant.
On May 10, 2006, at 6:29 PM, Shayne Hardesty wrote:> We switched our mail server over from mbox to maildir a few > months ago and couldn't be happier - the performance under normal > load is incredible. However we now have a problem with backup.. > Typically we would run tar on the mail server, sending its output > to another server via ssh.. Like this: tar cfp - /home | gzip | > ssh other.server (cd /backups && tar xfp -).. I have tried > gzipping the stream on the source side, gzipping on the > destination, and running without gzip, all three take an amazingly > long time to complete (> 14 hours). Ours is a modest server with > about 1300 users, about 300GB of mail total. The whole thing sits > on hardware RAID-10, so I'm primarily guarding against a MAJOR > hardware failure, a config blunder, or a hacker wiping out files. > How do you guys back up your maildir? I've considered exporing the > maildir with NFS and backing it up from another server.. Any > opinions on that?I am on FreeBSD with ufs2. Don't know what you Penguin folks have that would do similar. ufs2 supports "snapshots" and I just do a dump/restore with the -L option on dump, which triggers a snapshot. I dump to a file on another disk and gzip it (in a pipe). 19Gb takes less than an hour and the snapshot ensures consistency. I like the ZFS idea and am planning on implementing that when our backend switches (if it does -- in planning stages now) to Solaris 10... Similar things are available with ZFS. Chad> > Thanks, > Shayne >--- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://dovecot.org/pipermail/dovecot/attachments/20060514/b1a36da8/attachment-0001.html>