dovecot - Jan 2010 - Anyone successfully setup Continous Backup of mailboxes using rsync ?

Hi All..

Am working on continous backup of mailboxes using rsync(for e.g by running rsync
every 2 min)

Things gets more complicated when users create Subfolders in INBOX , SENT ,
etc..

If anyone among you did that plz guide & advise.

Thanks
CoolAtt

 		 	   		  
_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
https://signup.live.com/signup.aspx?id=60969

The answer, if you can sell the cost, might be network storage.  Most of 
them provide the capability of take inode snapshots
CoolAtt NNA wrote:
> Hi All..
>
> Am working on continous backup of mailboxes using rsync(for e.g by running
rsync every 2 min)
>
> Things gets more complicated when users create Subfolders in INBOX , SENT ,
etc..
>
> If anyone among you did that plz guide & advise.
>
> Thanks
> CoolAtt
>
>  		 	   		  
> _________________________________________________________________
> Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
> https://signup.live.com/signup.aspx?id=60969
>   
-- 
==== Once upon a time, the Internet was a friendly, 
neighbors-helping-neighbors small town, and no one locked their doors. 
Now it's like an apartment in Bed-Stuy: you need three heavy duty 
pick-proof locks, one of those braces that goes from the lock to the 
floor, and bars on the windows.... ==== Stewart Dean, Unix System Admin, 
Bard College, New York 12504 sdean at bard.edu voice: 845-758-7475, fax: 
845-758-7035

On 28.01.2010 09:07, CoolAtt NNA wrote:
>
> Hi All..
>
> Am working on continous backup of mailboxes using rsync(for e.g by running
rsync every 2 min)
>
> Things gets more complicated when users create Subfolders in INBOX , SENT ,
etc..
>
> If anyone among you did that plz guide&  advise.
>
You don't state much on what this runs on, but I do snapshot backups of 
my IMAP server consistently by using LVM, and just match my cronjob to 
snapshot it at a few given times, mounting the snapshot and backing up 
with rdiff-backup.

Very simple, very easy, and very realiable. :)
> Thanks
> CoolAtt
>
-A

On 1/28/10 12:07 AM, CoolAtt NNA wrote:
> 
> Hi All..
> 
> Am working on continous backup of mailboxes using rsync(for e.g by running
rsync every 2 min)
> 
> Things gets more complicated when users create Subfolders in INBOX , SENT ,
etc..
How?

> If anyone among you did that plz guide & advise.
> 
Sure:

rsync -e 'ssh -ax -i mykey' \
-a --delete --stats \
mailserver.host:/var/mail/vmail \
/localbackup


~Seth

Hi Luigi,

Thats great.
Can you please explain how you do the restore?

I plan to do restore of 1 or more mail files.
If I simply copy a mailfile  e.g "MailDir/cur/my-email-file:2,S" to
"MailDir/cur/" on the live system, are there issues associated
with it ?

Thanks
CoolAtt
                                        
> Date: Fri, 29 Jan 2010 06:39:31 +0100
> From: lists at luigirosa.com
> To: dovecot at dovecot.org
> Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of
mailboxes using rsync ?
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> CoolAtt NNA said the following on 29/01/10 06:21:
> 
> > rsync -avt --log-file=$log_file /home/vmail/ 10.0.0.x:/home/vmail
> > I run it in a script for e,g every 1 min.
> > By doing so if an email is deleted on the live server it still remain
on the backup server.
> > Then I give each mail file a retention period of n days so that it
automatically gets deleted from the backup server.
> 
> I use a similar strategy, with a more relaxed synchronization. But instead
of
> setting an expire on backup system, I simply do a rsync --del every night.
> 
> 
> 
> Ciao,
> luigi
> 
> - -- 
> /
> +--[Luigi Rosa]--
> \
> 
> Oo-ta goo-ta, Solo?
>     --Greedo, "Star Wars"
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAktidI8ACgkQ3kWu7Tfl6ZRfEACfRfJR4z99TIEgylKSywVAa4px
> l1kAnRA6xyigbYl+BVZoi6d5Vi5/n4Yk
> =f8lP
> -----END PGP SIGNATURE-----
 		 	   		  
_________________________________________________________________
Hotmail: Trusted email with Microsoft?s powerful SPAM protection.
https://signup.live.com/signup.aspx?id=60969

Am using Dovecot & the maildir format.

Create a new folder like the following ?? 

 vmail/coolatt/Maildir/restored-emails


> Date: Fri, 29 Jan 2010 07:20:37 +0100
> From: lists at luigirosa.com
> To: dovecot at dovecot.org
> Subject: Re: [Dovecot] Anyone successfully setup Continous Backup of
mailboxes using rsync ?
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> CoolAtt NNA said the following on 29/01/10 07:18:
> 
> > It means if I create a new mailbox and I restore the emails at mail
level(individual files) ,will it work correctly?
> > And if I restore the same mail to an existing mailbox ??
> 
> 
> Not a new mailbox, a new mail folder (maildir directory) of the actual
account.
> 
> I am assuming you are using IMAP ad maildir, of course.
> 
> 
> Ciao,
> luigi
> 
> - -- 
> /
> +--[Luigi Rosa]--
> \
> 
> Scott me up, Beamie!
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAktifjUACgkQ3kWu7Tfl6ZQalACfaEsS0ME6ab27uks49W6ERj59
> yAcAnRDmaN3ZCKAEzJrK41CjZK28B3aT
> =O5hj
> -----END PGP SIGNATURE-----
 		 	   		  
_________________________________________________________________
Hotmail: Powerful Free email with security by Microsoft.
https://signup.live.com/signup.aspx?id=60969

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

CoolAtt NNA said the following on 29/01/10 07:39:
> Create a new folder like the following ?? 
> 
>  vmail/coolatt/Maildir/restored-emails
Actually vmail/coolatt/Maildir/.restored-emails would be better :)
Done that, you copy from the folder to be restored cur, tmp, new.

Some email clients need to shutdown and restart to be aware of the new mail
folder.



Ciao,
luigi

- -- 
/
+--[Luigi Rosa]--
\

Moneypenny: In the conference room. Something pretty big.
Every double-o man in Europe has been rushed in.
And the home secretary too!
James Bond: His wife probably lost her dog.
    --"007 Thunderball"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktihLQACgkQ3kWu7Tfl6ZSMzQCghKGCWeqflRH4SatM/YItp3HG
Ba0AoMFM4CVAdHtvsjNv7k0FLWFeYlw+
=6eIt
-----END PGP SIGNATURE-----

On Thu, 28 Jan 2010 13:07:32 +0500
CoolAtt NNA <coolatt at live.com> wrote:
> 
> Hi All..
> 
> Am working on continous backup of mailboxes using rsync(for e.g by running
rsync every 2 min)
> 
> Things gets more complicated when users create Subfolders in INBOX , SENT ,
etc..
> 
> If anyone among you did that plz guide & advise.
> 
> Thanks
> CoolAtt
> 
Take a look on ccollect project http://unix.schottelius.org/ccollect/. It made
incremental backups using rsync. That will be working for maildirs
perfectly.
>  		 	   		  
> _________________________________________________________________
> Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
> https://signup.live.com/signup.aspx?id=60969

Le Thu, 28 Jan 2010 13:07:32 +0500,
CoolAtt NNA <coolatt at live.com> a ?crit :
> Am working on continous backup of mailboxes using rsync(for e.g by running
rsync every 2 min)
> Things gets more complicated when users create Subfolders in INBOX , SENT ,
etc..
> If anyone among you did that plz guide & advise.
I'm going to share what I've did. This is not rocket science but enough
for my SOHO needs.
Each user mail boxes are stored in ~HOME/mail/
 
# cat /opt/crontab-scripts/rsync-local-email-backup.sh
  [...]
  for i in /home/*/mail/
    rsync -a --delete $i ${PATHTOSAFEPLACE}/$(date +%u)/${i}/
    RETURNCODE=$?
    [...]
  done
  [...]

In clear, each day, I copy all mailboxes to a different (date +%u -> 0 to 6)
safe sub-directory.
I admit, it's very simple but enough for my needs and certainly could be
improved..


 Best regards,

-- 
 LMJ
 "May the source be with you my young padawan"
 http://sites.google.com/site/imatruelinuxmasterjedi/

CoolAtt NNA wrote:
> Hi All..
> 
> Am working on continous backup of mailboxes using rsync(for e.g by running
rsync every 2 min)
> 
> Things gets more complicated when users create Subfolders in INBOX , SENT ,
etc..
> 
> If anyone among you did that plz guide & advise.
We have a slightly different method of doing a back-up, at least of people's
inboxes (which is all I "archive").

I've written a program, which is run by cron once a minute, which searches
for
messages within all the users' inboxes and then hard-links them into a
backup
store elsewhere on the disk. Then, once a day a reaper process runs which checks
for all files in that store with only one link and have not been accessed for a 
set amount of time (in our case 90 days) and then deletes those files. In other 
words, the messages are kept for 90 days after they have been deleted or moved 
out of the user's inbox.

I've attached the source code tar file as you or someone else may find it
useful.

In your case you could rsync the back-up store instead of the user's main 
directory structure and maybe only do it once per day.

Steve
-- 
---------------------------------------------------------------------------
IT Systems Administrator,                     E-Mail:- steve at earth.ox.ac.uk
Department of Earth Sciences,                    Tel:-   +44 (0)1865 282110
University of Oxford, Parks Road, Oxford, UK.    Fax:-   +44 (0)1865 272072
-------------- next part --------------
A non-text attachment was scrubbed...
Name: backupinboxes.tar.gz
Type: application/x-gzip
Size: 14335 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20100210/aeb370e0/attachment-0002.gz>

i use drbd to keep the mailstore redundant and rsnapshot for backup.