Linda W
2013-Feb-01 23:58 UTC
[Samba] snapshot scripts run on linux to allow samba to export Windows "Previous copies" when looking at a network file?
Does anyone have scripts they use for doing this automatically that they use? I'd like to compare it to mine and see what features I might have left out that I might want... Currently nightly, the script determines the changes during the day (snaps take about 60-150 minutes to create, so once a day is all I do right now). It uses a simple config-file to drive what directories to ignore and what snaps to 'expire' (I only keep them about 3 weeks -- older than that I go to backups -- so it's mostly a convenience thing). It does NOT, like the windows version have ALL the files existing at that time.. The snaps only contain files that are different from their current state (or existed then but have been deleted) -- so it's a sparse snap that allows me to focus on what's changed vs. seeing some artificial state of the fs at some point in the past. I use rsync and lvm to create snapshots. It runs an active snap/day, then it starts the snap for the next day. Order of events: 1) Mount any unmounted snaps (good for when system first comes up), as you don't want the snaps in your fstab cluttering things up. 2) expire any old snaps via a config-file formula 3) check if a snap has been started for today yet, if not start one. 4) check if there is an "older" snap to diff from than the one just started. (so it can do more than one snap a day if it were forced -- an option). 5) Using a 3rd static volume, it copies the differences between yesterdays' state and today's (with state copied being up-to-the minute differences until processed by rsync. 6) When done deactivates and removes the older active snap it just made and creates a new, small static partition sized to hold just the changes & copies the changes from the 3rd static volume. 7 When done, it labels it, mounts it in the proper directory for windows to see it as a "previous copy cache", and cleans out the 3rd volume to leave it empty. I've had it running consistently since Dec 12 with few errors. (sometimes it can't find ACL information that it things should be there... but that seems to be rare).. Right now, log output sample for the plast 3 days looks like: Ishtar:law/var/log> for file in "snap.log-1301"{1{8,9},20}* do echo "------" echo $file: echo ====cat "$file" done ------ snap.log-130118-0507: ====0 snaps dated today. Snapshot of Home wanted: y Create snapshot Home-2013.01.17-05.07.02, size 1.0T Logical volume "Home-2013.01.17-05.07.02" created Oldest Snapshot = Home-2013.01.16-05.07.03, @/home/.snapdir/@GMT-2013.01.16-05.07.03 Performing rsync with 7 excludes used from config file. rsync took 115m, 11s Empty-directory removal took 0m, 48s Find size of dir @ /home.diff size=14248587264, minsize=17098304717, extent_size=4194304, nextents=4077 Old volume active: Deactivated. Removed. size=17100177408 Create vol. Home-2013.01.16-05.07.03, size 15.9G Logical volume "Home-2013.01.16-05.07.03" created About to copy base-diff dir to static Copying diffs to dated static snap...Time: 403m, 11s. mklabel@ /home/.snapdir/@GMT-2013.01.16-05.07.03/./._snapdat_=snap_copy_complete after copy2staticsnap, status=1, !------ snap.log-130119-0507: ====delete vol Home-2013.01.09-05.07.02 Old volume active: Deactivated. Removed. Deleted 1 snaps out of 14 for this base 0 snaps dated today. Snapshot of Home wanted: y Create snapshot Home-2013.01.18-05.07.03, size 1.0T Logical volume "Home-2013.01.18-05.07.03" created Oldest Snapshot = Home-2013.01.17-05.07.02, @/home/.snapdir/@GMT-2013.01.17-05.07.02 Performing rsync with 7 excludes used from config file. rsync took 109m, 32s Empty-directory removal took 0m, 46s Find size of dir @ /home.diff size=4158898176, minsize=4990677811, extent_size=4194304, nextents=1190 Old volume active: Deactivated. Removed. size=4991221760 Create vol. Home-2013.01.17-05.07.02, size 4.6G Logical volume "Home-2013.01.17-05.07.02" created About to copy base-diff dir to static Copying diffs to dated static snap...Time: 0m, 22s. mklabel@ /home/.snapdir/@GMT-2013.01.17-05.07.02/./._snapdat_=snap_copy_complete after copy2staticsnap, status=1, !------ snap.log-130120-0507: ====delete vol Home-2013.01.02-17.08.56 Old volume active: Deactivated. Removed. Deleted 1 snaps out of 14 for this base 0 snaps dated today. Snapshot of Home wanted: y Create snapshot Home-2013.01.19-05.07.08, size 1.0T Logical volume "Home-2013.01.19-05.07.08" created Oldest Snapshot = Home-2013.01.18-05.07.03, @/home/.snapdir/@GMT-2013.01.18-05.07.03 Performing rsync with 7 excludes used from config file. rsync took 175m, 44s Empty-directory removal took 0m, 48s Find size of dir @ /home.diff size=45600841728, minsize=54721010074, extent_size=4194304, nextents=13047 Old volume active: Deactivated. Removed. size=54723084288 Create vol. Home-2013.01.18-05.07.03, size 51.0G Logical volume "Home-2013.01.18-05.07.03" created About to copy base-diff dir to static Copying diffs to dated static snap...Time: 6m, 31s. mklabel@ /home/.snapdir/@GMT-2013.01.18-05.07.03/./._snapdat_=snap_copy_complete after copy2staticsnap, status=1, ! ------------------------------------- Currently my active snapshot look like this:> lvsLV VG Attr LSize Origin Snap% Move Log Copy% Convert Backups Backups -wi-ao 10.91t Home Home+Space owi-ao 1.00t Home-2013.01.08-05.07.02 Home+Space -wi-ao 21.32g Home-2013.01.12-05.07.03 Home+Space -wi-ao 1.38g Home-2013.01.16-05.07.03 Home+Space -wi-ao 15.93g Home-2013.01.18-05.07.03 Home+Space -wi-ao 50.96g Home-2013.01.20-05.07.02 Home+Space -wi-ao 1.38g Home-2013.01.22-15.08.15 Home+Space -wi-ao 240.84g Home-2013.01.25-17.45.05 Home+Space -wi-ao 1.35g Home-2013.01.26-05.07.04 Home+Space -wi-ao 1.64g Home-2013.01.27-05.07.02 Home+Space -wi-ao 5.59g Home-2013.01.28-05.07.02 Home+Space -wi-ao 3.11g Home-2013.01.29-05.07.02 Home+Space -wi-ao 2.26g Home-2013.01.30-05.07.02 Home+Space -wi-ao 3.15g Home-2013.01.31-05.07.01 Home+Space -wi-ao 65.70g Home-2013.02.01-05.07.03 Home+Space swi-ao 1.00t Home 0.03 Home.diff Home+Space -wc-ao 512.00g So I keep about 14-15 snaps going back about a month . I am currently keeping fewer files as it goes back - which means a change I might have wanted that happened on a day I deleted would be there (would have to restore from backups instead). This is not a backup replacement -- it's a convenience to easily allow a windows user to restore recent copies of a file they might have deleted or changed. It's written in pure perl, so it's open source when I publish it. I was wondering if others had tools they used to do this type of snapshotting? You can see by the sizes of the partitions the amount of change/day varies and why I size down the partition after creating the diff (on /Home.diff). I use the xfs file system because it stores ACL's and windows extended attributes. I was thinking of putting in a project on one of the project host systems (not sure which would be best at this point)..... Feedback? Other systems? Ignore? Whatever...;-) I have ideas for possible improvements, but have been working on another project lately -- so it's been a good 'testing period' just to let this one run.... Linda