Linda W
2013-Feb-01 23:58 UTC
[Samba] snapshot scripts run on linux to allow samba to export Windows "Previous copies" when looking at a network file?
Does anyone have scripts they use for doing this automatically that they
use?
I'd like to compare it to mine and see what features I might have left out
that I might want...
Currently nightly, the script determines the changes during the day (snaps
take about 60-150 minutes to create, so once a day is all I do right now).
It uses a simple config-file to drive what directories to ignore and what
snaps to 'expire' (I only keep them about 3 weeks -- older than that I
go to
backups -- so it's mostly a convenience thing).
It does NOT, like the windows version have ALL the files existing at that
time.. The snaps only contain files that are different from their current
state (or existed then but have been deleted) -- so it's a sparse snap that
allows me to focus on what's changed vs. seeing some artificial state of the
fs at some point in the past.
I use rsync and lvm to create snapshots. It runs an active snap/day, then
it starts the snap for the next day. Order of events:
1) Mount any unmounted snaps (good for when system first comes up), as you
don't want the snaps in your fstab cluttering things up.
2) expire any old snaps via a config-file formula
3) check if a snap has been started for today yet, if not start one.
4) check if there is an "older" snap to diff from than the one just
started.
(so it can do more than one snap a day if it were forced -- an option).
5) Using a 3rd static volume, it copies the differences between yesterdays'
state and today's (with state copied being up-to-the minute differences
until processed by rsync.
6) When done deactivates and removes the older active snap it just made and
creates a new, small static partition sized to hold just the changes
& copies the changes from the 3rd static volume.
7 When done, it labels it, mounts it in the proper directory for windows to
see it as a "previous copy cache", and cleans out the 3rd volume to
leave
it empty.
I've had it running consistently since Dec 12 with few errors.
(sometimes it can't find ACL information that it things should be there...
but that seems to be rare)..
Right now, log output sample for the plast 3 days looks like:
Ishtar:law/var/log> for file in "snap.log-1301"{1{8,9},20}*
do
echo "------"
echo $file:
echo ====cat "$file"
done
------
snap.log-130118-0507:
====0 snaps dated today.
Snapshot of Home wanted: y
Create snapshot Home-2013.01.17-05.07.02, size 1.0T
Logical volume "Home-2013.01.17-05.07.02" created
Oldest Snapshot = Home-2013.01.16-05.07.03,
@/home/.snapdir/@GMT-2013.01.16-05.07.03
Performing rsync with 7 excludes used from config file.
rsync took 115m, 11s
Empty-directory removal took 0m, 48s
Find size of dir @ /home.diff
size=14248587264, minsize=17098304717, extent_size=4194304, nextents=4077
Old volume active: Deactivated. Removed.
size=17100177408
Create vol. Home-2013.01.16-05.07.03, size 15.9G
Logical volume "Home-2013.01.16-05.07.03" created
About to copy base-diff dir to static
Copying diffs to dated static snap...Time: 403m, 11s.
mklabel@
/home/.snapdir/@GMT-2013.01.16-05.07.03/./._snapdat_=snap_copy_complete
after copy2staticsnap, status=1, !------
snap.log-130119-0507:
====delete vol Home-2013.01.09-05.07.02
Old volume active: Deactivated. Removed.
Deleted 1 snaps out of 14 for this base
0 snaps dated today.
Snapshot of Home wanted: y
Create snapshot Home-2013.01.18-05.07.03, size 1.0T
Logical volume "Home-2013.01.18-05.07.03" created
Oldest Snapshot = Home-2013.01.17-05.07.02,
@/home/.snapdir/@GMT-2013.01.17-05.07.02
Performing rsync with 7 excludes used from config file.
rsync took 109m, 32s
Empty-directory removal took 0m, 46s
Find size of dir @ /home.diff
size=4158898176, minsize=4990677811, extent_size=4194304, nextents=1190
Old volume active: Deactivated. Removed.
size=4991221760
Create vol. Home-2013.01.17-05.07.02, size 4.6G
Logical volume "Home-2013.01.17-05.07.02" created
About to copy base-diff dir to static
Copying diffs to dated static snap...Time: 0m, 22s.
mklabel@
/home/.snapdir/@GMT-2013.01.17-05.07.02/./._snapdat_=snap_copy_complete
after copy2staticsnap, status=1, !------
snap.log-130120-0507:
====delete vol Home-2013.01.02-17.08.56
Old volume active: Deactivated. Removed.
Deleted 1 snaps out of 14 for this base
0 snaps dated today.
Snapshot of Home wanted: y
Create snapshot Home-2013.01.19-05.07.08, size 1.0T
Logical volume "Home-2013.01.19-05.07.08" created
Oldest Snapshot = Home-2013.01.18-05.07.03,
@/home/.snapdir/@GMT-2013.01.18-05.07.03
Performing rsync with 7 excludes used from config file.
rsync took 175m, 44s
Empty-directory removal took 0m, 48s
Find size of dir @ /home.diff
size=45600841728, minsize=54721010074, extent_size=4194304, nextents=13047
Old volume active: Deactivated. Removed.
size=54723084288
Create vol. Home-2013.01.18-05.07.03, size 51.0G
Logical volume "Home-2013.01.18-05.07.03" created
About to copy base-diff dir to static
Copying diffs to dated static snap...Time: 6m, 31s.
mklabel@
/home/.snapdir/@GMT-2013.01.18-05.07.03/./._snapdat_=snap_copy_complete
after copy2staticsnap, status=1, !
-------------------------------------
Currently my active snapshot look like this:> lvs
LV VG Attr LSize Origin Snap% Move
Log Copy% Convert
Backups Backups -wi-ao 10.91t
Home Home+Space owi-ao 1.00t
Home-2013.01.08-05.07.02 Home+Space -wi-ao 21.32g
Home-2013.01.12-05.07.03 Home+Space -wi-ao 1.38g
Home-2013.01.16-05.07.03 Home+Space -wi-ao 15.93g
Home-2013.01.18-05.07.03 Home+Space -wi-ao 50.96g
Home-2013.01.20-05.07.02 Home+Space -wi-ao 1.38g
Home-2013.01.22-15.08.15 Home+Space -wi-ao 240.84g
Home-2013.01.25-17.45.05 Home+Space -wi-ao 1.35g
Home-2013.01.26-05.07.04 Home+Space -wi-ao 1.64g
Home-2013.01.27-05.07.02 Home+Space -wi-ao 5.59g
Home-2013.01.28-05.07.02 Home+Space -wi-ao 3.11g
Home-2013.01.29-05.07.02 Home+Space -wi-ao 2.26g
Home-2013.01.30-05.07.02 Home+Space -wi-ao 3.15g
Home-2013.01.31-05.07.01 Home+Space -wi-ao 65.70g
Home-2013.02.01-05.07.03 Home+Space swi-ao 1.00t Home 0.03
Home.diff Home+Space -wc-ao 512.00g
So I keep about 14-15 snaps going back about a month . I am currently
keeping fewer files as it goes back - which means a change I might have
wanted that happened on a day I deleted would be there (would have to
restore from backups instead).
This is not a backup replacement -- it's a convenience to easily allow
a windows user to restore recent copies of a file they might have deleted or
changed.
It's written in pure perl, so it's open source when I publish it.
I was wondering if others had tools they used to do this type of
snapshotting?
You can see by the sizes of the partitions the amount of change/day varies
and why I size down the partition after creating the diff (on /Home.diff).
I use the xfs file system because it stores ACL's and windows extended
attributes.
I was thinking of putting in a project on one of the project host systems
(not sure which would be best at this point).....
Feedback? Other systems? Ignore? Whatever...;-)
I have ideas for possible improvements, but have been working on another
project lately -- so it's been a good 'testing period' just to let
this
one run....
Linda
Maybe Matching Threads
Wisdom of the Ancients
