freebsd stable - Mar 2011 - Backup tool fot ZFS with all "classic dump(8)" fetatures -- what should I use? (or is here any way to make dump -L works well on large FFS2+SU?)

Hello, Freebsd-stable.

  Now I'm backing up my HOME filesystem with dump(8). It works
perfectly for 80GiB FS with many features: snapshot for consistency,
levels, "nodump" flag (my users use it a lot!), ability to extract
only one removed file from backup without restoring full FS, simple
sctipy wrap-up for levels schedule, etc.

  On new server I  have huge HOME (500GiB). And even if it is filled
up only with 25GiB of data, creating snapshot takes about 10 minutes,
freeze all I/O, and sometimes FAILS (!!!).

  I'm thinking to transfer GOME filesystem to ZFS. But I can not find
appropriate tools for backing it up. Here is some requirements:

 (1) One-file (one-stream) backup. Not directory mirror. I need to
     store it on FTP server and upload with single command.

 (2) Levels & increment backups. Now I have "Monthly (0) - Weekly
    (1,2,3) - daily (4,5,6,7,8,9)" scheme. I could afford other schemes,
    but if they doesn't store full backup every day and doesn't need
full
    backup more often than weekly.

 (3) Minimum of local metadata. Storing previous backups locally to
     calculate next one is not appropriate solution. "zfs send" needs
     previous snapshots for incremental backup, for example.

 (4) Working with snapshot (I think, it is trivial in case of ZFS).

 (5) Backup exclusions should be controlled by users (not super-user)
themselves,
     like "nodump" flag in case of FFS/dump(8). "zfs send"
can not
     provide this. I have very responsible users, so full backup
     now takes only up to 10GiB when all HOME FS is about 25GiB, so it
     is big help when backup is sent over Internet to other host.

 (6) Storing of ALL FS-specific information -- ACLs, etc.

 (7) Free :)

  Is here something like this for ZFS? "zfs send" looks promising,
 EXCEPT item (5) and, maybe, (3) :(

   gnu tar looks like everything but (6) :(

-- 
// Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>

on 28/03/2011 10:20 Lev Serebryakov said the following:
> Hello, Freebsd-stable.
> 
>   Now I'm backing up my HOME filesystem with dump(8). It works
> perfectly for 80GiB FS with many features: snapshot for consistency,
> levels, "nodump" flag (my users use it a lot!), ability to
extract
> only one removed file from backup without restoring full FS, simple
> sctipy wrap-up for levels schedule, etc.
> 
>   On new server I  have huge HOME (500GiB). And even if it is filled
> up only with 25GiB of data, creating snapshot takes about 10 minutes,
> freeze all I/O, and sometimes FAILS (!!!).
> 
>   I'm thinking to transfer GOME filesystem to ZFS. But I can not find
> appropriate tools for backing it up. Here is some requirements:
> 
>  (1) One-file (one-stream) backup. Not directory mirror. I need to
>      store it on FTP server and upload with single command.
> 
>  (2) Levels & increment backups. Now I have "Monthly (0) - Weekly
>     (1,2,3) - daily (4,5,6,7,8,9)" scheme. I could afford other
schemes,
>     but if they doesn't store full backup every day and doesn't
need full
>     backup more often than weekly.
> 
>  (3) Minimum of local metadata. Storing previous backups locally to
>      calculate next one is not appropriate solution. "zfs send"
needs
>      previous snapshots for incremental backup, for example.
> 
>  (4) Working with snapshot (I think, it is trivial in case of ZFS).
> 
>  (5) Backup exclusions should be controlled by users (not super-user)
themselves,
>      like "nodump" flag in case of FFS/dump(8). "zfs
send" can not
>      provide this. I have very responsible users, so full backup
>      now takes only up to 10GiB when all HOME FS is about 25GiB, so it
>      is big help when backup is sent over Internet to other host.
> 
>  (6) Storing of ALL FS-specific information -- ACLs, etc.
> 
>  (7) Free :)
> 
>   Is here something like this for ZFS? "zfs send" looks
promising,
>  EXCEPT item (5) and, maybe, (3) :(
> 
>    gnu tar looks like everything but (6) :(
I have a script built around zfs snapshot and star (archivers/star) that has
functionality similar to your requirements.

-- 
Andriy Gapon

On Mon, 28 Mar 2011 03:20, lev@ wrote:
> Hello, Freebsd-stable.
>
>  Now I'm backing up my HOME filesystem with dump(8). It works
> perfectly for 80GiB FS with many features: snapshot for consistency,
> levels, "nodump" flag (my users use it a lot!), ability to
extract
> only one removed file from backup without restoring full FS, simple
> sctipy wrap-up for levels schedule, etc.
>
>  On new server I  have huge HOME (500GiB). And even if it is filled
> up only with 25GiB of data, creating snapshot takes about 10 minutes,
> freeze all I/O, and sometimes FAILS (!!!).
>
>  I'm thinking to transfer GOME filesystem to ZFS. But I can not find
> appropriate tools for backing it up. Here is some requirements:
>
> (1) One-file (one-stream) backup. Not directory mirror. I need to
>     store it on FTP server and upload with single command.
>
> (2) Levels & increment backups. Now I have "Monthly (0) - Weekly
>    (1,2,3) - daily (4,5,6,7,8,9)" scheme. I could afford other
schemes,
>    but if they doesn't store full backup every day and doesn't need
full
>    backup more often than weekly.
>
> (3) Minimum of local metadata. Storing previous backups locally to
>     calculate next one is not appropriate solution. "zfs send"
needs
>     previous snapshots for incremental backup, for example.
>
> (4) Working with snapshot (I think, it is trivial in case of ZFS).
>
> (5) Backup exclusions should be controlled by users (not super-user)
themselves,
>     like "nodump" flag in case of FFS/dump(8). "zfs
send" can not
>     provide this. I have very responsible users, so full backup
>     now takes only up to 10GiB when all HOME FS is about 25GiB, so it
>     is big help when backup is sent over Internet to other host.
>
> (6) Storing of ALL FS-specific information -- ACLs, etc.
>
> (7) Free :)
>
>  Is here something like this for ZFS? "zfs send" looks promising,
> EXCEPT item (5) and, maybe, (3) :(
>
>   gnu tar looks like everything but (6) :(
>

There is information all over the place for this. I would suggest that you 
take the time and go over the required reading to understand ZFS and its 
concepts before you jump into conclusions.

Here is some docs to start with.
http://download.oracle.com/docs/cd/E19253-01/819-5461/index.html
http://hub.opensolaris.org/bin/view/Community+Group+zfs/faq
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide


-- 

  Regards,

  J. Hellenthal
  (0x89D8547E)
  JJH48-ARIN

Looked at rsync or tarsnap?

On Mon, 28 Mar 2011 09:20:07 +0200, Lev Serebryakov <lev@freebsd.org>  
wrote:
> Hello, Freebsd-stable.
>
>   Now I'm backing up my HOME filesystem with dump(8). It works
> perfectly for 80GiB FS with many features: snapshot for consistency,
> levels, "nodump" flag (my users use it a lot!), ability to
extract
> only one removed file from backup without restoring full FS, simple
> sctipy wrap-up for levels schedule, etc.
>
>   On new server I  have huge HOME (500GiB). And even if it is filled
> up only with 25GiB of data, creating snapshot takes about 10 minutes,
> freeze all I/O, and sometimes FAILS (!!!).
>
>   I'm thinking to transfer GOME filesystem to ZFS. But I can not find
> appropriate tools for backing it up. Here is some requirements:
>
>  (1) One-file (one-stream) backup. Not directory mirror. I need to
>      store it on FTP server and upload with single command.
>
>  (2) Levels & increment backups. Now I have "Monthly (0) - Weekly
>     (1,2,3) - daily (4,5,6,7,8,9)" scheme. I could afford other
schemes,
>     but if they doesn't store full backup every day and doesn't
need full
>     backup more often than weekly.
>
>  (3) Minimum of local metadata. Storing previous backups locally to
>      calculate next one is not appropriate solution. "zfs send"
needs
>      previous snapshots for incremental backup, for example.
>
>  (4) Working with snapshot (I think, it is trivial in case of ZFS).
>
>  (5) Backup exclusions should be controlled by users (not super-user)  
> themselves,
>      like "nodump" flag in case of FFS/dump(8). "zfs
send" can not
>      provide this. I have very responsible users, so full backup
>      now takes only up to 10GiB when all HOME FS is about 25GiB, so it
>      is big help when backup is sent over Internet to other host.
>
>  (6) Storing of ALL FS-specific information -- ACLs, etc.
>
>  (7) Free :)
>
>   Is here something like this for ZFS? "zfs send" looks
promising,
>  EXCEPT item (5) and, maybe, (3) :(
>
>    gnu tar looks like everything but (6) :(

On Mon, 28 Mar 2011 03:20:07 -0400, Lev Serebryakov <lev@freebsd.org>  
wrote:

>   I'm thinking to transfer GOME filesystem to ZFS. But I can not find
> appropriate tools for backing it up. Here is some requirements:
Have you considered a full-up backup solution, like bacula? It's a  
client/server/server model backup system - there's a server process that  
coordinates all actions ('director'), various server process that run on
machines with the devices/mounts/disks for storing the backups ('storage  
daemons') and then each client runs a little process to give access to the  
backup servers ('file daemons').

It allows you to specify a large amount of behavior. You can store backup  
to disk/file and to tape. If using disks/files, you can backup to the same  
file always, backup to files with 1gb max etc, or backup to a new file  
each time iirc. It has support for arbitrary schedules with each schedule  
being able to specify the dump level (full, incremental, differential) It  
uses a database in the director for metadata. And, iirc, it honors the  
nodump flag, stores ACLs, etc.

Most importantly, it has support for pre- and post-backup hooks, so you  
can tell it to snapshot beforehand and then (probably, see below) use the  
post-hook to push the data where you want.

Reading about your requirement #1, I'm guessing that the backup data is  
being collected locally and then sent over ftp for permanent storage. Do  
you have control over this remote machine? Could you replace ftp with  
bacula's networked client/server model? This might be the one spot that  
would be hard to make bacula work for you, I'm not sure since I haven't
played with bacula in this configuration and I'm not exactly sure what  
your restrictions are.

Even then, you could probably mount the FTP server as a 'file system'
ala
sshfs and have the storage daemon write it directly to the mounted file  
system.

And yeah, it's free. http://www.bacula.org

If you want to give it a shot, you can set it up on a little test machine  
and have it backup to itself. I might recommend doing this anyway since  
you'll want to be able to experiment with configuration and controls  
before trying it on your production machine.

--Kevin