Hi, I have a local btrfs file system with various sub-volumes that have had snapshots done on them. Is there some tool like rsync that I could copy all the data and snapshots to a backup system, but still only use the same amount of space as the source filesystem. I see a problem being getting a consistent and steady state during the rsync. I was thinking that I might be able to do this with LVM snapshots, but that would require something along these lines: 1) pause the btrfs filesystem into a consistent state that can be mounted cleanly 2) Do LVM snapshot on it. 3) un-pause btrfs filesystem. I can then do a block level backup of the LVM snapshot and it should be mountable on the backup server. So, the snapshot is not a snapshot of the current filesystem, it is a snapshot of all the snapshots and all the sub-volumes at a particular time, that is in a stable state to be backed up. I don''t know if 1 is supported? I suppose I am hoping for 1,2,3 to already be supported by some special btrfs command. Any ideas? Kind Regards James -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 3/26/12 10:30 AM, James Courtier-Dutton wrote:> Hi, > > I have a local btrfs file system with various sub-volumes that have > had snapshots done on them. > > Is there some tool like rsync that I could copy all the data and > snapshots to a backup system, but still only use the same amount of > space as the source filesystem. > I see a problem being getting a consistent and steady state during the rsync. > I was thinking that I might be able to do this with LVM snapshots, but > that would require something along these lines: > 1) pause the btrfs filesystem into a consistent state that can be > mounted cleanly > 2) Do LVM snapshot on it. > 3) un-pause btrfs filesystem. > > I can then do a block level backup of the LVM snapshot and it should > be mountable on the backup server. > So, the snapshot is not a snapshot of the current filesystem, it is a > snapshot of all the snapshots and all the sub-volumes at a particular > time, that is in a stable state to be backed up. > > I don''t know if 1 is supported? > I suppose I am hoping for 1,2,3 to already be supported by some > special btrfs command. > > > Any ideas?I''m not sure if I understand your problem right, but I would suggest: 1) Snapshot the subvolume on the source 2) rsync the snapshot to the destination 3) Snapshot the destination Kind regards, Felix> > Kind Regards > > James > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Mar 26, 2012 at 3:56 PM, Felix Blanke <felixblanke@gmail.com> wrote:> On 3/26/12 10:30 AM, James Courtier-Dutton wrote: >> Is there some tool like rsync that I could copy all the data and >> snapshots to a backup system, but still only use the same amount of >> space as the source filesystem.> I''m not sure if I understand your problem right, but I would suggest: > > 1) Snapshot the subvolume on the source > 2) rsync the snapshot to the destination > 3) Snapshot the destinationJames did say "only use the same amount of space as the source filesystem." Your approach would increase the usage when one or more subvolume shares the same space (e.g. when one subvolume starts as snapshot). AFAIK the (planned) way to do this is using "btrfs send | receive", which is not available yet. -- Fajar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Fajar A. Nugraha posted on Mon, 26 Mar 2012 16:01:54 +0700 as excerpted:> On Mon, Mar 26, 2012 at 3:56 PM, Felix Blanke <felixblanke@gmail.com> > wrote: >> On 3/26/12 10:30 AM, James Courtier-Dutton wrote: >>> Is there some tool like rsync that I could copy all the data and >>> snapshots to a backup system, but still only use the same amount of >>> space as the source filesystem. > > >> I''m not sure if I understand your problem right, but I would suggest: >> >> 1) Snapshot the subvolume on the source 2) rsync the snapshot to the >> destination 3) Snapshot the destination > > James did say "only use the same amount of space as the source > filesystem." Your approach would increase the usage when one or more > subvolume shares the same space (e.g. when one subvolume starts as > snapshot). > > AFAIK the (planned) way to do this is using "btrfs send | receive", > which is not available yet.What about rsyncing the snapshots, one at a time, snapshotting on the destination after each one? I think that''s what Felix'' idea was. On the source filesystem, if each of the snapshots is mounted in turn and rsynced across, then rsync should only catch the differences between the previous and currently rsyncing snapshots. On the destination, after the first rsync, a snapshot could be taken and mounted, so the second rsync is cumulative. Then a second snapshot can be taken, then it mounted, for the next rsync. Given COW, I''d think that''d work. That''s in contrast to an attempted rsync of the root filesystem, which would appear to rsync as if each snapshot was a separate directory tree, which would indeed kill the data sharing between them, thus taking up N times the space of one snapshot on the destination. But if each snapshot is mounted in turn on both sides, destination of course trailing source by one snapshot, in theory at least, it should work, tho it depends on the rsync implementation being COW-friendly and I''m not positive it is but expect that it should be. Here, I''d probably do it manually the first few snapshot generations, checking usage on the destination to see that it was working as intended as I went and ensuring I had the process down, then script parts of it, automating the parts, before ultimately combining the scripts into a full automation that, depending on the intent, could ideally be run from a cron job or the like. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Mar 26, 2012 at 4:26 PM, Duncan <1i5t5.duncan@cox.net> wrote:> Fajar A. Nugraha posted on Mon, 26 Mar 2012 16:01:54 +0700 as excerpted: > >> On Mon, Mar 26, 2012 at 3:56 PM, Felix Blanke <felixblanke@gmail.com> >> wrote: >>> On 3/26/12 10:30 AM, James Courtier-Dutton wrote: >>>> Is there some tool like rsync that I could copy all the data and >>>> snapshots to a backup system, but still only use the same amount of >>>> space as the source filesystem. >> >> >>> I''m not sure if I understand your problem right, but I would suggest: >>> >>> 1) Snapshot the subvolume on the source 2) rsync the snapshot to the >>> destination 3) Snapshot the destination >> >> James did say "only use the same amount of space as the source >> filesystem." Your approach would increase the usage when one or more >> subvolume shares the same space (e.g. when one subvolume starts as >> snapshot). >> >> AFAIK the (planned) way to do this is using "btrfs send | receive", >> which is not available yet. > > What about rsyncing the snapshots, one at a time, snapshotting on the > destination after each one? I think that''s what Felix'' idea was. > > On the source filesystem, if each of the snapshots is mounted in turn and > rsynced across, then rsync should only catch the differences between the > previous and currently rsyncing snapshots. > > On the destination, after the first rsync, a snapshot could be taken and > mounted, so the second rsync is cumulative. Then a second snapshot can > be taken, then it mounted, for the next rsync. Given COW, I''d think > that''d work. > > That''s in contrast to an attempted rsync of the root filesystem, which > would appear to rsync as if each snapshot was a separate directory tree, > which would indeed kill the data sharing between them, thus taking up N > times the space of one snapshot on the destination. > > But if each snapshot is mounted in turn on both sides, destination of > course trailing source by one snapshot, in theory at least, it should > work, tho it depends on the rsync implementation being COW-friendly and > I''m not positive it is but expect that it should be. > > Here, I''d probably do it manually the first few snapshot generations, > checking usage on the destination to see that it was working as intended > as I went and ensuring I had the process down, then script parts of it, > automating the parts, before ultimately combining the scripts into a full > automation that, depending on the intent, could ideally be run from a cron > job or the like. > > -- > Duncan - List replies preferred. No HTML msgs. > "Every nonfree program has a lord, a master -- > and if you use the program, he is your master." Richard Stallman > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.htmlI used this style of incremental backups for some time and it works well if you ignore the low performance when backing up very large files (e.g. VM images). I''m not 100% sure about this, but I think it''s important to use the --inplace option of rsync if you want to preserve COW. If not used with --inplace, rsync will create a new file every and after syncing, replace the orifinal. This will prevent COW at the moment. Maybe this won''t be required in the future when something like auto deduplication is implemented, but I currently don''t know about the plans for this feature. however, -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html