rsync - Feb 2017 - Huge directory tree: Get files to sync via tools like sysdig

As Ben mentioned, ZFS snapshots is one possible approach. Another approach is to
have a faster storage system. I have seen considerable speed improvements with
rsync on similar data sets by say upgrading the storage sub system.

--------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

In <E48BA632-D8D5-42AF-AE67-17438C201AC5 at gmail.com>, on 02/10/17
   at 12:38 PM, Henri Shustak <henri.shustak at gmail.com> said:
>As Ben mentioned, ZFS snapshots is one possible approach. Another
>approach is to have a faster storage system. I have seen considerable
>speed improvements with rsync on similar data sets by say upgrading the
>storage sub system.
This is true.  In addition different file systems has different
performance wrt stat().

A lot depends on what kind of backup that is required.  If a full backup
that is accurate to a point in time is required, then something like ZFS
makes sense.  If the system is servers that do in memory cachinng, there
will probably be a need to ensure that their on-disk state is consistent
before the snapshot is taken.

If the only requirment is to ensure that everything known to have changed
on disk is backed up, the event based solution will avoid the high cost of
stat-ing every file.

It might be interesting to evaluate a mixed solution.  Use events to track
directory changes and let rsync sort out what to do for each directory.

Steven

-- 
----------------------------------------------------------------------
"Steven Levine" <steve53 at earthlink.net>  Warp/DIY/BlueLion
etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------

On Fri, 10 Feb 2017 12:38:32 +1300
Henri Shustak <henri.shustak at gmail.com> wrote:
> As Ben mentioned, ZFS snapshots is one possible approach. Another
> approach is to have a faster storage system. I have seen considerable
> speed improvements with rsync on similar data sets by say upgrading
> the storage sub system.
Another possibility could be to use lvm and lvmcache to throw a ssd in
front of the spinning disks.  This would only improve things if
you didn't otherwise fill up the cache with data -- you want
the cache to contain inodes.  So this might work only if your
ssd cache was larger than whatever amount of data you typically
write between rsync runs, plus enough to hold all the inodes
in your rsync-ed fs.

I've not tried this.  I'm not even certain it's a good idea. 
It's
just a thought.

Regards,

Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

That sounds like it certinally would not hurt!

--------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

> On 10 Feb 2017, at 01:21, Karl O. Pinc <kop at meme.com> wrote:
> 
> On Fri, 10 Feb 2017 12:38:32 +1300
> Henri Shustak <henri.shustak at gmail.com> wrote:
> 
>> As Ben mentioned, ZFS snapshots is one possible approach. Another
>> approach is to have a faster storage system. I have seen considerable
>> speed improvements with rsync on similar data sets by say upgrading
>> the storage sub system.
> 
> Another possibility could be to use lvm and lvmcache to throw a ssd in
> front of the spinning disks.  This would only improve things if
> you didn't otherwise fill up the cache with data -- you want
> the cache to contain inodes.  So this might work only if your
> ssd cache was larger than whatever amount of data you typically
> write between rsync runs, plus enough to hold all the inodes
> in your rsync-ed fs.
> 
> I've not tried this.  I'm not even certain it's a good idea. 
It's
> just a thought.
It's also possible to have a SSD cache with ZFS (called the L2ARC).
You can even ask this cache to only store your metadata.

Some (same ?) changes may also be needed on receiver/server side too
(depending on its current setting) to see a performance improvement.

Ben