rsync - Feb 2017 - Alternatives to rsync. Was: Huge directory tree: Get files to sync via tools like sysdig

Am 09.02.2017 um 11:05 schrieb Ben RUBSON:
>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at
thomas-guettler.de> wrote:
>>
>> Hi,
>>
>> we have a huge directory tree.
>>
>>
>> * 17M files (number of files)
>> * 2.2TBytes of data.
>> * Only 0.1% changes per day
>>
>> Current pain: rsyncs directory tree traversal needs to long to discover
the changed files.
>
> Hi,
>
> On which type of FS is this directory ?
ext4


-- 
Thomas Guettler http://www.thomas-guettler.de/

> On 09 Feb 2017, at 16:10, Thomas Güttler <guettliml at
thomas-guettler.de> wrote:
> 
> Am 09.02.2017 um 11:05 schrieb Ben RUBSON:
>>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at
thomas-guettler.de> wrote:
>>> 
>>> Hi,
>>> 
>>> we have a huge directory tree.
>>> 
>>> 
>>> * 17M files (number of files)
>>> * 2.2TBytes of data.
>>> * Only 0.1% changes per day
>>> 
>>> Current pain: rsyncs directory tree traversal needs to long to
discover the changed files.
>> 
>> Hi,
>> 
>> On which type of FS is this directory ?
> 
> ext4
Any way to prefer snapshots in your backup strategy ?
Or to use a ZFS ready OS to benefit from a SSD cache (which would store your
metadata) ?

As Ben mentioned, ZFS snapshots is one possible approach. Another approach is to
have a faster storage system. I have seen considerable speed improvements with
rsync on similar data sets by say upgrading the storage sub system.

--------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

Am 09.02.2017 um 16:21 schrieb Ben RUBSON:
>
>> On 09 Feb 2017, at 16:10, Thomas Güttler <guettliml at
thomas-guettler.de> wrote:
>>
>> Am 09.02.2017 um 11:05 schrieb Ben RUBSON:
>>>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at
thomas-guettler.de> wrote:
>>>>
>>>> Hi,
>>>>
>>>> we have a huge directory tree.
>>>>
>>>>
>>>> * 17M files (number of files)
>>>> * 2.2TBytes of data.
>>>> * Only 0.1% changes per day
>>>>
>>>> Current pain: rsyncs directory tree traversal needs to long to
discover the changed files.
>>>
>>> Hi,
>>>
>>> On which type of FS is this directory ?
>>
>> ext4
>
> Any way to prefer snapshots in your backup strategy ?
> Or to use a ZFS ready OS to benefit from a SSD cache (which would store
your metadata) ?
Yes, I think rsync is coming to the edge of its capabilities here. I guess a
different strategy is needed.

I see these alternatives to rsync:

  - Incremental Snapshots at block-level device is one of them.
  - We get the application ported to access a storage server, and not file
server any more.
  - ....

Do you see other alternatives?

Regards,
   Thomas Güttler



-- 
Thomas Guettler http://www.thomas-guettler.de/