Thomas Güttler
2017-Feb-09 15:10 UTC
Huge directory tree: Get files to sync via tools like sysdig
Am 09.02.2017 um 11:05 schrieb Ben RUBSON:>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote: >> >> Hi, >> >> we have a huge directory tree. >> >> >> * 17M files (number of files) >> * 2.2TBytes of data. >> * Only 0.1% changes per day >> >> Current pain: rsyncs directory tree traversal needs to long to discover the changed files. > > Hi, > > On which type of FS is this directory ?ext4 -- Thomas Guettler http://www.thomas-guettler.de/
Ben RUBSON
2017-Feb-09 15:21 UTC
Huge directory tree: Get files to sync via tools like sysdig
> On 09 Feb 2017, at 16:10, Thomas Güttler <guettliml at thomas-guettler.de> wrote: > > Am 09.02.2017 um 11:05 schrieb Ben RUBSON: >>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote: >>> >>> Hi, >>> >>> we have a huge directory tree. >>> >>> >>> * 17M files (number of files) >>> * 2.2TBytes of data. >>> * Only 0.1% changes per day >>> >>> Current pain: rsyncs directory tree traversal needs to long to discover the changed files. >> >> Hi, >> >> On which type of FS is this directory ? > > ext4Any way to prefer snapshots in your backup strategy ? Or to use a ZFS ready OS to benefit from a SSD cache (which would store your metadata) ?
Henri Shustak
2017-Feb-09 23:38 UTC
Huge directory tree: Get files to sync via tools like sysdig
As Ben mentioned, ZFS snapshots is one possible approach. Another approach is to have a faster storage system. I have seen considerable speed improvements with rsync on similar data sets by say upgrading the storage sub system. -------------------------------------------------------------------- This email is protected by LBackup, an open source backup solution http://www.lbackup.org
Thomas Güttler
2017-Feb-10 10:38 UTC
Alternatives to rsync. Was: Huge directory tree: Get files to sync via tools like sysdig
Am 09.02.2017 um 16:21 schrieb Ben RUBSON:> >> On 09 Feb 2017, at 16:10, Thomas Güttler <guettliml at thomas-guettler.de> wrote: >> >> Am 09.02.2017 um 11:05 schrieb Ben RUBSON: >>>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote: >>>> >>>> Hi, >>>> >>>> we have a huge directory tree. >>>> >>>> >>>> * 17M files (number of files) >>>> * 2.2TBytes of data. >>>> * Only 0.1% changes per day >>>> >>>> Current pain: rsyncs directory tree traversal needs to long to discover the changed files. >>> >>> Hi, >>> >>> On which type of FS is this directory ? >> >> ext4 > > Any way to prefer snapshots in your backup strategy ? > Or to use a ZFS ready OS to benefit from a SSD cache (which would store your metadata) ?Yes, I think rsync is coming to the edge of its capabilities here. I guess a different strategy is needed. I see these alternatives to rsync: - Incremental Snapshots at block-level device is one of them. - We get the application ported to access a storage server, and not file server any more. - .... Do you see other alternatives? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/