rsync - Apr 2018 - [Bug 13388] New: Feature request: When deleting files only delete files that are over a certain age.

https://bugzilla.samba.org/show_bug.cgi?id=13388

            Bug ID: 13388
           Summary: Feature request: When deleting files only delete files
                    that are over a certain age.
           Product: rsync
           Version: 3.1.3
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: arthur.gouros at bom.gov.au
        QA Contact: rsync-qa at samba.org

Hi,

Feature request: New option: --min-age-delete=SECONDS    Only delete files that
are over a certain age.

This situation came about when I have source servers in multiple data centres
that very frequently update the same target via rsync using the --update
option. 

Being separate data centres they eventually generate the same file manifests
but sometimes there is a small period of time where the manifests of the source
servers differ between each data centre.

Using the --update option solves the problem of only updating the target with
the latest file, however if the source server generates files that incorporate
timestamps in their names then a condition can arise at the target server where
one source server's rsync deletes the timestamped files of another source
server's rsync. 

Consider this example. 

source-server-1
    example.html
        image_201801011400.gif

source-server-2
    example.html
        image_201801011401.gif

Suppose the resulting gif file is generated by some local process on the source
server (e.g. an ingestion process).

Using the --update option, whichever source server has the latest example.html
file then that will end up at the target destination. 
Unfortunately the same cannot be said for the gif file so what ends up at the
target destination is dependent on which source server ran the last rsync
(remember they run rsync very frequently).
This means there is a potential that the file example.html contains a reference
to a file that doesn't exist on the target rsync destination.

By having the following option available to rsync --min-age-delete=60 then any
files that qualify for deletion will need to be greater than 60 seconds of age.

Used in conjunction with the --update option this will allow me to use multiple
source servers to update the same target destination via rsync.

The default value for the proposed option --min-age-delete can be 0 which
preserves backwards compatability.

The proposed enhancement can be implemented around the Posix unlink(3c)
function call which will also preserve compatability with the existing
--delete-* options.

At the momement, to overcome this issue I need to run a two step process.
1. Produce a list of files that haven't changed in the last X seconds.
2. Add that file list to an rsync exclude option.
This is not very elegant and I would prefer to have this capability as a
built-in feature of rsync.

Thank you,
Arthur Gouros.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.