rsync - Aug 2011 - [Bug 8367] New: Add a feature to --move-existing files

https://bugzilla.samba.org/show_bug.cgi?id=8367

           Summary: Add a feature to --move-existing files
           Product: rsync
           Version: 3.0.8
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: devin.nate at cloudwerx.com
         QAContact: rsync-qa at samba.org


Created attachment 6766
  --> https://bugzilla.samba.org/attachment.cgi?id=6766
Patch file to add --move-existing

The use case is one where rsync is being used to replicate files from a source
to a destination, with the --remove-source-files flag being used. The files on
the source are created sequentially, based on the file existing or not.

Essentially, software creates a files named 1.tif, 2.tif, 3.tif, and so on,
based on the existance of the files. If rsync moves the files from the source
to the destination, then the files 1.tif, 2.tif, 3.tif, etc., no longer exist
on the source, so the software on the source re-starts it's numbering. Then,
when rsync is run again either: (a) it'll sync the files up, which
essentially
means all prior data was lost, since they are totally different files, or (b)
if --ignore-existing is used, the files back up on the source (good in that
there's no data loss, bad that the files don't move). 

The attached patch is a first attempt at adding a --move-existing functionality
to rsync. It does not honour --dry-run and probably a ton of other use cases.
It also uses strftime and some other functions which I see are in IFDEF blocks
in other areas. The new filename generator should probably be in util.c, and
it's logging is ugly right now. It's not well documented (aka no man
page
entry).

It will rename a file in this format:
SOURCE: filename.ext
DEST POST RENAME: filename_2011-08-09-13-44-22-1.ext

It's somewhat contrary to rsync individual file sync, since it suggests we
specifically don't want to sync a particular file. That said, it is in line
with the concept of wanting to sync a (poorly) created series of files in a
safe manner.

Thanks,
Devin

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

Devin Nate <devin.nate at cloudwerx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #6766|0                           |1
        is obsolete|                            |

--- Comment #1 from Devin Nate <devin.nate at cloudwerx.com> 2011-08-24
15:06:11 UTC ---
Created attachment 6803
  --> https://bugzilla.samba.org/attachment.cgi?id=6803
second version of patch

This is an enhancement of the original patch.

1. Adds better logging.
1a. errors are logged at all verbosity levels
1b. info messages at verbosity level > 2 (aka 3 -v's)

2. Confirmed it behaves properly with --dry-run, it does

3. CHANGED the flag from --move-existing to be --rename-existing

4. CHANGED the format of the rename logic. No longer does a long string based
on date. Now does epoch time in hex.

5. Moved to using snprintf for additional safety.

Thanks,
Devin

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

Devin Nate <devin.nate at cloudwerx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #6803|0                           |1
        is obsolete|                            |

--- Comment #2 from Devin Nate <devin.nate at cloudwerx.com> 2011-09-22
22:55:15 UTC ---
Created attachment 6931
  --> https://bugzilla.samba.org/attachment.cgi?id=6931
third version

- removes memory leak if an error occurs

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

Devin Nate <devin.nate at cloudwerx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #6931|0                           |1
        is obsolete|                            |

--- Comment #3 from Devin Nate <devin.nate at cloudwerx.com> ---
Created attachment 12104
  --> https://bugzilla.samba.org/attachment.cgi?id=12104&action=edit
rsync-3.1.2-rename-existingV6.patch

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

--- Comment #4 from Devin Nate <devin.nate at cloudwerx.com> ---
Comment on attachment 12104
  --> https://bugzilla.samba.org/attachment.cgi?id=12104
rsync-3.1.2-rename-existingV6.patch

Please find attached an updated version of this patch. It has been reworked for
additional safety (checking MAXPATHLEN and NAME_MAX better). This applies to
rsync 3.1.2, including the new debug and info message handling.

It's a pretty specific use case to need this patch, but we find it useful.
Posting here in case others have use for this also.

Thanks,
Devin

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

--- Comment #5 from Devin Nate <devin.nate at cloudwerx.com> ---
Comment on attachment 12104
  --> https://bugzilla.samba.org/attachment.cgi?id=12104
rsync-3.1.2-rename-existingV6.patch

Please find attached an updated version of this patch. It has been reworked for
additional safety (checking MAXPATHLEN and NAME_MAX better), and a different
renaming scheme (it simply appends a number starting from 1 to 9999 to the
filename, making sure to respect any file extension). There were a few other
changes and bug fixes made too. This applies to rsync 3.1.2, including the new
debug and info message handling.

It's a pretty specific use case to need this patch, but we find it useful.
Posting here in case others have use for this also.

Thanks,
Devin

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

--- Comment #6 from Devin Nate <devin.nate at cloudwerx.com> ---
Created attachment 12106
  --> https://bugzilla.samba.org/attachment.cgi?id=12106&action=edit
minor cleanups

minor cleanups

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

Devin Nate <devin.nate at cloudwerx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #12106|minor cleanups              |rsync-3.1.2-rename-existing
        description|                            |V7.patch

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8367

Claudius Ellsel <claudius.ellsel at live.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |claudius.ellsel at live.de

--- Comment #7 from Claudius Ellsel <claudius.ellsel at live.de> ---
I didn't have a very close look. Just for understanding, what is the
difference
to `-b` option? That the files remain in the folder and are not moved to the
backup directory?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.