rsync - Nov 2017 - Rsync to just upload new/updated files without comparing source/destination

Hi,

I have huge data on my destination system (>5 TB, > 400,000 files, nested
folderss) and i am trying to publish new files from our shared location to
destination system. I am looking forward to use rsync but want it to just
push new/updated files uploaded on shared server to destination. I don't
intent to keep all data to remain in sync on source server and want to
delete the source file as soon as it's uploaded to destination.

Essentially i want to keep same folder structure on source and destination
(with ability for users to add new folders) but don't want to keep files on
source. As per my understanding rsync compares files on source and
destination and then performs the sync. I am worried that this comparison
can take large amount of time due to data size.

Is there a way by which i can just upload new/updated files and new folders
to respective location in destination using rsync without doing any
comparisons? I read about --ignore-existing flag but not sure if
checksum/comparison between source and destination is still performed and
will we run into performance issues.

Is it feasible with rsync in an efficient way?

Thanks



--
Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html

If you remove the source files as rsync copies them
(--remove-source-files) then rsync will have nothing to compare.  It
will just go through the tree looking for files that aren't on the
target which would probably be any file it finds.  Just don't ever use
--delete in that scenario.  Adding --omit-dir-times will save you a lot
of output if you have any of --itemize-changes --verbose --progress.

When in doubt, do a --dry-run

On 11/19/2017 12:48 PM, Amit via rsync wrote:
> Hi,
> 
> I have huge data on my destination system (>5 TB, > 400,000 files,
nested
> folderss) and i am trying to publish new files from our shared location to
> destination system. I am looking forward to use rsync but want it to just
> push new/updated files uploaded on shared server to destination. I
don't
> intent to keep all data to remain in sync on source server and want to
> delete the source file as soon as it's uploaded to destination.
> 
> Essentially i want to keep same folder structure on source and destination
> (with ability for users to add new folders) but don't want to keep
files on
> source. As per my understanding rsync compares files on source and
> destination and then performs the sync. I am worried that this comparison
> can take large amount of time due to data size.
> 
> Is there a way by which i can just upload new/updated files and new folders
> to respective location in destination using rsync without doing any
> comparisons? I read about --ignore-existing flag but not sure if
> checksum/comparison between source and destination is still performed and
> will we run into performance issues.
> 
> Is it feasible with rsync in an efficient way?
> 
> Thanks
> 
> 
> 
> --
> Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 224 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20171119/b7d767d2/signature.sig>

Thanks for info. I was under the assumption that rsync will execute
followings steps:
1. Compare first to identify different files (new/updated) --> This is where
i am worried due to data size
2. Upload new/updated files
3 Remove source file

As per your comment, rsync will compare after uploading to check if files
are identical ?

Additionally i don't expect to delete directories as i want to maintain same
directory structure on source. Any insights on if following query will work
effeciently for adding new files to destination
rsync --remove-source-files -av -r <Source Root Folder> ssh
abc at destination.com:<Destination Root Folder>

Assuming 5 TB of data available at destination and source has about 50 files
(New or Updated i.e. files with same name and in same folder), source file
will be removed as soon as it's uploaded.

Thanks



--
Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html