Amit
2017-Nov-19 19:41 UTC
Rsync to just upload new/updated files without comparing source/destination
Thanks for info. I was under the assumption that rsync will execute followings steps: 1. Compare first to identify different files (new/updated) --> This is where i am worried due to data size 2. Upload new/updated files 3 Remove source file As per your comment, rsync will compare after uploading to check if files are identical ? Additionally i don't expect to delete directories as i want to maintain same directory structure on source. Any insights on if following query will work effeciently for adding new files to destination rsync --remove-source-files -av -r <Source Root Folder> ssh abc at destination.com:<Destination Root Folder> Assuming 5 TB of data available at destination and source has about 50 files (New or Updated i.e. files with same name and in same folder), source file will be removed as soon as it's uploaded. Thanks -- Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html
Kevin Korb
2017-Nov-19 21:43 UTC
Rsync to just upload new/updated files without comparing source/destination
If the source only has 50 files then rsync only needs to check for those 50 files on the target. If one exists rsync will compare the timestamp to see if it needs copying. Yes, rsync verifies a hash of the file after transfer. On 11/19/2017 02:41 PM, Amit via rsync wrote:> Thanks for info. I was under the assumption that rsync will execute > followings steps: > 1. Compare first to identify different files (new/updated) --> This is where > i am worried due to data size > 2. Upload new/updated files > 3 Remove source file > > As per your comment, rsync will compare after uploading to check if files > are identical ? > > Additionally i don't expect to delete directories as i want to maintain same > directory structure on source. Any insights on if following query will work > effeciently for adding new files to destination > rsync --remove-source-files -av -r <Source Root Folder> ssh > abc at destination.com:<Destination Root Folder> > > Assuming 5 TB of data available at destination and source has about 50 files > (New or Updated i.e. files with same name and in same folder), source file > will be removed as soon as it's uploaded. > > Thanks > > > > -- > Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 224 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20171119/5a673f20/signature.sig>
Amit
2017-Nov-20 20:28 UTC
Rsync to just upload new/updated files without comparing source/destination
Thanks for your inputs. Surprisingly i tried executing the following command but end up deleting ssh.exe in Cygwin folder when source folder path is incorrect or not reachable. rsync --remove-source-files -v -av -r /cygdrive/c/AmitTest/ ssh username at destinationserver:/cygdrive/e/AMIT/AmitTest/ I am wondering if remove source is doing some tricks there but unable to find root cause why ssh.exe will get deleted with this command. It has happened twice since yesteday and today i noticed this happened when i executed this command. Any idea why this can be happening? Unfortunately i don't have way to copy ssh to Cygwin\bin folder as i keep getting 'Access Denied' error from windows explorer and cmd (tried taking ownership of Cygwin folder as well). Thanks, Amit -- Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html
Maybe Matching Threads
- Rsync to just upload new/updated files without comparing source/destination
- Rsync to just upload new/updated files without comparing source/destination
- Rsync to just upload new/updated files without comparing source/destination
- Rsync to just upload new/updated files without comparing source/destination
- Very slow to start sync with millions of directories and files