Hello, I was reading your posts about RSYNC. We have a massive Oracle schema lots of datafiles about 750 GB size. We do rsync datafiles from source to target server but everytime we cleanup the datafiles on the target server and do rsync every 2 weeks. On the target side mostly the datafiles will be same but on source we might have added few datafiles or made some changes in data and as such the size of schema will increase every time since we add new datafiles. If we leave the old datafiles at the target end and rsync will it be faster ? What rysnc command we need to use without cleaning the old files so in the old files only the changes will be copied and new files also needs to be rsynced. What is the rsync command to be used ? Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-help-needed...-t1170765.html#a3075671 Sent from the Samba - rsync forum at Nabble.com.
Hi Isk,
Please see embedded.
Gian
lsk wrote:
>Hello,
>
>I was reading your posts about RSYNC. We have a massive Oracle schema lots
>of datafiles about 750 GB size. We do rsync datafiles from source to target
>server but everytime we cleanup the datafiles on the target server and do
>rsync every 2 weeks.
>
What do you mean by "cleanup the datafiles on the target server"? Are
you editing files on the target server?
> On the target side mostly the datafiles will be same
>but on source we might have added few datafiles or made some changes in data
>and as such the size of schema will increase every time since we add new
>datafiles.
>
>If we leave the old datafiles at the target end and rsync will it be faster
>?
>
If the target is not 100% different from the source, yes, keep it!
Rsync will upload the differences of the two. If 99% of the file is the
same, you just saved 99% of your transfer time. :)
See timestamps and checksum in the manual to see what option you need to
use in your schema. Checksum may take some time with 3/4 TB of data :-/
>What rysnc command we need to use without cleaning the old files so in the
>old files only the changes will be copied and new files also needs to be
>rsynced.
>
>
So you want to update files and transfer new files? How about:
rsync -e ssh -avz /source/data/files isk@target:/data/files
This command works over ssh (for encryption), uses an archive mode (see
manual for details), spits out information verbosely, and compresses
files during transfer to speed it up. If you would like to watch it
happen, add --progress for cool status view.
>What is the rsync command to be used ?
>
>Thanks,
>lsk.
>--
>View this message in context:
http://www.nabble.com/Rsync-help-needed...-t1170765.html#a3075671
>Sent from the Samba - rsync forum at Nabble.com.
>
>
>
Is this what you were looking for?
Gian
Gian, "What do you mean by "cleanup the datafiles on the target server"? Are you editing files on the target server? " lsk: That means I delete/rm the files from the target and then start rsync. "If the target is not 100% different from the source, yes, keep it! Rsync will upload the differences of the two. If 99% of the file is the same, you just saved 99% of your transfer time. :) See timestamps and checksum in the manual to see what option you need to use in your schema. Checksum may take some time with 3/4 TB of data :-/" lsk: This is oracle database the header information(timestamp..etc) on each datafile constantly changes which might be very small change but the data inside most of the datafiles are same they wont change much. New oracle datafiles will be added on the source which needs to be transferred. So what we do is every 2 weeks to refresh target server we remove all datafiles on target and rsync all datafiles again from source. My question is if we leave the older datafiles and rsync will it be faster ? Currently I use "rsync -czv" c for checksum. Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-help-needed...-t1170765.html#a3077044 Sent from the Samba - rsync forum at Nabble.com.
On Mon, 2006-02-27 at 06:58 -0800, lsk wrote:> Could you give an example with syntax for rsync using file > option "--files-rom=FILE".If my-list in the current directory contains a b b/c b/d b/d/e , then the command rsync -a src/ dest/ --files-from=my-list copies src/a -> dest/a src/b -> dest/b src/b/c -> dest/b/c src/b/d -> dest/b/d src/b/d/e -> dest/b/d/e . Rsync copies just those files and directories, preserving their attributes. It leaves alone any other files that might be in src/b, src/b/d, dest/b, or dest/b/d; even if --delete is given, it does not delete anything. Search the man page for --files-from for more information: http://rsync.samba.org/ftp/rsync/nightly/rsync.html -- Matt McCutchen hashproduct@verizon.net http://hashproduct.metaesthetics.net/
Matt McCutchen wrote:> On Mon, 2006-02-27 at 06:58 -0800, lsk wrote: >> Could you give an example with syntax for rsync using file >> option "--files-rom=FILE". > > If my-list in the current directory contains > a > b > b/c > b/d > b/d/e , > then the command > rsync -a src/ dest/ --files-from=my-list > copies > src/a -> dest/a > src/b -> dest/b > src/b/c -> dest/b/c > src/b/d -> dest/b/d > src/b/d/e -> dest/b/d/e . > Rsync copies just those files and directories, preserving their > attributes. It leaves alone any other files that might be in src/b, > src/b/d, dest/b, or dest/b/d; even if --delete is given, it does not > delete anything. Search the man page for --files-from for more > information: > http://rsync.samba.org/ftp/rsync/nightly/rsync.htmlMatt, Typically with Oracle databases, you will have datafiles residing on different mount points starting from the root such as: /p02/oradata/OSID/redo01.log /p03/oradata/OSID/redo02.log /p04/oradata/OSID/redo03.log /p01/oradata/OSID/system01.dbf /p04/oradata/OSID/undotbs01.dbf /p03/oradata/OSID/sysaux01.dbf /p03/oradata/OSID/users01.dbf /p03/oradata/OSID/example01.dbf /p03/oradata/OSID/data01.dbf /p02/oradata/OSID/index01.dbf /p01/oradata/OSID/control01.ctl /p02/oradata/OSID/control02.ctl /p03/oradata/OSID/control03.ctl /s01/oracle/product/10.2.0/dbs/initOSID.ora Requiring a command line something like this: rsync -a srchost:/ / --files-from=dbf-list and dbf-list would contain this: p02/oradata/OSID/redo01.log p03/oradata/OSID/redo02.log p04/oradata/OSID/redo03.log p01/oradata/OSID/system01.dbf p04/oradata/OSID/undotbs01.dbf p03/oradata/OSID/sysaux01.dbf p03/oradata/OSID/users01.dbf p03/oradata/OSID/example01.dbf p03/oradata/OSID/data01.dbf p02/oradata/OSID/index01.dbf p01/oradata/OSID/control01.ctl p02/oradata/OSID/control02.ctl p03/oradata/OSID/control03.ctl s01/oracle/product/10.2.0/dbs/initOSID.ora Do you see any problems with this?
On Tue, 2006-02-28 at 09:20 -0500, Linus Hicks wrote:> rsync -a srchost:/ / --files-from=dbf-list > > and dbf-list would contain this: > > p02/oradata/OSID/redo01.log > p03/oradata/OSID/redo02.log > p04/oradata/OSID/redo03.log > p01/oradata/OSID/system01.dbf > p04/oradata/OSID/undotbs01.dbf > p03/oradata/OSID/sysaux01.dbf > p03/oradata/OSID/users01.dbf > p03/oradata/OSID/example01.dbf > p03/oradata/OSID/data01.dbf > p02/oradata/OSID/index01.dbf > p01/oradata/OSID/control01.ctl > p02/oradata/OSID/control02.ctl > p03/oradata/OSID/control03.ctl > s01/oracle/product/10.2.0/dbs/initOSID.ora > > Do you see any problems with this?That should work fine; that is exactly how --files-from is typically used. Note that rsync looks for dbf-list on the computer on which you are starting rsync; if dbf-list is on the other computer, use --files-from=:dbf-list to indicate this. -- Matt McCutchen hashproduct@verizon.net http://hashproduct.metaesthetics.net/