rsync - Apr 2002 - Can rsync update files in place?

I've just subscribed, but a search of the archive doesn't indicate this
has been handled before...

Is there a way to get rsync to not create a new file while transferring
and then rename it, but to instead update the existing file in place,
i.e. simply write those blocks that have been updated and leave the rest
alone?

That would be ideal for what I wanted rsync for, namely updating copies
of Oracle data files. These reside on a volume of a NetApp Filer where
other stuff is also located, and snapshots are made regularly of that
volume. Simply deleting the old files and copying them again means that
the Filer considers each and every block of the old files "old", and
that mens they get preserved in the snapshots, which in turn means that
the snapshots take up WAY too much space (the snapshots are a sort of
"copy-on-write" mechanism: as long as only a few blocks are updated,
you
only need a few blocks to maintain a snapshot).

I thought that I could get around this by using rsync, because of the
way it only transfers those blocks that have changed. I now know a bit
more about how it works :-)  In no way does rsync increase disk IO
efficiency, it increases network IO efficiency... That of course is
fine, but unfortunatley not quite suited to the purpose I had in mind.

How difficult would it be to add an option so that it works in the way I
would want?  I understand that it would mean losing the "moved data"
optimization, but for Oracle data files where relatively little is
updated (and probably not moved at all) that wouldn't matter a whole
lot.  (The set of files I'm talking about is 40GB in about 20 files,
with 4 files of 8GB.)


In the meantime I'll probably hack up something that compares two files
and then updates those blocks in the second file that differ... won't
work over a network though :-)


Thanks,
Paul Slootman
--

On 16 Apr 2002, Paul Slootman <paul+rsync@wurtel.net> wrote:
> Is there a way to get rsync to not create a new file while transferring
> and then rename it, but to instead update the existing file in place,
> i.e. simply write those blocks that have been updated and leave the rest
> alone?
That would be pretty hard, though not impossible.

I suspect it would be roughly as hard to put it into rsync as to 
write a new program from scratch, depending on what features you need.

NetApp machines are kind of a special case because you can't (?)
run an rsync program directly on the server.  So every block 
is going across the network in at least one direction.

Perhaps Oracle's native replication features would be better?

Happy hacking,
-- 
Martin

Martin replied in an earlier thread:
> On 16 Apr 2002, Paul Slootman <paul+rsync@wurtel.net> wrote:
> > Is there a way to get rsync to not create a new file while
transferring
> > and then rename it, but to instead update the existing file in place,
> > i.e. simply write those blocks that have been updated and leave the
rest
> > alone?
> 
> That would be pretty hard, though not impossible.
> 
> I suspect it would be roughly as hard to put it into rsync as to 
> write a new program from scratch, depending on what features you need.
> 
> NetApp machines are kind of a special case because you can't (?)
> run an rsync program directly on the server.  So every block 
> is going across the network in at least one direction.
> 
> Perhaps Oracle's native replication features would be better?
> 
> Happy hacking,
> -- 
> Martin
NetApp snapshots are great for storing file deltas over time.  With
the introduction of their R100, they're enabling better practices for
doing disk to disk backup and recovery.  The only problem with the
current poduct and marketing is that the backup solution doesn't
address the storage waste of copying entire files instead of just
using incremental copies of the changed records.  Without incremental
record changes, multiple whole copies of the file would be stored in
snapshots instead of just the incremental changes.  (What a waste!)

The last time I looked, the block size on a NetApp WAFL filesystem was
4096 bytes. If you used the following algorithm, you could copy just
the changes:

  BEGIN
  {
     open source_file read_only
     open destination_file read_write
     for each block_position from 0 to int(source file size / 4096)
     {
        src_block = read source_file, 4096 bytes at block_position
        dst_block = read destination_file, 4096 bytes at block_position
        if src_block and dst_block are different
        {
           write src_block to destination_file at block_position
        }
     }
     if file is not block-aligned
     {
        src_block = read source_file, remaining bytes at block_position
        dst_block = read destination_file, remaining bytes at block_position
        if src_block and dst_block are different
        {
           write src_block to destination_file at block_position
        }
     }
     close source_file
     close destination_file
  }
  END

I've just written a Perl program that does an in-place incremental
block-level copy between files on a Unix system.

Take a look here:

   http://www.vix.com/~ziegast/src/blkcopy

Note: To protect myself, the program won't run properly until you
      modify the script to disable the annoying disclaimer statement.

If someone would like to try it out on a really large database
file (eg: MySQL, Oracle) and show the filesystem size differences
on a NetApp between using blkcopy versus other programs, let me
know.

If you work at Netapp or a backup software company, drop me a line
for more info.  You can modify your programs to support incremental
block-level backups just like the blkcopy script does.

--
Eric Ziegast