rsync - Feb 2013 - Destination file a lot larger then source (real size)

I have a script that syncs my backups to an NFS mount every day

The script works fine, without any errors, but there is a problem when
it comes to some large files

Let's take my pst file (8.9 gig) as an example

Source:

du -hs mypst.pst
8.9G    mypst.pst
ls -alh mypst.pst
-rw-rw---- 1 me me 8.9G Jan 25 17:07 mypst.pst

That seems OK


Let's do that on the destination:

du -hs mypst.pst
17G     mypst.pst
ls -alh mypst.pst
-rw-rw---- 1 root root 8.9G Jan 25 17:07 mypst.pst  # Permissions here  are
fine, disregard


Real file size is almost double size!


Extra info:

Source dir is an xfs partition
The NFS mount is also xfs on the NFS server
NFSv4

Full cmdline for the daily backup:
/usr/bin/rsync -rltgoD --no-perms --no-owner --no-group --delete <src>
<dest>

For the testing purposes, I've tried doing:
rsync /srcdir/mypst.pst /nfsmount/mypst.pst
and the result is the same: du reports 17 gigs, and ls -alh reports 8.9

Is there any way around this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20130205/f4dcfc14/attachment.html>

On 05.02.2013 12:32, milos.kaurin at gmail.com wrote:
> I have a script that syncs my backups to an NFS mount every day
> 
> The script works fine, without any errors, but there is a problem when
> it comes to some large files
> 
> Let's take my pst file (8.9 gig) as an example
> 
> Source:
> 
> du -hs mypst.pst
> 8.9G    mypst.pst
> ls -alh mypst.pst
> -rw-rw---- 1 me me 8.9G Jan 25 17:07 mypst.pst
> 
> That seems OK
> 
> 
> Let's do that on the destination:
> 
> du -hs mypst.pst
> 17G     mypst.pst
> ls -alh mypst.pst
> -rw-rw---- 1 root root 8.9G Jan 25 17:07 mypst.pst  # Permissions here  are
> fine, disregard
> 
> 
> Real file size is almost double size!
> 
> 
> Extra info:
> 
> Source dir is an xfs partition
> The NFS mount is also xfs on the NFS server
> NFSv4
> 
> Full cmdline for the daily backup:
> /usr/bin/rsync -rltgoD --no-perms --no-owner --no-group --delete
<src>
> <dest>
> 
> For the testing purposes, I've tried doing:
> rsync /srcdir/mypst.pst /nfsmount/mypst.pst
> and the result is the same: du reports 17 gigs, and ls -alh reports 8.9
> 
> Is there any way around this?
This is a feature of XFS called "speculative preallocation". If you 
open/append/close a file repeatetly XFS preallocates space in 
anticipation of more appends, the feature helps to prevent or at least 
reduce fragmentation.

You can see the extents of a file with xfs_bmap.

You can get rid of the preallocation in several ways:
- Trash your buffer-cache with unreleated things
  (The prellocation gets dropped when the inode of the file is evicted 
from cache)
- umount/mount cycle
- Drop whole cache, as root:
echo 3 > /proc/sys/vm/drop_caches
This drops the whole non-dirty cache of Linux!

There was talk on the maillinglist to drop preallocations after some 
time (like 5 minutes), but AFAIR there weren't patches. So IFF that is 
implemented that means kernel 3.9 or later at the earliest.

If you don't want speculative preallocation you can disable it with the 
mount-option allocsize. e.g. "allocsize=4k"




-- 

Matthias