I've been running some tests on files created by rsync and noticing fragmentation issues. I started the testing because our 5TB array started performing very slowly and it appears fragmentation was the culprit. The test I conducted was straighforward: 1. Copy over a 49GB file. Analyzed with contig (from sysinternals), no fragments. 2. Ran rsync and the file was recreated normally (rsync options were -ruvi --times --no-blocking-io --compress-level=9 --bwlimit=0 --stats --delete --modify-window=2) 3. Resulting file had almost 12000 fragments. It doesn't really surprise me since rsync grabs blocks as it needs them for the new. I was wondering why rsync doesn't preallocate the space like copy or other utilites do under Windows. This would insure you have the space to create the new file plus it would reduce fragmentation since rsync would be writing to a pre-allocated file space. I am running the cygwin version of rsync provided be Tev at www.itefix.no <http://www.itefix.no/> . His site is a huge help to those of us running rsync under Windows. -------------- next part -------------- HTML attachment scrubbed and removed
On 8/6/06, Rob Bosch <robbosch@msn.com> wrote:> I was wondering why rsync doesn't preallocate the space like copy > or other utilites do under Windows. This would insure you have the space to > create the new file plus it would reduce fragmentation since rsync would be > writing to a pre-allocated file space.I don't think there's a reason that rsync shouldn't preallocate the space. Since there is no way to preallocate in POSIX, I guess no one thought about preallocating in Windows. However, if there is a system call available through Cygwin, it would be easy to make rsync check for the call at configure time and use it if it is present. Matt
On 8/25/06, Rob Bosch <robbosch@msn.com> wrote:> This article suggests the ftruncate function is in the cygwin system > (http://www.cygwin.com/ml/cygwin-cvs/2006-q3/msg00114.html). Based on the > open base specifications the ftruncate should allow the file to be resized > to the desired level achieving what we want after the file is created?I took a look at the change to fhandler_disk_file.cc.diff linked from that message and found that posix_fallocate is the function we want. Cygwin correctly implements it using SetEndOfFile. So rsync's configure script should check for posix_fallocate, and the receiver should call it just after opening each received file. I might write a patch for this. Matt