rsync - Jun 2009 - Possible Bug or limitation in Cygwin 1.7 and Rsync and file number limit

I'm not sure if this is a bug or a limitation that can be worked around with
a setting somewhere, but I have found a problem with cygwin 1.7 while using
rsync.  I have been using rsync and cygwin v1.5 for quite some time.  I recently
started testing cygwin v1.7 and I ran into a problem with an apparent limit on
the number of files cygwin 1.7 along with rsync can process in a directory. 
First let me say that I don't store this many files in a directory manually
but rather we have a program that does.

I have not been able to identify the exact number of files, but it seems to be
somewhere over 6,000 in a single directory.  To help sort this out, I whipped up
the following perl code that creates 9,000 dummy files and puts them in a
c:\temp directory.  Code is as follows if anyone would like to try and reproduce
this:

my $path="c:/temp/";
my $number_of_files=9000;
my $base_file_name="file";
my $count=1;
my $file_name = $path . $base_file_name . $count;
while ($count <= $number_of_files) {
  open FILE, ">$file_name" or die $!;
  $count++;
  $file_name = $path . $base_file_name . $count;
  print FILE "1";
  close(FILE);
}

For those familiar with rsync, here are the options I am using in rsync and by
the way, I am currently reproducing this by copying from and to the same
workstation, however this same issue has been reproduced on more than one
machine.  This doesn't appear to be an rsync problem but just the same, I
have tried several different options for rsync and none seem to make a
difference.

--recursive
--human-readable
--delete
--times
--ignore-errors
--stats
--progress
-v -v

If I then run rsync 3.0.5 using cygwin 1.7, the processor will spike to 100%
utilization and it will process several thousand files before it just locks up
with the processor still spiked at 100%.  No errors are shown, it just hangs. 
If I then delete the directory I am using rsync to copy to and repeat the same
procedure with the same version of rsync only this time using cygwin 1.5.25-15,
the processor will again spike to 100% however the copy will finish as expected
and everything returns to normal.  Again, I have been able to repeat this on
more than one computer.

One more interesting thing.  If I first using rsync and cygwin v1.5 to copy the
files and THEN come back and use rsync and cygwin 1.7, it make it all the way
through.

Is this a bug in cygwin 1.7 or is there some setting that will allow me to work
around this limitation?

Thank you.


- Kyle

I'm not sure if this is a bug or a limitation that can be worked around with
a setting somewhere, but I have found a problem with cygwin 1.7 while using
rsync.  I have been using rsync and cygwin v1.5 for quite some time.  I recently
started testing cygwin v1.7 and I ran into a problem with an apparent limit on
the number of files cygwin 1.7 along with rsync can process in a directory. 
First let me say that I don't store this many files in a directory manually
but rather we have a program that does.

I have not been able to identify the exact number of files, but it seems to be
somewhere over 6,000 in a single directory.  To help sort this out, I whipped up
the following perl code that creates 9,000 dummy files and puts them in a
c:\temp directory.  Code is as follows if anyone would like to try and reproduce
this:

my $path="c:/temp/";
my $number_of_files=9000;
my $base_file_name="file";
my $count=1;
my $file_name = $path . $base_file_name . $count;
while ($count <= $number_of_files) {
  open FILE, ">$file_name" or die $!;
  $count++;
  $file_name = $path . $base_file_name . $count;
  print FILE "asdfasdf";
  close(FILE);
}

For those familiar with rsync, here are the options I am using in rsync and by
the way, I am currently reproducing this by copying from and to the same
workstation, however this same issue has been reproduced on more than one
machine.  This doesn't appear to be an rsync problem but just the same, I
have tried several different options for rsync and none seem to make a
difference.

--recursive
--human-readable
--delete
--times
--ignore-errors
--stats
--progress
-v -v

If I then run rsync 3.0.5 using cygwin 1.7, the processor will spike to 100%
utilization and it will process several thousand files before it just locks up
with the processor still spiked at 100%.  No errors are shown, it just hangs. 
If I then delete the directory I am using rsync to copy to and repeat the same
procedure with the same version of rsync only this time using cygwin 1.5.25-15,
the processor will again spike to 100% however the copy will finish as expected
and everything returns to normal.  Again, I have been able to repeat this on
more than one computer.

One more interesting thing.  If I first using rsync and cygwin v1.5 to copy the
files and THEN come back and use rsync and cygwin 1.7, it make it all the way
through.

Is this a bug in cygwin 1.7 or is there some setting that will allow me to work
around this limitation?

Thank you.


- Kyle

Skipped content of type multipart/alternative-------------- next part
--------------
A non-text attachment was scrubbed...
Name: cygcheck.out
Type: application/octet-stream
Size: 13163 bytes
Desc: not available
Url :
http://lists.samba.org/archive/rsync/attachments/20090618/45dd5324/cygcheck.obj

Skipped content of type multipart/alternative-------------- next part
--------------
A non-text attachment was scrubbed...
Name: cygcheck.out
Type: application/octet-stream
Size: 13163 bytes
Desc: not available
Url :
http://lists.samba.org/archive/rsync/attachments/20090618/87326f0a/cygcheck.obj