rsync - Jun 2008 - DO NOT REPLY [Bug 5567] New: REGRESSION: 3.0.3pre3 dies with "failed to open XXX, continuing: Too many open files"

https://bugzilla.samba.org/show_bug.cgi?id=5567

           Summary: REGRESSION: 3.0.3pre3 dies with "failed to open XXX,
                    continuing: Too many open files"
           Product: rsync
           Version: 3.0.3
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: timo.lindfors@iki.fi
         QAContact: rsync-qa@samba.org


I started getting a lot of errors like

rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fs/posix/acl.h",
continuing: Too many open files (24)
rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fujitsu/laptop.h",
continuing: Too many open files (24)
rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fuse/fs.h",
continuing: Too many open files (24)
rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/ctl.h",
continuing: Too many open files (24)
rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/fc.h",
continuing: Too many open files (24)
rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/lan.h",
continuing: Too many open files (24)
rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/logging.h",
continuing: Too many open files (24)
rsync: failed to open
"/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/sas.h",
continuing: Too many open files (24)

after I upgraded from rsync 3.0.3pre2 to 3.0.0pre3. I invoke rsync on a debian
stable system on x86 with the following arguments/environment:

execve("/usr/bin/rsync", ["rsync",
"--numeric-ids", "--delete",
"--delete-after", "-ax", "root@moniscreen:/",
"/tftpboot/moniscreen"],
["SHELL=/bin/bash", "TERM
=screen", "USER=root",
"http_proxy=http://proxy.kurp.hut.fi:3128/",
"SUDO_USER=lindi", "SUDO_UID=548",
"ftp_proxy=http://proxy.kurp.hut.fi:3128/",
"PATH=/usr/local/sb
in:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin"...,
"PWD=/root", "EDITOR=/home/lindi/bin/editor",
"SHLVL=1",
"SUDO_COMMAND=/bin/bash backup2 --stdin",
 "HOME=/home/lindi", "LESSCHARSET=latin1",
"LOGNAME=root", "LC_CTYPE=fi_FI",
"SUDO_GID=500", "_=/usr/bin/strace"])

and with the following ulimits:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 1004140
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) 1004140
max memory size         (kbytes, -m) 1004140
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) 0
stack size              (kbytes, -s) 4096
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) 1004140
file locks                      (-x) unlimited

I could raise ulimit of open files but shouldn't rsync cope just fine with
the
default limit?

The diff between pre2 and pre3 is quite small but I was unable to figure out
anything obvious. What extra info should I provide? The whole strace output is
96 megabytes and contains sensitive data. I was unable to reproduce the issue
with a simple

mkdir a
for i in `seq 1 4096`; do echo > a/$i; done
rsync --numeric-ids --delete --delete-after -ax a/ b


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5567





------- Comment #1 from timo.lindfors@iki.fi  2008-06-26 06:22 CST -------
Steps to reproduce:
1) cd /tmp
2) wget http://iki.fi/lindi/rsync/bug5567.tar.gz
3) sudo tar xzf bug5567.tar.gz
4) sudo rsync --numeric-ids --delete --delete-after -ax
localhost:/tmp/bug5567/source/ /tmp/bug5567/target

Expected results:
4) rsync does not give error messages

Actual results:
4) rsync complains loudly with

...
rsync: failed to open "/tmp/bug5567/target/scripts/mod/elfconfig.h",
continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/empty.o",
continuing:
Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/file2alias.o",
continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/mk_elfconfig",
continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/modpost",
continuing:
Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/modpost.o",
continuing:
Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/sumversion.o",
continuing: Too many open files (24)
rsync: opendir "/tmp/bug5567/target/." failed: Too many open files
(24)
IO error encountered -- skipping file deletion
rsync error: some files could not be transferred (code 23) at main.c(1506)
[generator=3.0.3pre3]

Please let me know if you are unable to reproduce this I am happy to provide
more info.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5567


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED




------- Comment #2 from wayned@samba.org  2008-06-26 09:55 CST -------
I can run rsync on the tar you specified with "ulimit -n 16" without
problem.
Your tar file didn't contain any xattrs, though -- are there xattrs
involved?
Are any patches applied to the default source?  Is selinux involved?  Can you
run "lsof -c rsync" while rsync is running to see if it has a lot of
open
files?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5567





------- Comment #3 from timo.lindfors@iki.fi  2008-06-26 10:14 CST -------
xattrs are probably not involved since I too can reproduce the problem by just
untarring bug5567.tar.gz (which does not have them) to /tmp (which should be
just regular ext3 without any special setup). No special patches have been
applied, this was compiled from

8b1eb7e42b36744cf580a2fba6224087  rsync-3.0.3pre3.tar.gz

I also don't think there is any selinux setup on the debian stable system 
(at
least I have never configured any). Finally, I started

sudo sh -c 'while true; do lsof |grep rsync|wc; done'

before running rsync and it printed

   17     153    1533
   1065    9591  126779
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
    178    1626   20262
      0       0       0

so rsync quite rapidly opened all the files it could and then started to give
out errors.

If you still can't reproduce I could prepare a clean qemu image where this
can
be reproduced, would that help?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5567





------- Comment #4 from wayned@samba.org  2008-06-26 10:14 CST -------
Created an attachment (id=3372)
 --> (https://bugzilla.samba.org/attachment.cgi?id=3372&action=view)
Close fd in the generator for empty files

I looked at the code after my reply, and found the problem.  If I add
--no-whole-file to the command options, I too get the errors, and the attached
patch fixes this.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5567


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED




------- Comment #5 from wayned@samba.org  2008-06-26 10:15 CST -------
This is now fixed in the git repository.

Thanks for your report!


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.