rsync - Mar 2014 - [Bug 10527] New: Rsync Deadlock when copying files

https://bugzilla.samba.org/show_bug.cgi?id=10527

           Summary: Rsync Deadlock when copying files
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: dah4k0r at gmail.com
         QAContact: rsync-qa at samba.org


I'm getting a deadlock when I'm rsyncing between two local drives.

I've attached my strace output.

The rsync command is

rsync -aAXvvvvi / /mnt/backup
--exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/var/lib/pacman/sync/*,/home/colin/data/*}
--delete

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #1 from Colin Rice <dah4k0r at gmail.com> 2014-03-30 09:48:36
UTC ---
Created attachment 9820
  --> https://bugzilla.samba.org/attachment.cgi?id=9820
rsync strace output

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #2 from Colin Rice <dah4k0r at gmail.com> 2014-04-03 03:40:24
UTC ---
Also happens with rsync 3.1.1pre1

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #3 from roland <devzero at web.de> 2014-04-08 21:34:43 UTC ---
does it always deadlock at the same file/position ?

please check with losf |grep rsync, to see at which file rsync got stuck...

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #4 from roland <devzero at web.de> 2014-04-08 21:42:19 UTC ---
pardon, typo - it should read   lsof|grep rsync

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #5 from Colin Rice <dah4k0r at gmail.com> 2014-04-09 02:53:35
UTC ---
lsof | grep rsync is showing no open files

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #6 from Colin Rice <dah4k0r at gmail.com> 2014-04-10 23:48:08
UTC ---
(In reply to comment #5)
> lsof | grep rsync is showing no open files
I was running lsof as a local user not root.

lsof | grep rsync is showing it stuck on the same file after repeated runs.
When that file is deleted it ends up getting stuck on a different file. It is
the file that the last match_sums was called on.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #7 from roland <devzero at web.de> 2014-04-12 10:28:30 UTC ---
can you tell if there is some special type of mount involved?

what type of filesystem on src / destination ?

large files ?

for how long did you wait for finish ? (
https://bugzilla.samba.org/show_bug.cgi?id=8315 )

i would try rsync via localhost, i.e. make rsync use the tcpip-stack and
perhaps also add bwlimit option, just to see if that makes a difference.

maybe we can see if this is an rsync issue or filesystem/disk issue.

i would also try another target path just to see how it behaves

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #8 from Colin Rice <dah4k0r at gmail.com> 2014-04-12 16:51:44
UTC ---
(In reply to comment #7)
> can you tell if there is some special type of mount involved?
It is copying from an ecrypted partition mounted with dm-crypt to another
encrypted partition mounted with dm-crypt
> 
> what type of filesystem on src / destination ?
Both are btrfs
> large files ?
Some files are in the 100m range. I'm doing a fully filesystem backup off of
a
linux machine.
> 
> for how long did you wait for finish ? (
> https://bugzilla.samba.org/show_bug.cgi?id=8315 )
I've waited overnight.
> 
> i would try rsync via localhost, i.e. make rsync use the tcpip-stack and
> perhaps also add bwlimit option, just to see if that makes a difference.
Will try
> 
> maybe we can see if this is an rsync issue or filesystem/disk issue.
> 
> i would also try another target path just to see how it behaves
Like try and backup a portion of the file system?

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #9 from roland <devzero at web.de> 2014-04-12 21:26:37 UTC
---
>Like try and backup a portion of the file system?
no, retry with one of the filesystems being standard ext3/4 to see if source or
target fs is the culprit.

btrfs is still considered experimental, so especially with rsync and dm-crypt i
would not wonder if you hit a bug....or some "interference".

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #10 from Colin Rice <dah4k0r at gmail.com> 2014-04-13 01:59:27
UTC ---
It appears that the problem is writing to a partition mounted on top of
dm-crypt. I tried btrfs and ext4 on top of dm-crypt and they both deadlocked.

Ext4 on top of a raw partition was fine.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #11 from Colin Rice <dah4k0r at gmail.com> 2014-04-13 05:47:05
UTC ---
I tried with btrfs on top of a raw partition as the destination and it
deadlocked.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #12 from Colin Rice <dah4k0r at gmail.com> 2014-04-13 08:07:19
UTC ---
My apologies, I wrote to the wrong mount. btrfs on a raw partition works fine.
So it appears the problem is related to dm-crypt.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #13 from roland <devzero at web.de> 2014-04-13 08:20:50 UTC
---
can you reproduce the hang without using rsync, e.g. by doing an ordinary cp ?

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #14 from Colin Rice <dah4k0r at gmail.com> 2014-04-13 08:43:26
UTC ---
If I use an ordinary cp -rx everything seems to work.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #15 from roland <devzero at web.de> 2014-04-13 09:01:04 UTC
---
i have no idea on how to proceed further.

if you use an older distro, try reproducing that on a recent one, with recent
kernel version.

maybe the dm-crypt people have an idea what to do to find the root cause:

dm-crypt at saout.de
http://www.saout.de/mailman/listinfo/dm-crypt
http://news.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #16 from Colin Rice <dah4k0r at gmail.com> 2014-04-13 09:02:32
UTC ---
Hmm, I'm pretty sure the problem appeared sometime in 3.13 since before then
rsync was working fine. I'll contact the dm-crypt people and ask for help.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #17 from roland <devzero at web.de> 2014-04-13 09:24:09 UTC
---
apparently, there seems to exist a deadlock patch for dmcrypt for about 5
months, but it seems it did not enter mainline kernel yet, so probably your
distro is also missing it:

https://github.com/pld-linux/kernel/blob/master/dm-crypt-fix-allocation-deadlock.patch

maybe this one is related and issue which this patch is fixing is not simply
theoretical one, but seen in the wild now :)

i give Mikulas Patocka a pointer to this report, maybe he can confirm that this
patch is related.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX

--- Comment #18 from Wayne Davison <wayned at samba.org> 2014-04-13
16:22:17 UTC ---
Having too much verbosity going is an easy way to cause rsync to hang.  If you
need it, try using --msgs2stderr so that the protocol doesn't have to deal
with
all that verbosity.

The strace looks like everyone is everyone is trying to write to their
pipe/socket file handle at the same time with nobody reading, so the above
should get you unstuck.  While it would be good to try to fix such a
high-verbosity deadlock, it is not something that is easy to do (since there
are times that a process must write before doing more reading, and the huge
quantity of messages clog things up).

If there is some other hang you are experiencing (without the high verbosity),
feel free to attach a strace of that run to this bug and re-open.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #19 from Colin Rice <dah4k0r at gmail.com> 2014-04-13 20:13:55
UTC ---
The extra verbosity was added as an attempt to debug the deadlocks. However as
it is not longer hanging without the extra verbosity, it works for me.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #20 from roland <devzero at web.de> 2014-04-15 21:37:37 UTC
---
what do you mean with "no longer hanging"? you mean you do not use
dm-crypt for
the target partition anymore and so the problem is solved for you?

it would be interesting to find out why the deadlock happens with dm-crypt,
though.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #21 from roland <devzero at web.de> 2014-04-16 18:54:51 UTC
---
> You should press alt-sysrq-w when the deadlock happen to see if there are 
> any processes deadlocked in the kernel. If yes, send me the stacktrace of 
> those processes.
> 
> If there are not any processes deadlocked in the kernel, then it may be 
> userspace problem - bug in rsync or something like that.
> 
> Mikulas
http://en.wikipedia.org/wiki/Magic_SysRq_key

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #22 from Colin Rice <dah4k0r at gmail.com> 2014-04-16 19:19:09
UTC ---
I removed the verbosity and it stopped hanging. If I drop back down to -vvvvi
again, it does hang.

I still use dm-crypt for the partition.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #23 from Colin Rice <dah4k0r at gmail.com> 2014-04-16 19:24:39
UTC ---
Created attachment 9854
  --> https://bugzilla.samba.org/attachment.cgi?id=9854
sysrq blocked tasks

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #24 from Colin Rice <dah4k0r at gmail.com> 2014-04-16 20:31:10
UTC ---
Originally it was hanging even without the vvvvi, but since then I've wiped
the
backup drive completely, and upgraded my kernel from 3.13.6 to 3.14 and it is
no longer hanging without the extra verbosity. I'm happy to keep providing
traces, but I'm not confident that I'm reproducing the same failure as
it no
longer hangs without the vvvvi.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10527

--- Comment #25 from roland <devzero at web.de> 2014-04-16 20:56:50 UTC
---
so, if kernel update fixed it we should consider this as being resolved

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.