thr3ads.net - rsync - rsync got stuck [Aug 2017]

If this information is useful, please help other people find it:
Share via:

Vangelis Katsikaros

2017-Aug-23 10:14 UTC

rsync got stuck

Hi

I rsync from 4 machines to a single destination one, and I am certain the
filenames they sync do not overlap. But from time to time rsync from a source
machine gets stuck (not always the same source machine).

I followed https://rsync.samba.org/issues.html and I gathered strace, lsof and
netstat when the freeze occurred. I'd like to ask for any further debug info
or ideas!

General info
===========
Both destination and source run the same version 
$ rsync --version
rsync  version 3.1.0  protocol version 31
Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Both destination and source are Ubuntu 14.04

SOURCE MACHINE
=============
$ sudo ps -fe | grep rsync
abc       3794  3793  0 01:12 ?        00:00:00 /bin/sh -c flock -xn
/tmp/LOCKFILE -c "/usr/bin/rsync --compress --compress-level=9
--bwlimit=512k --recursive --delay-updates --quiet --update
--exclude=\"/.*\" /SRC_PATH/ DEST_____IP:/DEST_PATH/"
abc       3795  3794  0 01:12 ?        00:00:00 flock -xn /tmp/LOCKFILE -c
/usr/bin/rsync --compress --compress-level=9 --bwlimit=512k --recursive
--delay-updates --quiet --update --exclude="/.*" /SRC_PATH/
DEST_____IP:/DEST_PATH/
abc       3796  3795  0 01:12 ?        00:00:00 /bin/sh -c /usr/bin/rsync
--compress --compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet
--update --exclude="/.*" /SRC_PATH/ DEST_____IP:/DEST_PATH/
abc       3797  3796  0 01:12 ?        00:03:14 /usr/bin/rsync --compress
--compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet --update
--exclude=/.* /SRC_PATH/ DEST_____IP:/DEST_PATH/
abc       3798  3797  0 01:12 ?        00:00:13 ssh DEST_____IP rsync --server
-urze.iLs --compress-level=9 --bwlimit=512 --delay-updates . /DEST_PATH/
vangelis  8431  8371  0 18:04 pts/7    00:00:00 grep --color=auto rsync

$ sudo strace -p 3794
Process 3794 attached
wait4(-1, ^CProcess 3794 detached
 <detached ...>

$ sudo strace -p 3795
Process 3795 attached
wait4(3796, ^CProcess 3795 detached
 <detached ...>

$ sudo strace -yy -p 3796
Process 3796 attached
wait4(-1, ^CProcess 3796 detached
 <detached ...>

$ sudo strace -yy -p 3797
Process 3797 attached
select(6, [], [5<socket:[1465572750]>], [], {19, 795683}^CProcess 3797
detached
 <detached ...>

$ sudo strace -yy -p 3798
Process 3798 attached
select(7, [3<socket:[1465572769]>], [5<socket:[1465572752]>], NULL,
NULL^CProcess 3798 detached
 <detached ...>

$ sudo lsof -p 3797
COMMAND  PID USER   FD   TYPE             DEVICE   SIZE/OFF       NODE NAME
rsync   3797  abc  cwd    DIR         182,861281       4096     530855 /SRC_PATH
rsync   3797  abc  rtd    DIR         182,861281       4096          2 /
rsync   3797  abc  txt    REG         182,861281     390704     140858
/usr/bin/rsync
rsync   3797  abc  mem    REG         182,861281      18624         93
/lib/x86_64-linux-gnu/libattr.so.1.1.0
rsync   3797  abc  mem    REG         182,861281    1857312        542
/lib/x86_64-linux-gnu/libc-2.19.so
rsync   3797  abc  mem    REG         182,861281      48112        328
/lib/x86_64-linux-gnu/libpopt.so.0.0.0
rsync   3797  abc  mem    REG         182,861281     100728         30
/lib/x86_64-linux-gnu/libz.so.1.2.8
rsync   3797  abc  mem    REG         182,861281      31168         91
/lib/x86_64-linux-gnu/libacl.so.1.1.0
rsync   3797  abc  mem    REG         182,861281     149120        560
/lib/x86_64-linux-gnu/ld-2.19.so
rsync   3797  abc  mem    REG         182,861281      26258     154026
/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
rsync   3797  abc    0r  FIFO                0,8        0t0 1465572739 pipe
rsync   3797  abc    1u   REG         182,861281          0     394802 
(deleted)/tmp/tmpfWc3cI4
rsync   3797  abc    2u   REG         182,861281          0     394802 
(deleted)/tmp/tmpfWc3cI4
rsync   3797  abc    3u   REG         182,861281          0     397245
/tmp/LOCKFILE
rsync   3797  abc    4r   REG         182,861281 1085878503     786474 
(deleted)/SRC_PATH/SUBDIR/file.gz
rsync   3797  abc    5u  unix 0xffff881bcb29e100        0t0 1465572750 socket
rsync   3797  abc    6u  unix 0xffff880a31641100        0t0 1465572751 socket

$ sudo lsof -p 3798
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME
ssh     3798  abc  cwd    DIR         182,861281     4096    1312999 /home/abc
ssh     3798  abc  rtd    DIR         182,861281     4096          2 /
ssh     3798  abc  txt    REG         182,861281   641664     140798
/usr/bin/ssh
ssh     3798  abc  mem    REG         182,861281    43616        536
/lib/x86_64-linux-gnu/libnss_files-2.19.so
ssh     3798  abc  mem    REG         182,861281    47760        571
/lib/x86_64-linux-gnu/libnss_nis-2.19.so
ssh     3798  abc  mem    REG         182,861281    97296        523
/lib/x86_64-linux-gnu/libnsl-2.19.so
ssh     3798  abc  mem    REG         182,861281    39824        566
/lib/x86_64-linux-gnu/libnss_compat-2.19.so
ssh     3798  abc  mem    REG         182,861281   141574        544
/lib/x86_64-linux-gnu/libpthread-2.19.so
ssh     3798  abc  mem    REG         182,861281    14256        338
/lib/x86_64-linux-gnu/libkeyutils.so.1.4
ssh     3798  abc  mem    REG         182,861281    43672     136697
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
ssh     3798  abc  mem    REG         182,861281    14592        275
/lib/x86_64-linux-gnu/libcom_err.so.2.1
ssh     3798  abc  mem    REG         182,861281   186824     136690
/usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
ssh     3798  abc  mem    REG         182,861281   831616     136694
/usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
ssh     3798  abc  mem    REG         182,861281   252032        280
/lib/x86_64-linux-gnu/libpcre.so.3.13.1
ssh     3798  abc  mem    REG         182,861281  1857312        542
/lib/x86_64-linux-gnu/libc-2.19.so
ssh     3798  abc  mem    REG         182,861281   285848     136692
/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
ssh     3798  abc  mem    REG         182,861281   101240        518
/lib/x86_64-linux-gnu/libresolv-2.19.so
ssh     3798  abc  mem    REG         182,861281   100728         30
/lib/x86_64-linux-gnu/libz.so.1.2.8
ssh     3798  abc  mem    REG         182,861281    14664        550
/lib/x86_64-linux-gnu/libdl-2.19.so
ssh     3798  abc  mem    REG         182,861281  1926432         33
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
ssh     3798  abc  mem    REG         182,861281   134296         36
/lib/x86_64-linux-gnu/libselinux.so.1
ssh     3798  abc  mem    REG         182,861281   149120        560
/lib/x86_64-linux-gnu/ld-2.19.so
ssh     3798  abc    0u  unix 0xffff880a37b1b1c0      0t0 1465572749 socket
ssh     3798  abc    1u  unix 0xffff880a3d5e1780      0t0 1465572752 socket
ssh     3798  abc    2u   REG         182,861281        0     394802 
(deleted)/tmp/tmpfWc3cI4
ssh     3798  abc    3u  IPv4         1465572769      0t0        TCP
feeds3.abc.co.uk:55340->feeds-web1.abc.co.uk:ssh (ESTABLISHED)
ssh     3798  abc    5u  unix 0xffff880a3d5e1780      0t0 1465572752 socket
ssh     3798  abc    6u   REG         182,861281        0     394802 
(deleted)/tmp/tmpfWc3cI4

$ sudo netstat -anp | egrep "(3794|3795|3796|3797|3798)"
tcp        0      0 IP_________37:55340      DEST_____IP:22          ESTABLISHED
3798/ssh
tcp        0      0 IP_________37:5553       IP________132:53796     
ESTABLISHED 8282/perl
unix  3      [ ]         STREAM     CONNECTED     1465572752 3798/ssh
unix  3      [ ]         STREAM     CONNECTED     1465572751 3797/rsync
unix  3      [ ]         STREAM     CONNECTED     1465572750 3797/rsync
unix  3      [ ]         STREAM     CONNECTED     1465572749 3798/ssh



DESTINATION MACHINE
==================
$ sudo netstat -atlp | grep "IP_________14"
tcp        0      0 IP_________14:ssh       IP_________40:55340     ESTABLISHED
32097/sshd: abc [pr

$ sudo netstat -antp | egrep "(32097|32212)"
tcp        0      0 IP_________14:22        IP_________40:55340     ESTABLISHED
32097/sshd: abc [pr

$ sudo pstree --show-pids 32097
sshd(32097)───sshd(32212)

$ sudo strace -yy -p 32097
Process 32097 attached
restart_syscall(<... resuming interrupted call ...>^CProcess 32097
detached
 <detached ...>

$ sudo strace -yy -p 32212
Process 32212 attached
select(14, [3<socket:[6021256]> 6<pipe:[6021267]>], [], NULL,
NULL^CProcess 32212 detached
 <detached ...>

$ sudo lsof -p 32212
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
sshd    32212  abc  cwd    DIR              252,2     4096        2 /
sshd    32212  abc  rtd    DIR              252,2     4096        2 /
sshd    32212  abc  txt    REG              252,2   770944  1052832
/usr/sbin/sshd
sshd    32212  abc  DEL    REG                0,5           6017648 /dev/zero
sshd    32212  abc  mem    REG              252,2    14464 12060640
/lib/x86_64-linux-gnu/security/pam_env.so
sshd    32212  abc  mem    REG              252,2    22896 12060651
/lib/x86_64-linux-gnu/security/pam_limits.so
sshd    32212  abc  mem    REG              252,2    10320 12060631
/lib/x86_64-linux-gnu/security/pam_mail.so
sshd    32212  abc  mem    REG              252,2    10344 12060643
/lib/x86_64-linux-gnu/security/pam_motd.so
sshd    32212  abc  mem    REG              252,2    14592 11931111
/lib/x86_64-linux-gnu/libpam_misc.so.0.82.0
sshd    32212  abc  mem    REG              252,2    38920 11930990
/lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0
sshd    32212  abc  mem    REG              252,2    96280 11931065
/lib/x86_64-linux-gnu/libnih.so.1.0.0
sshd    32212  abc  mem    REG              252,2   108480 11931142
/lib/x86_64-linux-gnu/libcgmanager.so.0.0.0
sshd    32212  abc  mem    REG              252,2    42864 12061049
/lib/x86_64-linux-gnu/security/pam_systemd.so
sshd    32212  abc  mem    REG              252,2    10376 12060633
/lib/x86_64-linux-gnu/security/pam_umask.so
sshd    32212  abc  mem    REG              252,2    10288 12060636
/lib/x86_64-linux-gnu/security/pam_keyinit.so
sshd    32212  abc  mem    REG              252,2    10344 12060673
/lib/x86_64-linux-gnu/security/pam_loginuid.so
sshd    32212  abc  mem    REG              252,2    18752 12060650
/lib/x86_64-linux-gnu/security/pam_selinux.so
sshd    32212  abc  mem    REG              252,2    10272 12060638
/lib/x86_64-linux-gnu/security/pam_nologin.so
sshd    32212  abc  mem    REG              252,2    18952 11931054
/lib/x86_64-linux-gnu/libcap.so.2.24
sshd    32212  abc  mem    REG              252,2    10376 12060669
/lib/x86_64-linux-gnu/security/pam_cap.so
sshd    32212  abc  mem    REG              252,2     6112 12060634
/lib/x86_64-linux-gnu/security/pam_permit.so
sshd    32212  abc  mem    REG              252,2     6024 12060632
/lib/x86_64-linux-gnu/security/pam_deny.so
sshd    32212  abc  mem    REG              252,2    60288 12060663
/lib/x86_64-linux-gnu/security/pam_unix.so
sshd    32212  abc  mem    REG              252,2    22952 11931136
/lib/x86_64-linux-gnu/libnss_dns-2.19.so
sshd    32212  abc  mem    REG              252,2    43616 11931097
/lib/x86_64-linux-gnu/libnss_files-2.19.so
sshd    32212  abc  mem    REG              252,2    47760 11932248
/lib/x86_64-linux-gnu/libnss_nis-2.19.so
sshd    32212  abc  mem    REG              252,2    39824 11931504
/lib/x86_64-linux-gnu/libnss_compat-2.19.so
sshd    32212  abc  mem    REG              252,2   101240 11931020
/lib/x86_64-linux-gnu/libresolv-2.19.so
sshd    32212  abc  mem    REG              252,2    14256 11931023
/lib/x86_64-linux-gnu/libkeyutils.so.1.4
sshd    32212  abc  mem    REG              252,2    43672  1442199
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
sshd    32212  abc  mem    REG              252,2   186824  1442370
/usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
sshd    32212  abc  mem    REG              252,2    31792 11931148
/lib/x86_64-linux-gnu/librt-2.19.so
sshd    32212  abc  mem    REG              252,2   141574 11931125
/lib/x86_64-linux-gnu/libpthread-2.19.so
sshd    32212  abc  mem    REG              252,2   252032 11931089
/lib/x86_64-linux-gnu/libpcre.so.3.13.1
sshd    32212  abc  mem    REG              252,2    14664 11931137
/lib/x86_64-linux-gnu/libdl-2.19.so
sshd    32212  abc  mem    REG              252,2    97296 11931061
/lib/x86_64-linux-gnu/libnsl-2.19.so
sshd    32212  abc  mem    REG              252,2  1857312 11931119
/lib/x86_64-linux-gnu/libc-2.19.so
sshd    32212  abc  mem    REG              252,2    14592 11931002
/lib/x86_64-linux-gnu/libcom_err.so.2.1
sshd    32212  abc  mem    REG              252,2   831616  1442299
/usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
sshd    32212  abc  mem    REG              252,2   290520  1442171
/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
sshd    32212  abc  mem    REG              252,2    43368 11931150
/lib/x86_64-linux-gnu/libcrypt-2.19.so
sshd    32212  abc  mem    REG              252,2   100728 11931120
/lib/x86_64-linux-gnu/libz.so.1.2.8
sshd    32212  abc  mem    REG              252,2    10680 11931014
/lib/x86_64-linux-gnu/libutil-2.19.so
sshd    32212  abc  mem    REG              252,2  1934624 11930991
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
sshd    32212  abc  mem    REG              252,2   281552 11930996
/lib/x86_64-linux-gnu/libdbus-1.so.3.7.6
sshd    32212  abc  mem    REG              252,2    14536  1442179
/usr/lib/x86_64-linux-gnu/libck-connector.so.0.0.0
sshd    32212  abc  mem    REG              252,2   134296 11931101
/lib/x86_64-linux-gnu/libselinux.so.1
sshd    32212  abc  mem    REG              252,2    55856 11931058
/lib/x86_64-linux-gnu/libpam.so.0.83.1
sshd    32212  abc  mem    REG              252,2   104936 11930995
/lib/x86_64-linux-gnu/libaudit.so.1.0.0
sshd    32212  abc  mem    REG              252,2    36632 11931139
/lib/x86_64-linux-gnu/libwrap.so.0.7.6
sshd    32212  abc  mem    REG              252,2   149120 11931276
/lib/x86_64-linux-gnu/ld-2.19.so
sshd    32212  abc  DEL    REG                0,5           6020490 /dev/zero
sshd    32212  abc    0u   CHR                1,3      0t0        6 /dev/null
sshd    32212  abc    1u   CHR                1,3      0t0        6 /dev/null
sshd    32212  abc    2u   CHR                1,3      0t0        6 /dev/null
sshd    32212  abc    3u  IPv4            6021256      0t0      TCP
IP_________14:ssh->IP_________40:55340 (ESTABLISHED)
sshd    32212  abc    4u  unix 0xffff88032fb46000      0t0  6017653 socket
sshd    32212  abc    5u  unix 0xffff880100cd6c00      0t0  6015704 socket
sshd    32212  abc    6r  FIFO               0,10      0t0  6021267 pipe
sshd    32212  abc    7w  FIFO               0,18      0t0      604
/run/systemd/sessions/1622.ref
sshd    32212  abc    8w  FIFO               0,10      0t0  6021267 pipe


Regards
Vangelis

Paul Slootman

2017-Aug-30 14:39 UTC

head link

rsync got stuck

On Wed 23 Aug 2017, Vangelis Katsikaros via rsync wrote:
> abc       3797  3796  0 01:12 ?        00:03:14 /usr/bin/rsync --compress
--compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet --update
--exclude=/.* /SRC_PATH/ DEST_____IP:/DEST_PATH/
Try running rsync without the --compress option, that has been the
source of problems in the past.
If you do need compression you could add that at the ssh level.


Paul

Vangelis Katsikaros

2017-Sep-05 09:04 UTC

head link

rsync got stuck

On 08/30/2017 05:39 PM, Paul Slootman wrote:> On Wed 23 Aug 2017, Vangelis Katsikaros via rsync wrote:
> 
>> abc       3797  3796  0 01:12 ?        00:03:14 /usr/bin/rsync
--compress --compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet
--update --exclude=/.* /SRC_PATH/ DEST_____IP:/DEST_PATH/
> 
> Try running rsync without the --compress option, that has been the
> source of problems in the past.
> If you do need compression you could add that at the ssh level.
Thanks Paul! I've given it a go, and I'll see how it goes.

For reference, regarding the --compress options do you have any specific bug(s)
in mind or is this something you have noticed in practice?

Regards
Vangelis

Possibly Parallel Threads

Search for more seemingly similar threads

rsync - Aug 2017 - rsync got stuck

rsync got stuck

rsync got stuck

rsync got stuck

Possibly Parallel Threads