Chris Schanzle
2021-Apr-14 03:36 UTC
[CentOS] rsync over ssh stalls after completing the job
On 4/13/21 5:00 PM, Frank Cox wrote:> On Tue, 13 Apr 2021 22:29:26 +0200 > Simon Matter wrote: > >> You could try running strace on the hanging process so see what it's doing. > [frankcox at mutt temp]$ rsync -avv ../temp/ jeff:temp > opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC . temp (7 args) > sending incremental file list > delta-transmission enabled > abc is uptodate > total: matches=0 hash_hits=0 false_alarms=0 data=0 > > Leaving that sit there apparently doing nothing (but still not giving me my cursor back) I switched to another terminal window and did the following: > > [frankcox at mutt ~]$ ps -FA | grep rsync > frankcox 5400 2435 0 60586 3160 5 14:52 pts/0 00:00:00 rsync -avv ../temp/ jeff:temp > frankcox 5401 5400 0 67980 7440 1 14:52 pts/0 00:00:00 sshjeff rsync --server -vvlogDtpre.iLsfxC . temp> frankcox 5526 5416 0 55476 1076 3 14:53 pts/1 00:00:00 grep --color=auto rsync > > [frankcox at mutt ~]$ strace -p 5401 > strace: Process 5401 attached > select(11, [5 9 10], [], NULL, NULL > > Then it just sits there with no further action. I get my cursor back when I hit ctrl-c. > > [frankcox at mutt ~]$ strace -p 5400 > strace: Process 5400 attached > restart_syscall(<... resuming interrupted nanosleep ...>) = 0 > wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 > nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 > wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 > nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 > wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 > nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 > wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 > nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 > wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 > nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 > wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 > nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 > wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 > > The wait4-etc line just keeps repeating endlessly until I hit ctrl-c. > > Unfortunately, I have no idea what any of the above actually means. Does it tell us anything interesting?Yay!? I am glad someone else on the planet is experiencing this.? I noticed this started happening to me after updating some CentOS Linux 8 systems today. I discovered if I set ForwardX11=no (either on ssh command line or in ~/.ssh/config) the hang does not happen.? But why does that matter?? No updates to openssh. It is not the systemd update doing something silly with session management.? I painfully downgraded manually and rebooted to no effect.? As an aside, why can't we we have nice things in life like 'dnf downgrade systemd\*' actually work?? I did the below - might be dumb, but it worked -- alternate suggestions to downgrade are appreciated - searching the list and my google-fu was off the mark today. ? cd [path-to-repo]/centos/8/BaseOS/x86_64/os/Packages ? dnf downgrade $(rpm -qa systemd\* | grep 239-41.el8_3.2 | sed -e 's/3\.2/3.1/' -e 's/^/.\//' -e 's/$/.rpm/') Chris
On 4/13/21 11:36 PM, Chris Schanzle via CentOS wrote:> On 4/13/21 5:00 PM, Frank Cox wrote: >> On Tue, 13 Apr 2021 22:29:26 +0200 >> Simon Matter wrote: >> >>> You could try running strace on the hanging process so see what it's doing. >> [frankcox at mutt temp]$ rsync -avv ../temp/ jeff:temp >> opening connection using: ssh jeff rsync --server -vvlogDtpre.iLsfxC .temp (7 args)>> sending incremental file list >> delta-transmission enabled >> abc is uptodate >> total: matches=0 hash_hits=0 false_alarms=0 data=0 >> >> Leaving that sit there apparently doing nothing (but still not giving me my cursor back) I switched to another terminal window and did the following: >> >> [frankcox at mutt ~]$ ps -FA | grep rsync >> frankcox 5400 2435 0 60586 3160 5 14:52 pts/0 00:00:00 rsync -avv ../temp/ jeff:temp >> frankcox 5401 5400 0 67980 7440 1 14:52 pts/0 00:00:00 ssh > jeff rsync --server -vvlogDtpre.iLsfxC . temp >> frankcox 5526 5416 0 55476 1076 3 14:53 pts/1 00:00:00 grep --color=auto rsync >> >> [frankcox at mutt ~]$ strace -p 5401 >> strace: Process 5401 attached >> select(11, [5 9 10], [], NULL, NULL >> >> Then it just sits there with no further action. I get my cursor back when I hit ctrl-c. >> >> [frankcox at mutt ~]$ strace -p 5400 >> strace: Process 5400 attached >> restart_syscall(<... resuming interrupted nanosleep ...>) = 0 >> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 >> nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 >> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 >> nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 >> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 >> nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 >> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 >> nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 >> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 >> nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 >> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 >> nanosleep({tv_sec=0, tv_nsec=20000000}, NULL) = 0 >> wait4(5401, 0x7ffd45105564, WNOHANG, NULL) = 0 >> >> The wait4-etc line just keeps repeating endlessly until I hit ctrl-c. >> >> Unfortunately, I have no idea what any of the above actually means. Does it tell us anything interesting? > > Yay!? I am glad someone else on the planet is experiencing this.? > I noticed this started happening to me after updating some CentOS Linux8> systems today. > > I discovered if I set ForwardX11=no (either on ssh command line or in~/.ssh/config) the hang does not happen.? But why does that matter?? No updates to openssh.> > It is not the systemd update doing something silly with session management.? I painfully downgraded manually and rebooted to no effect.?> As an aside, why can't we we have nice things in life like 'dnf downgrade > systemd\*' actually work?? I did the below - might be dumb, but itworked -- alternate suggestions to downgrade are appreciated - searching the list and my google-fu was off the mark today.> > ? cd [path-to-repo]/centos/8/BaseOS/x86_64/os/Packages > ? dnf downgrade $(rpm -qa systemd\* | grep 239-41.el8_3.2 | sed -e's/3\.2/3.1/' -e 's/^/.\//' -e 's/$/.rpm/')> > Chris[adjusted the subject, hope that is OK.] Found it!? It's the dbus update to 1.12.8-12.? Downgrade to -11 and ssh connections close normally. To clarify the problem, with the new dbus, simple ssh's like: ssh somehost uptime will print the uptime, but do not return to the local shell prompt until you hit ctrl-c.? It works normally if you downgrade dbus or ssh -o forwardx11=no somehost uptime I'm sure a bug report exists somewhere, but that's something to dig for or create tomorrow. To downgrade, packages were scattered in different locations, so I copied them to one directory and did dnf downgrade ./* The packages I needed to downgrade on a? x86_64 system were: dbus-1.12.8-11.el8.x86_64.rpm dbus-common-1.12.8-11.el8.noarch.rpm dbus-daemon-1.12.8-11.el8.x86_64.rpm dbus-devel-1.12.8-11.el8.x86_64.rpm dbus-libs-1.12.8-11.el8.x86_64.rpm dbus-tools-1.12.8-11.el8.x86_64.rpm dbus-x11-1.12.8-11.el8.x86_64.rpm
On Tue, 13 Apr 2021 23:36:48 -0400 Chris Schanzle via CentOS wrote:> I discovered if I set ForwardX11=no (either on ssh command line or in > ~/.ssh/config) the hang does not happen.? But why does that matter?? No > updates to openssh.Far out! That's the solution! Well, not really a solution but it's certainly a work-around that makes rsync over ssh work without hanging. I discovered that you have to put the "ForwardX11 no" into the default part of .ssh/config. It doesn't work if you just specify the host involved. This doesn't work: Host * ForwardX11 yes host jeff ForwardX11 no This does work: Host * ForwardX11 no Someone should probably file a bug report about this, but I have no idea what package it pertains to. As you said, openssh wasn't updated at the point that this broke. If it's of any value, here's a list of what was updated last on this computer. The ones that look most suspicious to me would be kernel, crypto-policies and/or systemd. Packages Altered: Install kernel-4.18.0-240.22.1.el8_3.x86_64 @baseos Install kernel-core-4.18.0-240.22.1.el8_3.x86_64 @baseos Install kernel-debug-devel-4.18.0-240.22.1.el8_3.x86_64 @baseos Install kernel-devel-4.18.0-240.22.1.el8_3.x86_64 @baseos Install kernel-modules-4.18.0-240.22.1.el8_3.x86_64 @baseos Upgrade dbus-x11-1:1.12.8-12.el8_3.x86_64 @appstream Upgraded dbus-x11-1:1.12.8-11.el8.x86_64 @@System Upgrade flatpak-1.6.2-6.el8_3.x86_64 @appstream Upgraded flatpak-1.6.2-5.el8_3.x86_64 @@System Upgrade flatpak-libs-1.6.2-6.el8_3.x86_64 @appstream Upgraded flatpak-libs-1.6.2-5.el8_3.x86_64 @@System Upgrade flatpak-selinux-1.6.2-6.el8_3.noarch @appstream Upgraded flatpak-selinux-1.6.2-5.el8_3.noarch @@System Upgrade flatpak-session-helper-1.6.2-6.el8_3.x86_64 @appstream Upgraded flatpak-session-helper-1.6.2-5.el8_3.x86_64 @@System Upgrade java-1.8.0-openjdk-1:1.8.0.282.b08-2.el8_3.x86_64 @appstream Upgraded java-1.8.0-openjdk-1:1.8.0.275.b01-1.el8_3.x86_64 @@System Upgrade java-1.8.0-openjdk-devel-1:1.8.0.282.b08-2.el8_3.x86_64 @appstream Upgraded java-1.8.0-openjdk-devel-1:1.8.0.275.b01-1.el8_3.x86_64 @@System Upgrade java-1.8.0-openjdk-headless-1:1.8.0.282.b08-2.el8_3.x86_64 @appstream Upgraded java-1.8.0-openjdk-headless-1:1.8.0.275.b01-1.el8_3.x86_64 @@System Upgrade bpftool-4.18.0-240.22.1.el8_3.x86_64 @baseos Upgraded bpftool-4.18.0-240.15.1.el8_3.x86_64 @@System Upgrade crypto-policies-20210209-1.gitbfb6bed.el8_3.noarch @baseos Upgraded crypto-policies-20200713-1.git51d1222.el8.noarch @@System Upgrade crypto-policies-scripts-20210209-1.gitbfb6bed.el8_3.noarch @baseos Upgraded crypto-policies-scripts-20200713-1.git51d1222.el8.noarch @@System Upgrade dbus-1:1.12.8-12.el8_3.x86_64 @baseos Upgraded dbus-1:1.12.8-11.el8.x86_64 @@System Upgrade dbus-common-1:1.12.8-12.el8_3.noarch @baseos Upgraded dbus-common-1:1.12.8-11.el8.noarch @@System Upgrade dbus-daemon-1:1.12.8-12.el8_3.x86_64 @baseos Upgraded dbus-daemon-1:1.12.8-11.el8.x86_64 @@System Upgrade dbus-libs-1:1.12.8-12.el8_3.x86_64 @baseos Upgraded dbus-libs-1:1.12.8-11.el8.x86_64 @@System Upgrade dbus-tools-1:1.12.8-12.el8_3.x86_64 @baseos Upgraded dbus-tools-1:1.12.8-11.el8.x86_64 @@System Upgrade file-5.33-16.el8_3.1.x86_64 @baseos Upgraded file-5.33-16.el8.x86_64 @@System Upgrade file-libs-5.33-16.el8_3.1.x86_64 @baseos Upgraded file-libs-5.33-16.el8.x86_64 @@System Upgrade kernel-headers-4.18.0-240.22.1.el8_3.x86_64 @baseos Upgraded kernel-headers-4.18.0-240.15.1.el8_3.x86_64 @@System Upgrade kernel-tools-4.18.0-240.22.1.el8_3.x86_64 @baseos Upgraded kernel-tools-4.18.0-240.15.1.el8_3.x86_64 @@System Upgrade kernel-tools-libs-4.18.0-240.22.1.el8_3.x86_64 @baseos Upgraded kernel-tools-libs-4.18.0-240.15.1.el8_3.x86_64 @@System Upgrade python3-perf-4.18.0-240.22.1.el8_3.x86_64 @baseos Upgraded python3-perf-4.18.0-240.15.1.el8_3.x86_64 @@System Upgrade systemd-239-41.el8_3.2.x86_64 @baseos Upgraded systemd-239-41.el8_3.1.x86_64 @@System Upgrade systemd-libs-239-41.el8_3.2.x86_64 @baseos Upgraded systemd-libs-239-41.el8_3.1.x86_64 @@System Upgrade systemd-pam-239-41.el8_3.2.x86_64 @baseos Upgraded systemd-pam-239-41.el8_3.1.x86_64 @@System Upgrade systemd-udev-239-41.el8_3.2.x86_64 @baseos Upgraded systemd-udev-239-41.el8_3.1.x86_64 @@System Upgrade zlib-1.2.11-16.2.el8_3.x86_64 @baseos Upgraded zlib-1.2.11-16.el8_2.x86_64 @@System Upgrade zlib-devel-1.2.11-16.2.el8_3.x86_64 @baseos Upgraded zlib-devel-1.2.11-16.el8_2.x86_64 @@System Removed kernel-4.18.0-240.1.1.el8_3.x86_64 @@System Removed kernel-core-4.18.0-240.1.1.el8_3.x86_64 @@System Removed kernel-debug-devel-4.18.0-240.1.1.el8_3.x86_64 @@System Removed kernel-devel-4.18.0-240.1.1.el8_3.x86_64 @@System Removed kernel-modules-4.18.0-240.1.1.el8_3.x86_64 @@System -- MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com