openssh bugs - Aug 2024 - [Bug 3720] New: ssh-keygen -R fails and/or leaves temp files when run concurrently

https://bugzilla.mindrot.org/show_bug.cgi?id=3720

            Bug ID: 3720
           Summary: ssh-keygen -R fails and/or leaves temp files when run
                    concurrently
           Product: Portable OpenSSH
           Version: 9.0p1
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: ssh-keygen
          Assignee: unassigned-bugs at mindrot.org
          Reporter: jason.cipriani at gmail.com

Created attachment 3832
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3832&action=edit
Python 3.10 program to repro issue.

Versions affected (at least):
- 7.9p1 on Debian 10
- 8.9p1 on Ubuntu 22
- 9.0p1 on macOS 13

When multiple `ssh-keygen -R` instances are run concurrently,
occasionally one of two things happens:

1. It creates a "known_hosts.xxxxxxxx" temporary file, the operation
succeeds, but it fails to delete the temporary file on exit, OR
2. It fails to link "known_hosts.old" and the operation fails.

I've attached a Python (3) program that reliably reproduces the issue
on all of the above versions and OSes. The program generates a test
file named "test_known_hosts" containing 100 hosts, then attempts to
use `ssh-keygen -R` to remove all of those hosts on 100 separate
threads concurrently.

Running the Python program will, more often than not, leave a
collection of temp files remaining as well as log a few
"known_hosts.old"-based failures, in which case examination of the
"test_known_hosts" file will show some hosts left over (i.e. some
failed operations).

If the Python program is modified to serialize all of the `ssh-keygen`
runs, then the problem will not occur.

Note that this implies that running ssh-keygen from multiple
*processes* concurrently could also run into issues.

For context, I have a server running an automated process that, long
story short, runs `ssh-keygen -R` a few hundred times every 20 minutes
or so (don't ask), and I recently discovered 750,000 temporary files
left over in ~/.ssh taking up approximately 34GB of hard drive space. I
solved the problem by removing the concurrency on `ssh-keygen` runs. 

Note also that `ssh` itself does *not* appear to have concurrency
issues when adding hosts to the known_hosts file.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.