Hi All, I hope everyone is doing well. I have a question and I would appreciate if anyone can answer but first I want to thank so much all contributors for deliverig previous as well as the latest releases with its promising login session separation implementation having each session is spawned in its own process which I hope isnt what is causing my issue. Before I write my question, it might help understanding if I provide the context first, and therefore I come from background of having negative experience with sshd that drove me to analyse every corner in linux system I can and try to understand what is happening and more write thousands of lines of code trying to resolve the issue yet to no avail I found answer to the issue which unfoled with question and I am not being able to rectify it and after so many months of distraught suspecting my server is being compromised, I thought now to reach to the developers of opensshd with hope they might be able to answer my question which I couldnt find as it would allow me to rectify the issue with itself. I wrote an EDR application and part of it, has a functionality to keep track of logged in sessions, and report whenever new ssh session was started. I had this experience, after I terminate all sessions I started, sometimes hours later like today and other days later, and in random times, I get notification of new ssh session was started, and when I login to server to inspect, I find no login session and no hinting traces in netstat, top, secure and messages log, utmp dump etc. I noticed all of session with specific PIDs which is getting reported as was active for very short period of time, are identical to previous sessions I started and have already terminated. After observing the server for months, I noticed I get report of new session report whenever I run sudo -i who -a command, given because it was only through sudo by root, for strange reasons, it activates latest ssh session which I have started and terminated it from previous session with identical PID and reuses to perform the sudo command. But now I observed the logs in messages and secure file and monitored all services which may run any command with sudo and tested them, none report new session likewise sudo -i who -a and no application runs even this command itslef, and still till today in random times, I get report in server ssh session started via openssh 10, while its pid belongs to previous session I made 100% sure it was terminated, based on sudo experience, it looks like somewhere some commands gets executed in linux in suspicious times when I am not on my laptop, and it requires root and uses sudo, which cause to activate and reuses terminated ssh sessions, out of no where like sudo -i who -a did. If anyone want to confirm, run it continously with && who -a and see the iteration of different PIDs everytime gets printed in the latest terminated session at bottom of the output. Now, I have no application nor service runs any command with sudo, which may cause the terminated session reuse and makes me wonder more which part of linux is responsible of such ssh login session? Anyone have any idea? Isnt my server is being looking like compromised? I really wanted to build the most secure email application in the world, and I have been working on for years still not being able to reach the confidence I can deliver it with this random ssh session gets activated by someone isn't myself, could it be my VPS hosting provider, runs some periodic and random maintenance and executes some sudo commands? I believed that I developed very smart EDR application as part of one of my dreams hoping to be completed soon, as most secure email application yet this issue puzzles and doubts me and more than ever again I will appreciate if anyone have any idea about what is truly happening to share it. Also, as part of EDR, I developed reverse shell, hidden and suspicious processes as well as suspicious connections and opened ports for processes which isnt expected to establish, also kernel modules tampering, detection features and in times when I receive the ssh session gets activated for few seconds, and itself based on sudo -i who -a experience again, I inferred, usually such what I hope a false positive can happen when sudo commands gets executed, my applicaion detect no suspicious activity beside for some MSs to 1 second duration root ssh session gets activated? I hope its not another next level of sophisticated rootkit implanted in kernel or memory but, in short if anyone has any clarification, to prove my server isnt compromised and no one had an ability to escalate privileges and run root commands yet then why randomly I see active ssh session when no one truly is logged? Any answer, again I will appreciate it. Lastly, here is my most relevant question, Does and How sshd spawns and reuses ssh session with previous pids? and how it spawns each session processes and allocate its PID? Could it be something like other services which runs sudo commands actually yields enumeration through sshd pid range continously until it hits previous recent terminated session which activates for as long as it runs, and thus produce almost like to my tracking of ssh pids, pid collision effect? I really doubted every conclusion I reached, and I hope if anyone can help me on reaching a resolve and understand what is happening and help me deliver my dream email application soon to the world, any suggestion of place to investigate in sshd configs, service, process or anywhere else in Linux e.g. log any attempt to start ssh session before it gets started and the code execution which started? Any recommendation to do this as part of sshd binary? E.g. I can apply some code tweaks on source before compiling? With thanks. Zak.
On Mon, 28 Jul 2025, Zakaria via openssh-unix-dev wrote: [snip]> Lastly, here is my most relevant question, Does and How sshd spawns > and reuses ssh session with previous pids? and how it spawns each > session processes and allocate its PID?sshd just starts processes via fork(2), it can't really decide the PIDs those processes are assigned - that's up to the kernels. Different kernels use different PID assignment policies; some assign randomly, some use (basically) last_pid++, some use mininum free PID. As a general comment, your application should be prepared for PIDs being reused. It will always happen eventually. If you are using PIDs to identify processes without detecting when a process exits (e.g. using eBPF on Linux), then you're setting yourself up for confusion.> Could it be something like other services which runs sudo commands > actually yields enumeration through sshd pid range continously until > it hits previous recent terminated session which activates for as long > as it runs, and thus produce almost like to my tracking of ssh pids, > pid collision effect?Brute-forcing fork(2) until you get the PID you want is basically the only way for a user-space process to "select" a PID. I'd expect this to be pretty noisy as, to be effective, a process doing this would need to create a lot of processes and ideally hold them open until they hit the PID they wanted.> I really doubted every conclusion I reached, and I hope if anyone can > help me on reaching a resolve and understand what is happening and > help me deliver my dream email application soon to the world, any > suggestion of place to investigate in sshd configs, service, process > or anywhere else in Linux e.g. log any attempt to start ssh session > before it gets started and the code execution which started? Any > recommendation to do this as part of sshd binary? E.g. I can apply > some code tweaks on source before compiling?I don't know what you mean by "log any attempt to start ssh session before it gets started" - sshd logs pretty much everything it does. You could try turning on verbose or debug logging to see more of what it is doing. -d
hi, On Mon, Jul 28, 2025 at 10:36:36PM +0100, Zakaria via openssh-unix-dev wrote:> After observing the server for months, I noticed I get report of new session > report whenever I run sudo -i who -a command, given because it was only > through sudo by root, for strange reasons, it activates latest ssh session > which I have started and terminated it from previous session with identical > PID and reuses to perform the sudo command.sudo has recently grown the annoying behaviour of opening a pseudo-tty (pty) to run the command you specified in. Which might be reported as "a new session" (new pty, new process group, etc). ceterum censeo sudo delendam esse... gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany gert at greenie.muc.de
On 28.07.25 23:36, Zakaria wrote:> ... drove me to analyse every corner in linux system ... > ... sometimes hours later like today and other days later, and in > random times, I get notification of new ssh session was started, and > when I login to server to inspect, I find no login session and no > hinting traces in netstat, top, secure and messages log, utmp dump etc. > I noticed all of session with specific PIDs which is getting reported as > was active for very short period of time, are identical to previous > sessions I started and have already terminated. > > After observing the server for months, I noticed I get report of new > session report whenever I run sudo -i who -a command ...This is from a rather busy (for our standards), Linux-based SFTP server where I looked up the "history" of the sshd PID involved in my own login:> # grep ' sshd\[26671\]: .* session ' /var/log/secure | sed -e 's/:.. .*://' -e 's/\( user [^b]\).*/\1.../' > Jul 27 21:05 session opened for user t... > Jul 27 21:05 session closed for user t... > Jul 28 07:39 session opened for user C... > Jul 28 07:39 session closed for user C... > Jul 28 14:40 session opened for user C... > Jul 28 14:40 session closed for user C... > Jul 29 00:25 session opened for user l... > Jul 29 00:25 session closed for user l... > Jul 29 03:38 session opened for user C... > Jul 29 03:38 session closed for user C... > Jul 29 05:33 session opened for user n... > Jul 29 05:33 session closed for user n... > Jul 29 11:58 session opened for user bern by (uid=0)As you can see, PIDs getting reused for different sessions is a perfectly normal thing. The *frequency* at which it happens depends on use (is your server's SSH port open to the entire Internet? If yes, you're bound to be hit with scans 24/7) and other factors (our SFTP(!) server still uses 32 bit PIDs), of course. Having that said, if you see an *SSH session* being reported when running the "sudo -i who -a" command (*without* SSHing into the server anew for every time you run it), something's amiss. It'll certainly create "sessions" of some kind, so there's room for a misinterpretation, depending on what inputs your detector uses, but it shouldn't involve sshd, neither for itself nor as in "resurrecting" already-terminated SSH sessions. Since you mention that the server in question is being hosted, would it be possible to run the detector and an artificial SSH-logins load on a local machine to see whether the symptom appears there as well? Kind regards, -- Jochen Bern Systemingenieur Binect GmbH -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4336 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20250729/6c8f85a8/attachment-0001.p7s>