Hi, We've landed some fairly significant changes in OpenSSH recently and would appreciate your help in testing them. The biggest of the changes are: 1. Conversion of the ssh and sshd mainloop from select() to poll() This should be entirely invisible to users, so any behaviour change is a bug. If you see something and want to help debug it further, uncomment the DEBUG_CHANNEL_POLL #define in channels.c for helps of extra debug logging. 2. Restricted agent keys. This is a large set of changes to add destination- and path-restricted keys to ssh-agent. A full writeup is at on the website at https://www.openssh.com/agent-restrict.html - I'm interested to hear feedback on how this works in practice, UI and things that could be improved (as well as bug reports). 3. Running down the remaining RSA/SHA2 corner-cases There has been a fair bit of work to identify and fix the remaining cases where various things behaved badly wrt RSA signature algorithms. Recent fixes include hostbased authentication and UpdateHostkeys. Again, [almost] any change in visible behaviour here is a bug. All of these changes are in git and will be in tomorrow's snapshot (20220108). Thanks! Damien Miller
On 2022-01-07 at 09:52 +1100, Damien Miller wrote:> Hi, > > We've landed some fairly significant changes in OpenSSH recently and > would appreciate your help in testing them. The biggest of the > changes > are: > (...) > > All of these changes are in git and will be in tomorrow's snapshot > (20220108). > > Thanks! > > Damien MillerHello Damien That's very interesting. I have thought about this some time ago (maybe it was mentioned on this list?), but I had only considered each ssh(1) prepending its parameters to the agent. This approach is more robust, although I would include in SSH_AGENTC_EXTENSION a string for identifying how the connection was made (typically the ssh call, which could be untrue, of course). This could be useful for debugging, or when there are earlier hops through unknown hosts. I think the document should reference somewhere the more secure option of chaining ssh connections using ProxyJump, that this restriction doesn't apply there and ssh-agent considers such connections as local. As a further step, I would add the ability to specify wildcard destinations, e.g. ssh-add -h "*.prod.example.com" ~/.ssh/production.key for a key which could be used on any subdomain of prod.example.com, but not anywhere else. The restrictions are expressed in a right-associative way, which IMHO seem unintuitive. Something like "charybdis.example.org from scylla.example.org" or "charybdis.example.org<scylla.example.org" would perhaps emphasize better the beginning of the restriction is not fixed. What do other people think about it? It's not mentioned in the document, but I expect that when ssh-add parses restrictions that they support ports as well ssh-add -h "scylla.example.org>medea at charybdis.example.org:2222" I think "the keys for all the hosts that the user lists must be present in the right place (the machine running ssh-add) and the right time." intends to be "*at* the right time.", but it's probably better to state explicitly "at the time ssh-add is run". I see ssh-add gets added -H to choose a known_hosts file, good. But the text in ssh-add.1 "ssh-add will use the default ssh_config(5) known hosts files:", while strictly correct, may make it seem that it will somehow parse ~/.ssh/config or /etc/ssh/ssh_config for UserKnownHostsFile directives. I suggest "ssh-add will use the known hosts files: ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2, which is the default used by ssh_config(5)" A potential issue I see is, given the example: $ ssh-add -h "perseus at cetus.example.org" \ -h "scylla.example.org" \ -h "scylla.example.org>medea at charybdis.example.org" \ ~/.ssh/id_ed25519 Poseidon is root on cetus, and wants some files that medea saves in charybdis. When victim logs in as perseus at cetus.example.org (forwarding ssh- agent), Poseidon impersonates perseus and logs in to poseidon at scylla.example.org *without using this key* (e.g. with keyboard-authentication). He is then able to connect to medea at charybdis.example.org using the forwarded ssh-agent. (he could have used an evil ssh(1) to lie and run it directly from cetus with a fake binding, but since he needs a valid login to scylla, it's easier this way) This is likely unexpected, but I don't see an easy way to avoid it. One could force the whole path to be connected with the same key (maybe the code already does this?), but that would forbid using multiple keys along the path (e.g. one key for the bastion host, then jump from it to the dev environment that uses its own dev key) First of all, I think we need to be able to state the source user: -h "perseus at scylla.example.org>medea at charybdis.example.org" which requires a new field 'string from_username' on SSH_AGENT_CONSTRAIN_EXTENSION. And perhaps ssh-add should default that field to the current user if unset, although I would consider alternate User values in ~/.ssh/config to be relatively common. Finally, two minor nitpicks: a stray backslash in '\public' and s/pursues/pursued/ Sorry for the long list of comments! Best regards ?ngel
On 06.01.22 23:52, Damien Miller wrote:> 2. Restricted agent keys. > This is a large set of changes to add destination- and path-restricted > keys to ssh-agent. A full writeup is at on the website at > https://www.openssh.com/agent-restrict.html - I'm interested to hear > feedback on how this works in practice, UI and things that could be > improved (as well as bug reports).Quick introductory remark 1: The text speaks of "italicised" and "bolded" parts of the syntax definitions, but there seems to be no such markup within the HTML's '<pre class="proto">' blocks that contain those fixed-width-font snippets. QIR2: Haven't got the time, offhand, to actually run tests; I hope to get at that later. Hence, purely theoretical ranting for now. --------------- I have to admit that I'm somewhat losing orientation among all the variations of multi-hop connections that OpenSSH offers by now. To wit: ------- A. Plain CLI Chaining: hostA$ ssh -A hostB hostB$ ssh [ -A ] hostC [...] also: hostA$ ssh -A hostB 'ssh hostC ...' Out-of-the-box minimal-config-effort way to do it. (Also what we're bound to see when a third party sets up a "proper" non-GUI jump host into their networks and gives us access, as it puts config of the second hop entirely into the jump host and thus *their* hands, but that's unlikely to be relevant to the topic of agent forwarding, of course.) ------- B. Tunneling: hostA$ ssh -D $PORT hostB hostA$ ssh -o "ProxyCommand $THRU_LCL_SOCKS_AT_$PORT" hostC (There are variants, like using LocalForward's or some non-OpenSSH-provided proxy, but I don't see that making for a relevant difference in *this* discusson.) We use that one *a lot*, because in order to fully support customers' systems, we have to access them with browsers as well as per SSH. Our own new *outgoing*-to-customers jump host is essentially based on such configs. ------- C. ControlMaster D. ProxyJump Haven't used either much myself, so I'm unclear on relevant details for the time being. I see ProxyJump recommended in your document, and I hear that ControlMaster is quite popular in larger sites where a Nagios/Icinga/... runs checks on target hosts through (multi-hop) SSH logins (because it wraps the thousands/h of "virtual" SSH logins into a few actual TCP connections, and thus avoids triggering rate limits all over the place). ------- Now, back to the topic at hand: Agent forwarding is unnecessary in scenario B, and in scenario A, while the agent runs on hostA, all the automatic maintenance of known_hosts for the hosts you actually *use* the forwarding for happens on hostB. Yet, having them properly listed in the known_hosts *on hostA* is vital for the new usage-restrictions mechanism. Also, in scenario B, the user will likely know that he's accessing hostC via hostB, and might try to restrict the key to "hostB>hostC", but hostB actually remains transparent to the ssh connecting to hostC and the restriction will fail. (If someone could pinpoint where known_hosts automaintenance actually happens in cases C and D, I'd be grateful. Since both are implemented *exclusively* in OpenSSH AFAIK, there *would* be the remote possibility of having them automaint known_hosts *both* on the jump host and the machine running the agent ...) --------------- Pondering the concepts behind the term "hostnames": Our company doesn't maintain oodles of internally-visible-only DNS entries. The supporters typically have a well-padded ~/.ssh/config with lots of entries like, e.g., Host CustX-SrvY HostName 12.34.56.78 # Customer X's public IP Host CustX-SrvZ HostName 192.168.76.54 # To be gatewayed through SrvY (We're used to having to set up a new *server* to replace the old one for the customers every now and then, so propagating new/changed IPs/entries by word of mouth works quite well. :-3 ) So, we have a loosely-defined "Host" name, an IP for the "HostName", and possibly a couple DNS cherries on top, in case OpenSSH makes any automated lookups. AFAICT it's the "HostName" that becomes the key in the known_hosts file, and thus, we would have to use *that* in the "ssh-add -h ..." parameters, too ... ... ugh. Finding and editing *one* entry in .ssh/config when an IP changes is OK-ish, but sifting through possibly elaborate destination constraints for every instance where the host may have been chosen to serve as a jump host ... I might be missing loads of other people's use cases here, but if an abstract concept of forwarding constraints were to be introduced in our company, the most viable approach would IMHO be that of administrative zones. As in, "this is a cert issued to me by customer X's CA, and I want it forwarded *only* *via* the known gateways into X's internal network, so let me markup those in my .ssh/config with 'AuthZone CustX' and load the cert with 'ssh-add -A CustX $CERTFILE'" ... (... which *might* be doable with several agents on different ports *as long as* I do not also have to use my "normal" keypairs to get around within X's network ...) Kind regards, -- Jochen Bern Systemingenieur Binect GmbH -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3449 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20220107/2fe34409/attachment-0001.p7s>
Darren Tucker
2022-Jan-10 11:04 UTC
OS X poll breakage (Was: Please help test recent changes)
On Fri, Jan 07, 2022 at 09:52:09AM +1100, Damien Miller wrote:> 1. Conversion of the ssh and sshd mainloop from select() to poll() > > This should be entirely invisible to users, so any behaviour change > is a bug. If you see something and want to help debug it further, > uncomment the DEBUG_CHANNEL_POLL #define in channels.c for helps of > extra debug logging.This change breaks (at least) the exit-status test on Mac OS X. I haven't had a chance to investigate yet, $ git bisect run sh -c "autoreconf && ./configure --without-openssl && make t-exec LTESTS=exit-status" [... lots ...] 17877bc81db3846e6e7d4cfb124d966bb9c9296b is the first bad commit commit 17877bc81db3846e6e7d4cfb124d966bb9c9296b Author: djm at openbsd.org <djm at openbsd.org> Date: Thu Jan 6 21:48:38 2022 +0000 upstream: convert ssh, sshd mainloops from select() to poll(); feedback & ok deraadt@ and markus@ has been in snaps for a few months OpenBSD-Commit-ID: a77e16a667d5b194dcdb3b76308b8bba7fa7239c $ uname -a Darwin osx-highsierra 17.7.0 Darwin Kernel Version 17.7.0: Fri Oct 30 13:34:27 PDT 2020; root:xnu-4570.71.82.8~1/RELEASE_X86_64 x86_64 -- Darren Tucker (dtucker at dtucker.net) GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new) Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
Quoting Damien Miller (2022-01-06 23:52:09)> 2. Restricted agent keys. > > This is a large set of changes to add destination- and path-restricted > keys to ssh-agent. A full writeup is at on the website at > https://www.openssh.com/agent-restrict.html - I'm interested to hear > feedback on how this works in practice, UI and things that could be > improved (as well as bug reports).Can this be made to work when SSHFP host verification is used (VerifyHostKeyDNS=yes) rather than known_hosts? Otherwise this is great news - I've largely replaced my old key with a FIDO token and one of the annoyances is the confirmation window not telling me which host is it for. -- Anton Khirnov
On Jan 7 09:52, Damien Miller wrote:> Hi, > > We've landed some fairly significant changes in OpenSSH recently and > would appreciate your help in testing them. The biggest of the changes > are: > > 1. Conversion of the ssh and sshd mainloop from select() to poll() > > This should be entirely invisible to users, so any behaviour change > is a bug. If you see something and want to help debug it further, > uncomment the DEBUG_CHANNEL_POLL #define in channels.c for helps of > extra debug logging. > > 2. Restricted agent keys. > > This is a large set of changes to add destination- and path-restricted > keys to ssh-agent. A full writeup is at on the website at > https://www.openssh.com/agent-restrict.html - I'm interested to hear > feedback on how this works in practice, UI and things that could be > improved (as well as bug reports). > > 3. Running down the remaining RSA/SHA2 corner-cases > > There has been a fair bit of work to identify and fix the remaining > cases where various things behaved badly wrt RSA signature algorithms. > Recent fixes include hostbased authentication and UpdateHostkeys. > Again, [almost] any change in visible behaviour here is a bug. > > All of these changes are in git and will be in tomorrow's snapshot > (20220108).Took me a while but today I tested this on recent Cygwin. The testsuite fails at one point: run test hostkey-agent.sh ... [...] cert type sk-ssh-ed25519-cert-v01 at openssh.com cert type sk-ssh-ed25519-cert-v01 at openssh.com failed bad SSH_CONNECTION key type sk-ssh-ed25519-cert-v01 at openssh.com [...] bad SSH_CONNECTION key type sk-ecdsa-sha2-nistp256-cert-v01 at openssh.com failed hostkey agent Looking into cat failed-sshd.log I notice this message for *all* agent-key.*.pub files: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0644 for '/home/corinna/tmp/openssh/regress/agent-key.ecdsa-sha2-nistp256.pub' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. Unable to load host key "/home/corinna/tmp/openssh/regress/agent-key.ecdsa-sha2-nistp256.pub": bad permissions Shouldn't the testsuite have generated the files with correct permissions in the first place? And then again, these are PUB files. Shouldn't a 644 permission suffice? Corinna