openssh unix dev - Oct 2024 - sshd fails when using cryptodev-linux to compute hmac

Hi Damien,
> I don't know anything about cryptodev-linux, but I assume it's an
openssl engine?
Cryptodev-linux is a kernel module that provides access to kernel crypto
drivers, especially hardware-accelerated crypto, through the /dev/crypto device.
Openssl implements an engine which interfaces to it.
> If so it's possible sshd's multiprocess model and/or file
descriptor handling is confusing it.
This seems like a reasonable explanation based on what I've seen so far.
> It's not a configuration we test, so you're mostly on your own to
debug it. It's entirely possible there's a bug there; if so, I'd
expect it to be something like a fd being closed while devcrypto is still
depending on it.
>
> I'd suggest turning on LogVerbose=* so you can see which process
(represented by it's PID) is doing what, though that probably won't be
represented in the devcrypto debug messages unless you hack something similar
in.
Too bad, I was hoping it was a tested/supported configuration. Since that
doesn't seem to be the case, I suspect the easiest way forward for me is
going to be disabling the openssl engine entirely so that openssh works
properly. I doubt that hardware-accelerated crypto is going to have much benefit
for SSH workloads anyway.

Thanks,
Peter

On 10/9/24 1:31 PM, Peter Rashleigh wrote:
> Too bad, I was hoping it was a tested/supported configuration. Since that
doesn't seem to be the case, I suspect the easiest way forward for me is
going to be disabling the openssl engine entirely so that openssh works
properly. I doubt that hardware-accelerated crypto is going to have much benefit
for SSH workloads anyway.
That might not be entirely true. In workflows that use ssh for bulk data 
transfers the encryption method can be a bottleneck. Offloading the work 
to a hardware accelerator may improve throughput performance in these 
cases. It also would depend on what encryption method you are using. 
That said, even with transfers with sub ms RTTs hardware offloading will 
push the bottleneck to other codepaths. It's tough to get more than 
10Gbs even if your crypto can handle higher throughput.

Hi Peter,

?? 9.10.24 ?. ? 20:31 ?., Peter Rashleigh ??????:
> Hi Damien,
> 
>> I don't know anything about cryptodev-linux, but I assume it's
an openssl engine?
> Cryptodev-linux is a kernel module that provides access to kernel crypto
drivers, especially hardware-accelerated crypto, through the /dev/crypto device.
Openssl implements an engine which interfaces to it.
OpenBSD hates loadable modules. So you cannot expect support for such 
functionality.
> 
>> If so it's possible sshd's multiprocess model and/or file
descriptor handling is confusing it.
> This seems like a reasonable explanation based on what I've seen so
far.

PKIX-SSH support loadable modules(engines) and so, try to ensure proper 
management of cryptographic library. This includes slightly different 
management of file descriptors taking into account that some openssl 
configurations require open descriptors to devices. Also note that this 
could be a loadable module.

>> It's not a configuration we test, so you're mostly on your own
to debug it. It's entirely possible there's a bug there; if so, I'd
expect it to be something like a fd being closed while devcrypto is still
depending on it.
>>
>> I'd suggest turning on LogVerbose=* so you can see which process
(represented by it's PID) is doing what, though that probably won't be
represented in the devcrypto debug messages unless you hack something similar
in.
> Too bad, I was hoping it was a tested/supported configuration. Since that
doesn't seem to be the case, I suspect the easiest way forward for me is
going to be disabling the openssl engine entirely so that openssh works
properly. I doubt that hardware-accelerated crypto is going to have much benefit
for SSH workloads anyway.

Unfortunately or not I'm not able to confirm issue on 64 bit AMD. 
Environment:
  - qemu virtual machine,
  - OpenSUSE leap 15.4,
  - current pxis-ssh (15.3+)
  - current openssl stable (3.3.2+),
  - current cryptodev kernel modile (1.14+),
  - openssl configuration with described  devcrypto engine and 
activated(init = 1)

All regression test pass. Also putty interoperability regression tests 
pass as well.


About '... custom buildroot Linux 6.1.53 running on ARMv8...'

You could test build without so called "hardening" options. Some
options
are not maintained and code may fail on non-x86 processors.

> 
> Thanks,
> Peter
Roumen

On Wed, 9 Oct 2024, Peter Rashleigh wrote:
> > I'd suggest turning on LogVerbose=* so you can see which process
> > (represented by it's PID) is doing what, though that probably
won't
> > be represented in the devcrypto debug messages unless you hack
> > something similar in.
>
> Too bad, I was hoping it was a tested/supported configuration. Since
> that doesn't seem to be the case, I suspect the easiest way forward
> for me is going to be disabling the openssl engine entirely so that
> openssh works properly. I doubt that hardware-accelerated crypto is
> going to have much benefit for SSH workloads anyway.
Yeah, we used to support /dev/crypto on OpenBSD and it only really
helped with pre-2005ish CPUs; the costs of massaging/moving the data
around and user/kernel context switching erased the gains. Maybe
this would be better if the application was explicitly designed for
the kernel interface, but OpenSSH isn't

Hardware-accelerated crypto is still a huge win when it doesn't need
to go via the kernel, e.g. AES-NI.

-d