openssh unix dev - Jun 2024 - An Analysis of the DHEat DoS Against SSH in Cloud Environments

On Wed, 2024-06-19 at 16:11 -0400, Joseph S. Testa II
wrote:
> I suppose in the next few days, I'll try reproducing my original
> steps
> with the new version and see what happens.
I managed to do some limited testing with a local VM, and the results
are... interesting.

I installed openssh-SNAP-20240626.tar.gz on a fresh and fully-updated
Ubuntu Linux 24.04 LTS VM with 1 vCPU.  While leaving the default sshd
options unchanged, I was able to reduce idle time to 0.0% using "./ssh-
audit.py --dheat=16 target_host".

Next, I increased the vCPUs to 4.  The same ssh-audit command yielded
54% idle time (averaged over 60 seconds).  That's still a lot of strain
on the target, despite the fact that the logs claim that the
PerSourcePenalties noauth:1 restriction was being triggered.

After that, I tried simply flooding the target with open connections
without performing the DHEat attack ("ssh-audit.py --conn-rate-test=16
target_host").  This caused the 60-second average idle time to come all
the way down to 6%!  Additionally, I noticed that the systemd-journal
process was consuming about 50% CPU and /var/log/auth.log grew by
nearly 14MB.  Aside from CPU exhaustion, some may say that causing log
growth at a rate of 14MB/minute would constitute a disk space
exhaustion problem.

Seems like the new PerSourcePenalties implementation/default settings
still allow a denial-of-service by attackers with low-latency network
connections.

   - Joe

Joseph S. Testa II wrote in
 <700cf304a8e24450e5e6e32d7a0020f5516cea3d.camel at positronsecurity.com>:
 |On Wed, 2024-06-19 at 16:11 -0400, Joseph S. Testa II wrote:
 ...
 |After that, I tried simply flooding the target with open connections
 |without performing the DHEat attack ("ssh-audit.py --conn-rate-test=16
 |target_host").  This caused the 60-second average idle time to come all
 |the way down to 6%!  Additionally, I noticed that the systemd-journal
 |process was consuming about 50% CPU and /var/log/auth.log grew by
 |nearly 14MB.  Aside from CPU exhaustion, some may say that causing log
 |growth at a rate of 14MB/minute would constitute a disk space
 |exhaustion problem.
 |
 |Seems like the new PerSourcePenalties implementation/default settings
 |still allow a denial-of-service by attackers with low-latency network
 |connections.

You mean without any otherwise assisting firewall settings, and
without "a plan" for log partition spacing and/or log rotation
and/or rotated log storage (partition).
I have not looked at the feature yet, and if it allows "hooks"
that can be used to run shell snippets or what to adjust firewall,
but despite that, how likely is the first condition?
I would *guess* that the new feature is meant as / thought for
being one more tunable tool in the fight against malicious
players that adds "onto existing infrastructure", not as a fully
satisfying standalone does it all thing.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

On Tue, 25 Jun 2024, Joseph S. Testa II wrote:
>the way down to 6%!  Additionally, I noticed that the systemd-journal
You should test without that thing as well. It?s reportedly a
known bottleneck (someone on, I think, IRC said that regarding
a different problem some days ago, incidentally).

Just use a real syslogd (inetutils-syslogd is nice, for example,
and rsyslogd and syslog-ng both have fans), (importantly) without
message routing through systemd-journald.

bye,
//mirabilos
-- 
Infrastrukturexperte ? Qvest Digital AG
Am Dickobskreuz 10, D-53121 Bonn ? https://www.qvest-digital.com/
Telephon +49 228 54881-393 ? Fax: +49 228 54881-235
HRB AG Bonn 18196 ? USt-ID (VAT): DE274355441
Vorstand: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
Vorsitzender Aufsichtsrat: Peter N?then

I'd like to withdraw the last set of metrics I reported.  I couldn't
reproduce some of them, and I suspect I made a mistake during testing.

Being more careful this time, I set up another fully updated Ubuntu
24.04 VM with 4 vCPUs running openssh-SNAP-20240628.tar.gz with all
defaults unchanged.

When running using "ssh-audit.py --conn-rate-test=16 target_host", the
system idle time averaged over 60 seconds was 50%.  The
/var/log/auth.log file grew 73MB in this time (nearly 400,000 lines
were messages produced by the new PerSourcePenalties logging in
sshd.c:627).

Next, I modified the logging in sshd.c:627 to always use
SYSLOG_LEVEL_DEBUG1 instead of SYSLOG_LEVEL_INFO.  Re-running the above
test resulted in 73% average idle time and 8KB of log growth.

Lastly, from an m7i.2xlarge source EC2 instance in AWS, I targeted an
m7i.large instance using "ssh-audit --dheat=4:diffie-hellman-group18-
sha512:4 target_host".  In my original research article, this caused
the average idle time to drop to 0.01%; against openssh-SNAP-
20240628.tar.gz with the log level in sshd.c:627 changed to DEBUG1, the
idle time was observed to be 84%.

My conclusion is that the default user configuration of
PerSourcePenalties sufficiently stops the DHEat DoS.  However, the
logging implementation should be modified to prevent disk resource
exhaustion.  Aside from possibly changing the log level to VERBOSE (or
DEBUG1?),  perhaps the level can remain at INFO and message aggregation
can be added (e.g.: the Linux kernel sometimes logs a single line
followed by "(the above message was repeated 2168 times)").

   - Joe