freebsd stable - Apr 2020 - ZFS server has gone crazy slow

I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirror (of
two GPT disks) as it?s disk.  It?s many years old, and has only been misbehaving
like this for a day or so.  I?m trying to figure out what?s wrong.

I confirmed that internet connectivity isn?t the problem, and a reboot didn?t
fix it.  (The reboot took 10-15 minutes to finish going multi-user, starting
daemons, due to the underlying problem described below.)

Truss?ing a very basic command (date), I can see that close() and exit() calls
are taking 1-2 seconds.  All of the files being opened are on ZFS, but I don?t
know if that?s for sure related.  Similarly, using shell builtin ?echo foo?
always is immediate, but ?/bin/echo? sometimes works quickly, but sometimes the
close() on /var/run/ld-elf.so.hints takes 3-5 seconds.

I _think_ this is a filesystem problem.  It?s very hard to diagnose because
logging in, and doing anything, takes many seconds per command.  zpool status
shows my mirror as online, so I?m not sure where I should check.

I?d appreciate any help!  Thanks much?

           - Chris

12.04.2020 0:36, Chris Ross wrote:
> I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirror
(of two GPT disks) as it?s disk.  It?s many years old, and has only been
misbehaving like this for a day or so.  I?m trying to figure out what?s wrong.
> 
> I confirmed that internet connectivity isn?t the problem, and a reboot
didn?t fix it.  (The reboot took 10-15 minutes to finish going multi-user,
starting daemons, due to the underlying problem described below.)
> 
> Truss?ing a very basic command (date), I can see that close() and exit()
calls are taking 1-2 seconds.  All of the files being opened are on ZFS, but I
don?t know if that?s for sure related.  Similarly, using shell builtin ?echo
foo? always is immediate, but ?/bin/echo? sometimes works quickly, but sometimes
the close() on /var/run/ld-elf.so.hints takes 3-5 seconds.
> 
> I _think_ this is a filesystem problem.  It?s very hard to diagnose because
logging in, and doing anything, takes many seconds per command.  zpool status
shows my mirror as online, so I?m not sure where I should check.
> 
> I?d appreciate any help!  Thanks much?
First of all you should check if any of your ZFS pools is low on space.