freebsd stable - Nov 2016 - Help! two machines ran out of swap and corrupted their zpools!

Pete French wrote:
> So, I am off sick and my colleagues decided to load test our set of five
> servers excesively. All ran out of swap. So far so irritating, but whats
has
> happened is that twoof them now will not boot, as it appears the ZFS pool
> they are booting from has become corrupted.
>
> One starts to boot, then crases importing the root pool. The other doenst
> even get that far with gptzfsboot saying it can't find the pool to boot
from!
>
> Now I can recover these, but I am a bit worried, that it got like this at
> all, as I havent ever seen ZFS corrupt a pool like this. Anyone got any
insights,
> or suggstions as to how to stop it happening again ?
>
> We are swapping to a separate partition, not to the pool by theway.
Good.

Try downloading live disc or mfsBSD and importing pool r/o from there.

zpool import -N -O readonly=on -f -R /mnt/somezpoool

If that doesn't help try:

zpool import -N -O readonly=on -f -R /mnt/somezpoool -Fn

Drop us a line of your configuration and used ZFS features. Like dedup, 
snapshots, external l2 logs and caches.

-- 
Sphinx of black quartz judge my vow.

> zpool import -N -O readonly=on -f -R /mnt/somezpoool
>
> If that doesn't help try:
>
> zpool import -N -O readonly=on -f -R /mnt/somezpoool -Fn
I got someone to do this (am still having toruble finding time
as am supposed to be off sick) and it causes instant kernel panic
on trying to import the pool. Same as it does on boot.
> Drop us a line of your configuration and used ZFS features. Like dedup, 
> snapshots, external l2 logs and caches.
10.3-STABLE r303832 from start of August. One simple pool, two
drives mirrored, GPT formatted drives. No dedup, no snapshots,
no external logs or caches. We have lz4 comression enabled on the
filesystem, but apart from that its an utterly bog-standard setup.

Am an leaning towards faulty hardware now actually... seems most likely...

-pete.