freebsd stable - Aug 2017 - a strange and terrible saga of the cursed iSCSI ZFS SAN

On Aug 5, 2017 10:09 AM, "Eugene M. Zheganin" <emz at
norma.perm.ru> wrote:


And I want to also ask - what happens when the system's memory isn't
enough
for deduplication - does it crash, or does the problem of mounting the pool
appear, like some articles mention ?


Can't really help with the other issues, but can speak to this one.

"Insufficient" RAM for dedupe won't be an issue during normal
operation,
reading and writing to the pool. Performance will slow down over time as
the DDT increases in size taking up ARC/L2ARC space, possibly even spilling
over into raw disk reads.

Where the issues crop up is during snapshot deletion and filesystem
destruction. Those operations can run the system out of usable kmem and
lock things up. (Deletions cause a lot of churn in the DDT.) On reboot, the
pool import process will continue the deletion process, which will run the
system out of kmen locking it up. Rinse and repeat for a week until the
deletion process completes. L2ARC helps, but not nearly as much as adding
more RAM!

Device replacement will also be an issue as the resilver process will be
extremely slow (at least, it is for us with a very full, very fragmented
pool; equally full pools without d we dedupe resilver very quickly). And
don't try to replace two drives in separate vdevs unless you want to
increase the resilver time by a huge factor.

This is from experience with a 24-drive system with only 48 GB of RAM, and
a 90-disk system with 128 GB of RAM. Both with dedupe enabled. Both running
at about 90% full, very fragmented as they create and delete snapshots of
all filesystems on a daily basis. They've been running file about 5 years
now, possibly longer.

Cheers,
Freddie