freebsd stable - Nov 2019 - ZFS deadlock

Hi!

I have RAIDZ1 with five GELI-encrypted SSDs da[2-6].eli (non-boot pool).

I've exported the pool, destroyed da2.eli then successfully imported pool
back in degraded state.
Then I've mounted some file systems successfully but zfs mount for next one
hung on [tx->tx_sync_done_cv]
for 4400 seconds and counting.

# procstat -kk -L 55464
  PID    TID COMM                TDNAME              KSTACK
55464 102422 zfs                 -                   mi_switch+0xeb
sleepq_wait+0x2c _cv_wait+0x16e txg_wait_synced+0xa5 dmu_tx_assign+0x48
zfs_rmnode+0x122 zfs_freebsd_reclaim+0x4e VOP_RECLAIM_APV+0x80 vgonel+0x213
vrecycle+0x46 zfs_freebsd_inactive+0xd VOP_INACTIVE_APV+0x80 vinactive+0xf0
vputx+0x2c3 zfs_unlinked_drain+0x1b8 zfsvfs_setup+0x5e zfs_mount+0x5f5
vfs_domount+0x573

It looks like deadlock for me.

What can I do to resolve this? FreeBSD 11.3-STABLE/amd64 r354667.

30.11.2019 15:48, Eugene Grosbein wrote:
> Hi!
> 
> I have RAIDZ1 with five GELI-encrypted SSDs da[2-6].eli (non-boot pool).
> 
> I've exported the pool, destroyed da2.eli then successfully imported
pool back in degraded state.
> Then I've mounted some file systems successfully but zfs mount for next
one hung on [tx->tx_sync_done_cv]
> for 4400 seconds and counting.
> 
> # procstat -kk -L 55464
>   PID    TID COMM                TDNAME              KSTACK
> 55464 102422 zfs                 -                   mi_switch+0xeb
sleepq_wait+0x2c _cv_wait+0x16e txg_wait_synced+0xa5 dmu_tx_assign+0x48
zfs_rmnode+0x122 zfs_freebsd_reclaim+0x4e VOP_RECLAIM_APV+0x80 vgonel+0x213
vrecycle+0x46 zfs_freebsd_inactive+0xd VOP_INACTIVE_APV+0x80 vinactive+0xf0
vputx+0x2c3 zfs_unlinked_drain+0x1b8 zfsvfs_setup+0x5e zfs_mount+0x5f5
vfs_domount+0x573
> 
> It looks like deadlock for me.
> 
> What can I do to resolve this? FreeBSD 11.3-STABLE/amd64 r354667.
"zfs mount" just completed successfully after trim -f /dev/da2 (3.5TB)
finished in 76 minutes.