freebsd stable - Jan 2017 - Horrible zpool resilver perormance on local hast

I have a pair of machines running hast cross two pairs of
drives - i.e. 4 drives total, two in each box, and hence
two hast resources, cbert0 and cebrt1.

Its been running mysql fir a long time, but there only 6 gig
of data actually in use in the pool. I rebuilt the secondary
machine, and wanted to resync the hast devices, but that would involve
copying 450 gig for only 6 gig of actual data.

"easy" I though, "just rebuild each has device in the pool
so it only has the current data dirty"

seems reasonbale enough ?

so I disconnected one bit of the mirror, did a 'hast create' to
zero it, and reattached it to the mirror.

(note that the secondary side is entirely disconnected, this is local
to one machine).

what I am seeing is that the zpool is resilvering at a rate
of 170K/s - currently it says this:

        622M scanned out of 6.53G at 170K/s, 10h9m to go

But the hastctl status outout says this:

  status: degraded
  workerpid: 950
  dirty: 136363114496 (127GB)
  statistics:

I onyl expect dirty blocks when the ZFS has ritten something -0 127 gig
dirty seems excessive for 622 meg written!

anyone got any idea what might be happening ?

-pete.

Partially answering my own question here, as it occured to
me that the zpool is scattering writes across the disc in 4k blocks,
but hast has a minimum extent size of 2meg by default. Thats eems like
a likely culprit for the 'dirty blocks' multiplcation I am seeing.

I shrunk down the extent size to 16k to see if that helped, and I
now see a far smaller dirty block to written block ration, but
also the write speed has now shrunk down even more to around 50k/s :-(

Any advice on tuning this properly ?

-pete.