Hi Everyone, I have a couple of systems running opensolaris b118, one of which sends hourly snapshots to the other. This has been working well, however as of today, the receiving zfs process has started running extremely slowly, and is running at 100% CPU on one core, completely in kernel mode. A little bit of exploration with lockstat and dtrace seems to imply that the issue is around the "dbuf_free_range" function - or at least, that''s what it looks like to my inexperienced eye! The system is very unresponsive while this problem is occurring, with frequent multi-second delays between my typing into an ssh session and getting a response. Has anyone seen this before, or know if there''s anything I can do about it? Lockstat says: Adaptive mutex hold: 52299 events in 1.694 seconds (30873 events/sec) ------------------------------------------------------------------------------- Count indv cuml rcnt nsec Lock Caller 50 87% 87% 0.00 23410134 0xffffff0273d6dcb8 dbuf_free_range+0x268 nsec ------ Time Distribution ------ count 33554432 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 50 ------------------------------------------------------------------------------- Count indv cuml rcnt nsec Lock Caller 53 0% 87% 0.00 117124 0xffffff025f404140 txg_rele_to_quiesce+0x18 nsec ------ Time Distribution ------ count 131072 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 53 ------------------------------------------------------------------------------- Count indv cuml rcnt nsec Lock Caller 2536 0% 88% 0.00 2194 0xffffff024f09bd40 kmem_cache_alloc+0x84 nsec ------ Time Distribution ------ count 2048 |@@@@@@@@@@@@@ 1121 4096 |@@@@@@@@@@@@@@@@ 1383 8192 | 0 16384 | 32 ------------------------------------------------------------------------------- Count indv cuml rcnt nsec Lock Caller 415 0% 88% 0.00 7968 0xffffff025120ce00 anon_resvmem+0xb4 [snip] I modified the "procsystime" script from the dtrace toolkit to trace fbt:zfs::entry/return rather than syscalls, and I get an output like this (approximately a 5sec sample): Function nsecs l2arc_write_eligible 46155456 spa_last_synced_txg 49211711 zio_inherit_child_errors 50039132 zil_commit 50137305 buf_hash 50839858 dbuf_rele 56388970 zio_wait_for_children 67533094 dbuf_update_data 70302317 dsl_dir_space_towrite 72179496 parent_delta 77685327 dbuf_hash 79075532 free_range_compar 82479196 dbuf_free_range 2715449416 TOTAL: 4156078240 Thanks, Tristan _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Tristan Ball wrote:> Hi Everyone, > > I have a couple of systems running opensolaris b118, one of which sends > hourly snapshots to the other. This has been working well, however as of > today, the receiving zfs process has started running extremely slowly, > and is running at 100% CPU on one core, completely in kernel mode. A > little bit of exploration with lockstat and dtrace seems to imply that > the issue is around the "dbuf_free_range" function - or at least, that''s > what it looks like to my inexperienced eye!This is probably RFE 6812603 "zfs send can aggregate free records", which is currently being worked on. --matt
OK, Thanks for that.
From reading the RFE, it sound''s like having a faster machine on the
receive side will be enough to alleviate the problem in the short term?
The hardware I''m using at the moment is quite old, and not particularly
fast - although this is the first out & out performance limitation
I''ve
had with using it as a opensolaris storage system.
Regards,
Tristan.
Matthew Ahrens wrote:> Tristan Ball wrote:
>> Hi Everyone,
>>
>> I have a couple of systems running opensolaris b118, one of which
>> sends hourly snapshots to the other. This has been working well,
>> however as of today, the receiving zfs process has started running
>> extremely slowly, and is running at 100% CPU on one core, completely
>> in kernel mode. A little bit of exploration with lockstat and dtrace
>> seems to imply that the issue is around the "dbuf_free_range"
>> function - or at least, that''s what it looks like to my
inexperienced
>> eye!
>
> This is probably RFE 6812603 "zfs send can aggregate free
records",
> which is currently being worked on.
>
> --matt
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
Tristan Ball wrote:> OK, Thanks for that. > > From reading the RFE, it sound''s like having a faster machine on the > receive side will be enough to alleviate the problem in the short term?That''s correct. --matt