search for: dmu_objset_sync

...oup_alloc+0x1e0 zfs`metaslab_alloc_dva+0x114 zfs`metaslab_alloc+0x2c zfs`zio_dva_allocate+0x4c zfs`zio_write_compress+0x1ec zfs`arc_write+0xe4 zfs`dbuf_sync+0x6c0 zfs`dnode_sync+0x35c zfs`dmu_objset_sync_dnodes+0x6c zfs`dmu_objset_sync+0x54 zfs`dsl_dataset_sync+0xc zfs`dsl_pool_sync+0x64 zfs`spa_sync+0x1b0 zfs`txg_sync_thread+0x134 unix`thread_start+0x4 82 genunix`avl_walk+0xa0...

Hi. System is snv_56 sun4u sparc SUNW,Sun-Fire-V440, zil_disable=1 We see many operation on nfs clients to that server really slow (like 90 seconds for unlink()). It''s not a problem with network, there''s also plenty oc CPU available. Storage isn''t saturated either. First strange thing - normally on that server nfsd has about 1500-2500 number of threads. I did

...tage+0xb3(ffffff008f722790) zio_write_compress+0x239(ffffff008f722790) zio_next_stage+0xb3(ffffff008f722790) zio_wait_for_children+0x5d(ffffff008f722790, 1, ffffff008f7229e0) zio_wait_children_ready+0x20(ffffff008f722790) zio_next_stage_async+0xbb(ffffff008f722790) zio_nowait+0x11(ffffff008f722790) dmu_objset_sync+0x196(ffffff008e4e5000, ffffff008f722a10, ffffff008f260a80) dsl_dataset_sync+0x5d(ffffff008df47e00, ffffff008f722a10, ffffff008f260a80) dsl_pool_sync+0xb5(ffffff00882fb800, 3fd0f1) spa_sync+0x1c5(ffffff008f2d1a80, 3fd0f1) txg_sync_thread+0x19a(ffffff00882fb800) thread_start+8() And here is the p...

...) 87a61bd0 zfs:zio_write_compress+2c8 (88e56dc0) 87a61bec zfs:zio_next_stage+aa (88e56dc0) 87a61c0c zfs:zio_wait_for_children+46 (88e56dc0, 1, 88e56f) 87a61c20 zfs:zio_wait_children_ready+18 (88e56dc0) 87a61c34 zfs:zio_next_stage_async+ac (88e56dc0) 87a61c48 zfs:zio_nowait+e (88e56dc0) 87a61c94 zfs:dmu_objset_sync+184 (85fe96c0, 88757ae0,) 87a61cbc zfs:dsl_dataset_sync+40 (813ad000, 88757ae0,) 87a61d0c zfs:dsl_pool_sync+a3 (8291c0c0, 286de2, 0) 87a61d6c zfs:spa_sync+1fc (82a7b900, 286de2, 0) 87a61dc8 zfs:txg_sync_thread+1df (8291c0c0, 0) 87a61dd8 unix:thread_start+8 () on second reboot it also Oct 11 18:17...

...oup_alloc+0x1b0 zfs`metaslab_alloc_dva+0x10c zfs`metaslab_alloc+0x2c zfs`zio_dva_allocate+0x50 zfs`zio_write_compress+0x1e4 zfs`arc_write+0xbc zfs`dbuf_sync+0x6b0 zfs`dnode_sync+0x300 zfs`dmu_objset_sync_dnodes+0x68 zfs`dmu_objset_sync+0x50 zfs`dsl_dataset_sync+0xc zfs`dsl_pool_sync+0x60 zfs`spa_sync+0xe0 zfs`txg_sync_thread+0x130 unix`thread_start+0x4 316769 genunix`avl_find+0x38...

...dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0x9bd dmu_objset_sync() at dmu_objset_sync+0x120 dsl_pool_sync() at dsl_pool_sync+0x72 spa_sync() at spa_sync+0x2f3 txg_sync_thread() at txg_sync_thread+0x2cd Do you have any ideas how to fix it? Kris has a way to reproduce it in his environment and I''m sure he could try a patch, if you could provide one. --...

...dva+0xab() metaslab_alloc+0x51() zio_dva_allocate+0x3f() zio_next_stage+0x72() zio_checksum_generate+0x5f() zio_next_stage+0x72() zio_write_compress+0x136() zio_next_stage+0x72() zio_wait_for_children+0x49() zio_wait_children_ready+0x15() zio_next_stage_async+0xae() zio_wait+0x2d() arc_write+0xcc() dmu_objset_sync+0x141() dsl_dataset_sync+0x23() dsl_pool_sync+0x7b() spa_sync+0x116() txg_sync_thread+0x115() thread_start+8() It appears ZFS is still able to read the labels from the drive: $ zdb -lv /dev/rdsk/c3t50002AC00039040Bd0p0 -------------------------------------------- LABEL 0 ------------------------...

...einstein genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 00000600033cb9e8 0000000000000003 00000600033cba48 Jun 16 14:44:42 einstein %l4-7: 0000060002c66400 00000600033cb9eb 000000000000000c 00000600033cba40 Jun 16 14:44:42 einstein genunix: [ID 723222 kern.notice] 000002a101553620 zfs:dmu_objset_sync_dnodes+6c (60002e02800, 60002e02940, 60009dd4dc0, 600033cb9 90, 0, 0) Jun 16 14:44:42 einstein genunix: [ID 179002 kern.notice] %l0-3: 0000060000fb8500 00000000000a4653 0000060002ee00f8 0000000000000001 Jun 16 14:44:42 einstein %l4-7: 0000000000000000 0000000070190000 0000000000000007 000006000...

...zfs:dbuf_sync_leaf+1ac () Sep 13 14:13:22 cypress genunix: [ID 655072 kern.notice] ffffff000840d950 zfs:dbuf_sync_list+51 () Sep 13 14:13:22 cypress genunix: [ID 655072 kern.notice] ffffff000840d9c0 zfs:dnode_sync+214 () Sep 13 14:13:22 cypress genunix: [ID 655072 kern.notice] ffffff000840da00 zfs:dmu_objset_sync_dnodes+55 () Sep 13 14:13:22 cypress genunix: [ID 655072 kern.notice] ffffff000840da80 zfs:dmu_objset_sync+13d () Sep 13 14:13:22 cypress genunix: [ID 655072 kern.notice] ffffff000840dad0 zfs:dsl_dataset_sync+5d () Sep 13 14:13:22 cypress genunix: [ID 655072 kern.notice] ffffff000840db40 zfs:dsl_po...

We are using MySQL, and love the idea of using zfs for this. We are used to using Direct I/O to bypass file system caching (let the DB do this). Does this exist for zfs? This message posted from opensolaris.org

...0 zfs:zio_wait_children_ready+18 (e3c0a580) d2c8ab04 zfs:zio_next_stage_async+ac (e3c0a580, 0, f8dbfe) d2c8ab1c zfs:zio_nowait+e (e3c0a580) d2c8ab3c zfs:arc_write+7b (e44c9780, d895e8c0,) d2c8abec zfs:dbuf_sync+5f3 (dbd6ef00, e44c9780,) d2c8ac4c zfs:dnode_sync+33a (d34fbb30, 1, e44c97) d2c8ac80 zfs:dmu_objset_sync_dnodes+7e (d2380240, d23802fc,) d2c8acd0 zfs:dmu_objset_sync+5d (d2380240, e96f1e80) d2c8ad1c zfs:dsl_pool_sync+121 (d244a180, 15e234, 0) d2c8ad6c zfs:spa_sync+10a (d895e8c0, 15e234, 0) d2c8adc8 zfs:txg_sync_thread+1df (d244a180, 0) d2c8add8 unix:thread_start+8 () I now have a chicken and egg prob...

...s:zio_next_stage_async+bb () ffffff000fd0a560 zfs:zio_nowait+11 () ffffff000fd0a870 zfs:dbuf_sync_leaf+1ac () ffffff000fd0a8b0 zfs:dbuf_sync_list+51 () ffffff000fd0a900 zfs:dbuf_sync_indirect+cd () ffffff000fd0a940 zfs:dbuf_sync_list+5e () ffffff000fd0a9b0 zfs:dnode_sync+23b () ffffff000fd0a9f0 zfs:dmu_objset_sync_dnodes+55 () ffffff000fd0aa70 zfs:dmu_objset_sync+13d () ffffff000fd0aac0 zfs:dsl_dataset_sync+5d () ffffff000fd0ab30 zfs:dsl_pool_sync+b5 () ffffff000fd0abd0 zfs:spa_sync+208 () ffffff000fd0ac60 zfs:txg_sync_thread+19a () ffffff000fd0ac70 unix:thread_start+8 () syncing file systems... 1 1 done er...

I just had a quick play with gzip compression on a filesystem and the result was the machine grinding to a halt while copying some large (.wav) files to it from another filesystem in the same pool. The system became very unresponsive, taking several seconds to echo keystrokes. The box is a maxed out AMD QuadFX, so it should have plenty of grunt for this. Comments? Ian