Scott Kammer
2007-Jan-12 05:06 UTC
[dtrace-discuss] Is this normal due to too much activity???
I tried running the vopstat dtrace script from the toolkit on a file system on one of my 25K''s. Big system and very busy and I encountered this. I was concerned about the errors a little ways down. Is this error just a result of the system being too busy??? I am going to try and use fop probes to check on certain problems we are having but I can''t get dtrace to output them consistently without errors. Any suggestions? VOP Physical IO Count fop_getpage 2 fop_read 2 fop_write 506 VOP Count Count fop_read 11 fop_write 191 fop_seek 212 fop_rwlock 343 fop_rwunlock 364 fop_getpage 1144 fop_dispose 1474 VOP Wall Time mSeconds fop_rwunlock 3 fop_seek 6 fop_rwlock 1423 fop_getpage 5418 fop_write 14356 fop_read 2362623329 fop_dispose 2362632231 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: 5935 dynamic variable drops with non-empty rinsing list dtrace: 8399 dynamic variable drops with non-empty dirty list dtrace: 4159 dynamic variable drops with non-empty rinsing list dtrace: 2659 dynamic variable drops with non-empty dirty list dtrace: 8892 dynamic variable drops with non-empty rinsing list dtrace: 2267 dynamic variable drops with non-empty dirty list dtrace: 6362 dynamic variable drops with non-empty rinsing list dtrace: 1623 dynamic variable drops with non-empty dirty list dtrace: 10736 dynamic variable drops with non-empty rinsing list dtrace: 12032 dynamic variable drops with non-empty dirty list dtrace: 8462 dynamic variable drops with non-empty rinsing list dtrace: 26140 dynamic variable drops with non-empty dirty list VOP Physical IO Count fop_getpage 2 fop_write 78 VOP Count Count fop_getattr 29 fop_write 34 fop_seek 41 fop_getpage 43 fop_rwlock 44 fop_rwunlock 56 fop_dispose 112 VOP Wall Time mSeconds fop_getattr 0 fop_seek 6 fop_rwlock 32 fop_write 413 fop_rwunlock 1450 fop_dispose 1624 fop_getpage 4491 dtrace: 68599 dynamic variable drops with non-empty rinsing list dtrace: 9457 dynamic variable drops with non-empty dirty list dtrace: 106985 dynamic variable drops with non-empty rinsing list dtrace: 21773 dynamic variable drops with non-empty dirty list dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 VOP Physical IO Count fop_seek 4 fop_rwunlock 16 fop_write 144 VOP Count Count fop_read 2 fop_getattr 9 fop_write 42 fop_seek 56 fop_rwlock 78 fop_rwunlock 86 fop_getpage 170 fop_dispose 303 VOP Wall Time mSeconds fop_getattr 0 fop_rwlock 8 fop_seek 63 fop_rwunlock 293 fop_write 2014 fop_getpage 3571 fop_read 2362633559 fop_dispose 2362634774 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 dtrace: 161812 dynamic variable drops with non-empty rinsing list dtrace: 30041 dynamic variable drops with non-empty dirty list dtrace: processing aborted: Abort due to systemic unresponsiveness This message posted from opensolaris.org
Please edit /kernel/drv/fasttrap.conf and double the values. It should alleviate the dynamic variable drops. regards benoit Scott Kammer wrote:> I tried running the vopstat dtrace script from the toolkit on a file system on one of my 25K''s. Big system and very busy and I encountered this. I was concerned about the errors a little ways down. Is this error just a result of the system being too busy??? I am going to try and use fop probes to check on certain problems we are having but I can''t get dtrace to output them consistently without errors. Any suggestions? > > VOP Physical IO Count > fop_getpage 2 > fop_read 2 > fop_write 506 > > VOP Count Count > fop_read 11 > fop_write 191 > fop_seek 212 > fop_rwlock 343 > fop_rwunlock 364 > fop_getpage 1144 > fop_dispose 1474 > > VOP Wall Time mSeconds > fop_rwunlock 3 > fop_seek 6 > fop_rwlock 1423 > fop_getpage 5418 > fop_write 14356 > fop_read 2362623329 > fop_dispose 2362632231 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: 5935 dynamic variable drops with non-empty rinsing list > dtrace: 8399 dynamic variable drops with non-empty dirty list > dtrace: 4159 dynamic variable drops with non-empty rinsing list > dtrace: 2659 dynamic variable drops with non-empty dirty list > dtrace: 8892 dynamic variable drops with non-empty rinsing list > dtrace: 2267 dynamic variable drops with non-empty dirty list > dtrace: 6362 dynamic variable drops with non-empty rinsing list > dtrace: 1623 dynamic variable drops with non-empty dirty list > dtrace: 10736 dynamic variable drops with non-empty rinsing list > dtrace: 12032 dynamic variable drops with non-empty dirty list > dtrace: 8462 dynamic variable drops with non-empty rinsing list > dtrace: 26140 dynamic variable drops with non-empty dirty list > VOP Physical IO Count > fop_getpage 2 > fop_write 78 > > VOP Count Count > fop_getattr 29 > fop_write 34 > fop_seek 41 > fop_getpage 43 > fop_rwlock 44 > fop_rwunlock 56 > fop_dispose 112 > > VOP Wall Time mSeconds > fop_getattr 0 > fop_seek 6 > fop_rwlock 32 > fop_write 413 > fop_rwunlock 1450 > fop_dispose 1624 > fop_getpage 4491 > dtrace: 68599 dynamic variable drops with non-empty rinsing list > dtrace: 9457 dynamic variable drops with non-empty dirty list > dtrace: 106985 dynamic variable drops with non-empty rinsing list > dtrace: 21773 dynamic variable drops with non-empty dirty list > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > VOP Physical IO Count > fop_seek 4 > fop_rwunlock 16 > fop_write 144 > > VOP Count Count > fop_read 2 > fop_getattr 9 > fop_write 42 > fop_seek 56 > fop_rwlock 78 > fop_rwunlock 86 > fop_getpage 170 > fop_dispose 303 > > VOP Wall Time mSeconds > fop_getattr 0 > fop_rwlock 8 > fop_seek 63 > fop_rwunlock 293 > fop_write 2014 > fop_getpage 3571 > fop_read 2362633559 > fop_dispose 2362634774 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: error on enabled probe ID 183 (ID 8889: io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > dtrace: 161812 dynamic variable drops with non-empty rinsing list > dtrace: 30041 dynamic variable drops with non-empty dirty list > dtrace: processing aborted: Abort due to systemic unresponsiveness > > > This message posted from opensolaris.org > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >
Brendan Gregg - Sun Microsystems
2007-Jan-12 06:32 UTC
[dtrace-discuss] Is this normal due to too much activity???
G''Day Scott, Sorry about that - the scripts for the DTraceToolkit have been written for moderate workloads, but some will need tweaking for very large workloads - such as your very busy F25K! To counter dynamic variable drops, increase the dynvarsize (the buffer size for dynamic variables). If you haven''t done this before, find the line in the script which reads, #pragma D option quiet and add beneath it, #pragma D option dynvarsize=64m to make that buffer 64m. Make the buffer bigger if drops persist, so long as you have the RAM to spare. This is a trick of the trade that has been documented at least here: http://blogs.sun.com/roller/resources/bmc/dtrace_tips.pdf On Thu, Jan 11, 2007 at 10:02:08PM -0800, benoit wrote:> Please edit /kernel/drv/fasttrap.conf and double the values. It should > alleviate the dynamic variable drops.no, no, please don''t do this. If you are ever tweaking /kernel/drv, then either there is a better way or there should be a better way... Brendan -- Brendan [CA, USA]> regards > benoit > Scott Kammer wrote: > >I tried running the vopstat dtrace script from the toolkit on a file > >system on one of my 25K''s. Big system and very busy and I encountered > >this. I was concerned about the errors a little ways down. Is this error > >just a result of the system being too busy??? I am going to try and use > >fop probes to check on certain problems we are having but I can''t get > >dtrace to output them consistently without errors. Any suggestions? > > > >VOP Physical IO Count > >fop_getpage 2 > >fop_read 2 > >fop_write 506 > > > >VOP Count Count > >fop_read 11 > >fop_write 191 > >fop_seek 212 > >fop_rwlock 343 > >fop_rwunlock 364 > >fop_getpage 1144 > >fop_dispose 1474 > > > >VOP Wall Time mSeconds > >fop_rwunlock 3 > >fop_seek 6 > >fop_rwlock 1423 > >fop_getpage 5418 > >fop_write 14356 > >fop_read 2362623329 > >fop_dispose 2362632231 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: 5935 dynamic variable drops with non-empty rinsing list > >dtrace: 8399 dynamic variable drops with non-empty dirty list > >dtrace: 4159 dynamic variable drops with non-empty rinsing list > >dtrace: 2659 dynamic variable drops with non-empty dirty list > >dtrace: 8892 dynamic variable drops with non-empty rinsing list > >dtrace: 2267 dynamic variable drops with non-empty dirty list > >dtrace: 6362 dynamic variable drops with non-empty rinsing list > >dtrace: 1623 dynamic variable drops with non-empty dirty list > >dtrace: 10736 dynamic variable drops with non-empty rinsing list > >dtrace: 12032 dynamic variable drops with non-empty dirty list > >dtrace: 8462 dynamic variable drops with non-empty rinsing list > >dtrace: 26140 dynamic variable drops with non-empty dirty list > >VOP Physical IO Count > >fop_getpage 2 > >fop_write 78 > > > >VOP Count Count > >fop_getattr 29 > >fop_write 34 > >fop_seek 41 > >fop_getpage 43 > >fop_rwlock 44 > >fop_rwunlock 56 > >fop_dispose 112 > > > >VOP Wall Time mSeconds > >fop_getattr 0 > >fop_seek 6 > >fop_rwlock 32 > >fop_write 413 > >fop_rwunlock 1450 > >fop_dispose 1624 > >fop_getpage 4491 > >dtrace: 68599 dynamic variable drops with non-empty rinsing list > >dtrace: 9457 dynamic variable drops with non-empty dirty list > >dtrace: 106985 dynamic variable drops with non-empty rinsing list > >dtrace: 21773 dynamic variable drops with non-empty dirty list > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >VOP Physical IO Count > >fop_seek 4 > >fop_rwunlock 16 > >fop_write 144 > > > >VOP Count Count > >fop_read 2 > >fop_getattr 9 > >fop_write 42 > >fop_seek 56 > >fop_rwlock 78 > >fop_rwunlock 86 > >fop_getpage 170 > >fop_dispose 303 > > > >VOP Wall Time mSeconds > >fop_getattr 0 > >fop_rwlock 8 > >fop_seek 63 > >fop_rwunlock 293 > >fop_write 2014 > >fop_getpage 3571 > >fop_read 2362633559 > >fop_dispose 2362634774 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: error on enabled probe ID 183 (ID 8889: > >io:genunix:bdev_strategy:start): invalid address (0x0) in action #2 > >dtrace: 161812 dynamic variable drops with non-empty rinsing list > >dtrace: 30041 dynamic variable drops with non-empty dirty list > >dtrace: processing aborted: Abort due to systemic unresponsiveness > > > > > >This message posted from opensolaris.org > >_______________________________________________ > >dtrace-discuss mailing list > >dtrace-discuss at opensolaris.org > > > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org