On Mon, 2005-01-17 at 05:29, Brendan Gregg wrote:> I was careful not to suggest that these reads were actually a bug or a > problem; I haven''t read the source to SMC, I can only assume there is a > good reason for this behaviour.since lseek() isn''t turning up in your profile, i have to assume that these are sequential 1-byte reads. sequential 1-byte reads on plain files are.. evidence of lack of tuning. - Bill _______________________________________________ DTrace mailing list DTrace@opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
On Mon 17 Jan 2005 at 09:29PM, Brendan Gregg wrote:> G''Day Folks, > > A few of us were discussing today why SMC took so long to boot the first > time, so I pulled out DTrace and started looking. Pretty quickly we had a > good idea why (eg, several million 1 byte read()s), and as it made for a > nice demo of DTrace I just wrote a website for it. > > http://www.brendangregg.com/DTrace/smc.html > > I was careful not to suggest that these reads were actually a bug or a > problem; I haven''t read the source to SMC, I can only assume there is a > good reason for this behaviour. > > Why I''m writing is if anyone thinks I should be more careful about what I > say about SMC, please let me know. It''s a good demo of DTrace, I wouldn''t > want people to misinterpret it as highlighting an SMC problem that may > well not be a problem...Brendan, I''d call it a (performance) bug, and you should feel free to do so, as far as I''m concerned. You''re very generous to "assume there is a good reason". Remember that a lot of this code was written before we had DTrace; and also, it could be an interaction bug which is new. It would really help if you could enhance your analysis to show how much time these reads are actually chewing up (ex. aggregate on syscall by vtime spent in the syscall)-- that would give us some understanding of how much we''re paying. Could you also please use the jstack() action to figure out the java stack trace which is inducing these reads? That will strengthen the analysis as well. I did a little digging in the admininstall source base, and found that registry.ser is a file which contains a set of serialized java objects. I filed 6218290 smc seems to pound registry.ser on startup, wasting cycles On your behalf, and basically just pointed at your webpage (so if you could leave it up, that would be helpful!) -dp -- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp _______________________________________________ DTrace mailing list DTrace@opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Bill, On Mon, 17 Jan 2005, Bill Sommerfeld wrote:> On Mon, 2005-01-17 at 05:29, Brendan Gregg wrote: > > > I was careful not to suggest that these reads were actually a bug or a > > problem; I haven''t read the source to SMC, I can only assume there is a > > good reason for this behaviour. > > since lseek() isn''t turning up in your profile, i have to assume that these > are sequential 1-byte reads. sequential 1-byte reads on plain files are.. > evidence of lack of tuning.Surely 7 million 1 byte sequential reads can''t be wrong? ;-) Brendan _______________________________________________ DTrace mailing list DTrace@opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Dan, On Mon, 17 Jan 2005, Dan Price wrote:> On Mon 17 Jan 2005 at 09:29PM, Brendan Gregg wrote: > > G''Day Folks, > > > > A few of us were discussing today why SMC took so long to boot the first > > time, so I pulled out DTrace and started looking. Pretty quickly we had a > > good idea why (eg, several million 1 byte read()s), and as it made for a > > nice demo of DTrace I just wrote a website for it. > > > > http://www.brendangregg.com/DTrace/smc.html > > > > I was careful not to suggest that these reads were actually a bug or a > > problem; I haven''t read the source to SMC, I can only assume there is a > > good reason for this behaviour.[...]> Brendan, > > I''d call it a (performance) bug, and you should feel free to do so, as > far as I''m concerned. You''re very generous to "assume there is a good > reason". Remember that a lot of this code was written before we had > DTrace; and also, it could be an interaction bug which is new.Well, it''s a public website so I ought to assume the best. Anyone who knows performance will make up their own mind anyway. I imagine DTrace will indeed shine light on many new things. (I ought to find a database server to pick on instead).> It would really help if you could enhance your analysis to show how > much time these reads are actually chewing up (ex. aggregate on syscall > by vtime spent in the syscall)-- that would give us some understanding > of how much we''re paying.I ran into an ugly stumbling block before realising what you meant by vtime. I''m now using vtimestamps to measure syscall time on CPUs - it really helps! This is now done and on the site. Those reads are using 78% of the total syscall time (or 94 seconds).> Could you also please use the jstack() action to figure out the java > stack trace which is inducing these reads? That will strengthen the > analysis as well.Good idea, done.> I did a little digging in the admininstall source base, and found > that registry.ser is a file which contains a set of serialized java objects. > I filed > > 6218290 smc seems to pound registry.ser on startup, wasting cycles > > On your behalf, and basically just pointed at your webpage (so if you > could leave it up, that would be helpful!)No worries, Thanks Dan! :) Brendan _______________________________________________ DTrace mailing list DTrace@opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Folks,
I''ve just added to the troubleshooting examples with a DTrace look at
catman,
http://www.brendangregg.com/DTrace/lostcpu.html
I''m making a list of these at the bottom of my main dtrace website.
I picked catman as it creates loads of short lived processes that are
difficult to troubleshoot using traditional tools, yet it can still flog
the CPUs. (Imagine trying to spot it using ps -ef. :)
I hope to add a handful more examples in about a month.
cheers,
Brendan
[Sydney, Australia]
On Mon, 17 Jan 2005, Brendan Gregg wrote:
> G''Day Folks,
>
> A few of us were discussing today why SMC took so long to boot the first
> time, so I pulled out DTrace and started looking. Pretty quickly we had a
> good idea why (eg, several million 1 byte read()s), and as it made for a
> nice demo of DTrace I just wrote a website for it.
>
> http://www.brendangregg.com/DTrace/smc.html
>
[...]
_______________________________________________
DTrace mailing list
DTrace@opensolaris.org
https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Folks, A few of us were discussing today why SMC took so long to boot the first time, so I pulled out DTrace and started looking. Pretty quickly we had a good idea why (eg, several million 1 byte read()s), and as it made for a nice demo of DTrace I just wrote a website for it. http://www.brendangregg.com/DTrace/smc.html I was careful not to suggest that these reads were actually a bug or a problem; I haven''t read the source to SMC, I can only assume there is a good reason for this behaviour. Why I''m writing is if anyone thinks I should be more careful about what I say about SMC, please let me know. It''s a good demo of DTrace, I wouldn''t want people to misinterpret it as highlighting an SMC problem that may well not be a problem... thanks, Brendan [Sydney, Australia] _______________________________________________ DTrace mailing list DTrace@opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace