dtrace@opensolaris.org''s email at 11/26/2004 5:26 PM, said:> G''Day Jarod, > > On Fri, 26 Nov 2004 dtrace@opensolaris.org wrote: > >>Howdy Brendan. >> >>dtrace@opensolaris.org''s email at 11/26/2004 6:13 AM, said: >> > > [...] > >>I have given presentations to well over a thousand folks so far. Ain''t it just tons of fun? > > > It''s lots of fun, and great to hear it''s reaching so many people. A > thousand is a quite a lot by our standards! (Eg, I think Sun only has > around 400 staff Australia wide).The vast majority (85% or so) are all customers with the biggest chunk of those being financial services types. So, actually, we are getting through to a ton of customers.>>Jarod >> >>[ Texas ] ;) > > > Wow - I''ve always thought you were from Sillicon Valley - how useful was > that! :)Nope. I am a life-long Texan. Born in West Texas and lived in Texas my whole life. On a completely unrelated side note, does anyone else think it is stupid that the mail list (see below) is DTrace@www.opensolaris.com. What the hell is up with the www? Even if the "From" is screwed up, at least it doesn''t have the www. Thanks, Jarod> > Brendan > > [Sydney, Australia] > > > > _______________________________________________ > DTrace mailing list > DTrace@www.opensolaris.org > https://www.opensolaris.org/mailman/listinfo/dtrace >_______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
Hello Brendan, dtrace@www.opensolaris.org wrote:>From: Brendan Gregg <brendan.gregg@tpg.com.au> > >On Sat, 27 Nov 2004 dtrace@opensolaris.org wrote: > > > >>>On a completely unrelated side note, does anyone else think it is >>>stupid that the mail list (see below) is DTrace@www.opensolaris.com. >>>What the hell is up with the www? Even if the "From" is screwed up, at >>>least it doesn''t have the www. >>> >>> >>> >>I''d second that and this coming from the people that make the ''net work ;-) >> >>Nathan >> >> > > >Line 1 of this email is a hand typed From line, I thought it might be an >amusing thing to try. > >And I have no idea which "Nathan" that is! The mail header timestamp put >you in Europe.. :) > >Brendan > >[Sydney, Australia] > >Sorry about that, I assumed that my name would be in the From header; A stupid assumption given the topic of conversation (It''s Saturday morning here, that''s my excuse and I am sticking with it). My mail server is in the UK, but I am (sort of) just down the road from you in Wagga Wagga, NSW. Nathan Dietsch _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
dtrace@opensolaris.org wrote:> dtrace@opensolaris.org wrote: > >> Hello Brendan, >> >> dtrace@www.opensolaris.org wrote: >> >>> From: Brendan Gregg <brendan.gregg@tpg.com.au> >>> >>> On Sat, 27 Nov 2004 dtrace@opensolaris.org wrote: >>> >>> >>> >>>>> On a completely unrelated side note, does anyone else think it is >>>>> stupid that the mail list (see below) is DTrace@www.opensolaris.com. >>>>> What the hell is up with the www? Even if the "From" is screwed >>>>> up, at >>>>> least it doesn''t have the www. >>>>> >>>>> >>>> >>>> >>>> I''d second that and this coming from the people that make the ''net >>>> work ;-) >>>> >>>> Nathan >>>> >>> >>> >>> >>> >>> Line 1 of this email is a hand typed From line, I thought it might >>> be an >>> amusing thing to try. >>> >>> And I have no idea which "Nathan" that is! The mail header timestamp >>> put >>> you in Europe.. :) >>> >>> Brendan >>> >>> [Sydney, Australia] >>> >>> >> Sorry about that, I assumed that my name would be in the From header; >> A stupid assumption given the topic of conversation (It''s Saturday >> morning here, that''s my excuse and I am sticking with it). >> >> My mail server is in the UK, but I am (sort of) just down the road >> from you in Wagga Wagga, NSW. >> >> Nathan Dietsch >> _______________________________________________ >> DTrace mailing list >> DTrace@www.opensolaris.org >> https://www.opensolaris.org/mailman/listinfo/dtrace > > > > I sent an email to Derek after the post I made last week about this. > Hopefully, it will get fixed shortly.Sorry for the late response folks, I was out of the office for a while. This issue is fixed. Derek> > alan. >-- Derek Cicero Program Manager Solaris Kernel Group, Software Division p: 650.786.6128 _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
On Sat, 27 Nov 2004 dtrace@opensolaris.org (Nathan) wrote: [...]> >2. The dissapearance of University Unix servers. > > I''ve met Comp Sci graduates who have never used Unix/Linux, ever. > > Now they have landed a job as a Solaris admin. Gone are the days where > > *everyone* at Uni would use Unix - the days of pine, elm, tin, finger, > > talk, screen, MUDs, WYSE terminals, Archie, Veronica, etc, etc, ... > > <RANT> > > Who the hell thought it would be a good idea to get rid of UNIX from > Universities?Microsoft? [...]> >3. The appearence of 6 month become-an-IT-professional training courses. > > Their marketing campaigns go along the lines of - in just 6 easy months > > you too will be a highly paid IT professional. > > > This just plain sh*ts me. The industry itself is suffering at the hands > of these so called "professionals".Especially when they join a site as a Solaris admin, stuff it all up, and then blame Solaris. Not so good for Sun''s reputation. [...]> This is compounded by many developer''s opinions of system > administrators. Many developers believe that system administrators > simply get in the way; > However, there are good developers and there are good sys-admins. > Conversely, there are bad developers and bad sys-admins. To be good at > either of these two fields, you need an appreciation of the other.Bad sysadmins *do* get in the way of developers ;)> >Yes, I figured iosnoop.d would have to be the most obvious example to > >deliver to sysadmins - disk I/O by process has always bugged me in the > >past with Unix (next on my hit list is networking by process). But still, > >some sysadmins don''t know of the existance of the iostat command - let > >alone understand why more info would be useful. This go back to the first > >point you made! > > > >[...] > > > iosnoop is really cool. I just wish that I had come across the previous > versions of it earlier. Nice work.Gee, thanks!> >>What we also need to do is to layer graphical tools on top of DTrace. As > >>much > >>as some of us might not like it, people like graphical interfaces and > >>are not > >>happy cranking out scripts. > >> > For those of us who are happy cranking out scripts, I found pumping the > *stat family''s output into gnuplot quite helpful for creating a > graphical representation. > I used it once to monitor the I/O used in a number of batch jobs once > using vxstat. Little did I know that my manager two levels above me was > watching it too. > While I didn''t appreciate the bollocking I got when it broke while I was > trying to extend it, it was cool to know that the higher-ups were > actually using the tool that I had built and the graphs really helped > with that. I wish I had Solaris 10 and DTrace at the time of that contract.Managers love pretty pictures, red = bad, green = good! In the long run this may add to the misconception that IT is easy. GUI''ing DTrace for long term monitoring sounds good, but I''d want to study the performance suggestions from the answerbook in detail. Should DTrace be permanently be running and dumping to a database, or should it be kicked off every so often like sar? As disk I/O is measured in milliseconds I imagine the impact will be acceptable, for disk I/O anyway. I guess this depends on what is to be measured... [...]> > > I would suggest gnuplot for shell scripts. > > I apologise for the somewhat off-topic nature of my post, but my rant > was unstoppable once you mentioned the 6 Month IT professionals.hahaha! sounds like you may have met a few ;)> Regards, > > Nathan Dietsch > [Wagga Wagga, Australia]Fairdinkum - another Aussie! :) Brendan [Sydney, Australia] _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Jarod, On Fri, 26 Nov 2004 dtrace@opensolaris.org wrote:> > Howdy Brendan. > > dtrace@opensolaris.org''s email at 11/26/2004 6:13 AM, said: >[...]> I have given presentations to well over a thousand folks so far. Ain''t it just tons of fun?It''s lots of fun, and great to hear it''s reaching so many people. A thousand is a quite a lot by our standards! (Eg, I think Sun only has around 400 staff Australia wide). [...]> > problems that DTrace solves well, then it''s obvious that it''s amazing; if > > you haven''t, then it''s not so obvious... I don''t quite know an answer to > > this problem... > > Ask them what their number one problem is. Then show them how easily it can be solved with > DTrace. That is how I do it. Suddenly, when they see that their pain can be easily solved they > "get it".Yep, that sounds like the best approach, especially with developers.> > 2. It seems fewer and fewer sysadmins these days know C. Many take a look > > and say they''ll never program in DTrace, all they want is examples to use. > > You don''t really have to know C. I plan to take one day this weekend and teach my wife > (non-technical physical therapist type) D. I figure she''ll know the *language* in a couple of > hours. Now how to apply it, that is a different question and takes months. It just pisses me > off when people (including some Sun employees) complain about the size of the user guide and > how they don''t have time to learn a new language.I agree - it''s a fairly easy language and I''m not impressed when people whinge about learning new languages like this (they can whinge about learning languages like VB, C# or INTERCAL - that would be justified ;) [...]> Jarod > > [ Texas ] ;)Wow - I''ve always thought you were from Sillicon Valley - how useful was that! :) Brendan [Sydney, Australia] _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
Funny that you should bring this up Brendan. The dtrace folks have done a gathering of scripts (from across the company) to go into /usr/demo/dtrace. I''m not sure if this made build 72 for the next Express, but you should certainly be able to find them under the source tree under usr/src/cmd/dtrace/demo. How''s that for a start? alan. dtrace@opensolaris.org wrote:>G''Day Folks, > >Well, I''ve been demo''ing DTrace to other sysadmins again which have led to >the same conclusions - you would know these already, this might just add >weight to the argument. > > >I wrote some DTrace scripts on http://www.brendangregg.com/dtrace.html >(and more to come as I get the chance), which I''ve shown to many sysadmins >to get them enthusiastic about DTrace. It went along the lines of - see >the sort of things you can do, and with a comfy C-like language the sky''s >the limit. > >There''s been two problems with this: > >1. Some don''t have the background to understand why DTrace is so >fantastic. Even with prompting like "how do we snoop live exec()''s with >all their details?", don''t trigger the expected discussions on truss, >process accounting and BSM auditing; instead this gets blank looks and >shrugs as if to say "why would that be useful?". I find explaining DTrace >is like telling a joke - some get it and laugh but others don''t, and >having to explain the punchline ruins the impact. (Not that I''m saying >DTrace is a joke, of course! ;).. If you''ve rolled around in the sort of >problems that DTrace solves well, then it''s obvious that it''s amazing; if >you haven''t, then it''s not so obvious... I don''t quite know an answer to >this problem... > >2. It seems fewer and fewer sysadmins these days know C. Many take a look >and say they''ll never program in DTrace, all they want is examples to use. > > >So this brings me to a point (I had a point?), > >Customers want DTrace examples, I know you have some in the answerbook - >they ought to be copied to /usr/lib/dtrace. I''d write a list of ideas of >what I''d like to see in /usr/lib/dtrace - but I guess my website could >serve as a suggestion list of ideas. > >Picture /opt/RICHPse/examples/* from the SE Toolkit. It had 80 examples, >and is just what many people want. It''s suprising to see how uncomfortable >people seem to be to cut-n-paste from the answerbook - I guess people >might find DTrace is a little intimidating. > >... > >It seems Sun has a habit of creating wonderfly powerful solutions that >customers end up not using. For example, Solaris 8''s Sun::Solaris::Kstat. >How cool was that?? A Perl library that made it dead simple to pluck out >kernel stats on loads of interesting topics. How many customers used it? >Try google''ing for Sun::Solaris::Kstat - not many hits (and I''m quite a >few of them). > > >I think DTrace is way cool and deserves every chance at mega success - >a /usr/lib/dtrace of examples can''t hurt. > > >Brendan Gregg > >[Sydney, Australia] > > >_______________________________________________ >DTrace mailing list >DTrace@www.opensolaris.org >https://www.opensolaris.org/mailman/listinfo/dtrace > >-- Alan Hargreaves Senior Technical Support Specialist Phone : +61 2 9844 5379 Kernel/VOSJEC/Performance Engineer Mobile : +61 416 207 573 Product Technical Support (APAC) Fax : +61 2 9844 5311 Sun Microsystems http://blogs.sun.com/tpenta 828 Pacific Highway, Gordon NSW 2072 perl -e ''print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);'' _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Alan, On Fri, 26 Nov 2004 dtrace@opensolaris.org wrote:> Funny that you should bring this up Brendan. The dtrace folks have done > a gathering of scripts (from across the company) to go into > /usr/demo/dtrace. I''m not sure if this made build 72 for the next > Express, but you should certainly be able to find them under the source > tree under usr/src/cmd/dtrace/demo. > > How''s that for a start?Superb! I was a bit worried that many would be obscurely specific to particular Sun faults (that only a Sun person could love), but a quick look around shows that many are general and straightforward - good stuff. :) Brendan _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
Howdy Brendan. dtrace@opensolaris.org''s email at 11/26/2004 6:13 AM, said:> G''Day Folks, > > Well, I''ve been demo''ing DTrace to other sysadmins again which have led to > the same conclusions - you would know these already, this might just add > weight to the argument.I have given presentations to well over a thousand folks so far. Ain''t it just tons of fun?> > > I wrote some DTrace scripts on http://www.brendangregg.com/dtrace.html > (and more to come as I get the chance), which I''ve shown to many sysadmins > to get them enthusiastic about DTrace. It went along the lines of - see > the sort of things you can do, and with a comfy C-like language the sky''s > the limit. > > There''s been two problems with this: > > 1. Some don''t have the background to understand why DTrace is so > fantastic. Even with prompting like "how do we snoop live exec()''s with > all their details?", don''t trigger the expected discussions on truss, > process accounting and BSM auditing; instead this gets blank looks and > shrugs as if to say "why would that be useful?". I find explaining DTrace > is like telling a joke - some get it and laugh but others don''t, and > having to explain the punchline ruins the impact. (Not that I''m saying > DTrace is a joke, of course! ;).. If you''ve rolled around in the sort of > problems that DTrace solves well, then it''s obvious that it''s amazing; if > you haven''t, then it''s not so obvious... I don''t quite know an answer to > this problem...Ask them what their number one problem is. Then show them how easily it can be solved with DTrace. That is how I do it. Suddenly, when they see that their pain can be easily solved they "get it".> > 2. It seems fewer and fewer sysadmins these days know C. Many take a look > and say they''ll never program in DTrace, all they want is examples to use.You don''t really have to know C. I plan to take one day this weekend and teach my wife (non-technical physical therapist type) D. I figure she''ll know the *language* in a couple of hours. Now how to apply it, that is a different question and takes months. It just pisses me off when people (including some Sun employees) complain about the size of the user guide and how they don''t have time to learn a new language.> ><SNIP /usr/demo/dtrace lead in> >> > I think DTrace is way cool and deserves every chance at mega success - > a /usr/lib/dtrace of examples can''t hurt. >DTrace definately deserves a chance at success and I think it is getting it. We just need more capable people like yourself (and me) out evangelizing it. I have DTrace''d over 120 applications to date at many major companies. 100% of the time I have gotten a win. If that doesn''t clearly indicate the power of DTrace, I don''t know what does. Thanks, Jarod [ Texas ] ;) _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
Hi Brendan, The two points you raise below are excellent and I come across these all the time. They are both really a product of the dumbing down that has happened in the computing industry in recent years. Comments below and apologies for the length.>1. Some don''t have the background to understand why DTrace is so >fantastic. Even with prompting like "how do we snoop live exec()''s with >all their details?", don''t trigger the expected discussions on truss, >process accounting and BSM auditing; instead this gets blank looks and >shrugs as if to say "why would that be useful?". I find explaining DTrace >is like telling a joke - some get it and laugh but others don''t, and >having to explain the punchline ruins the impact. (Not that I''m saying >DTrace is a joke, of course! ;).. If you''ve rolled around in the sort of >problems that DTrace solves well, then it''s obvious that it''s amazing; if >you haven''t, then it''s not so obvious... I don''t quite know an answer to >this problem... >Most if not all the developers I''ve ever engaged with around DTrace always just get it. They mostly have a lot of background and, very importantly, they have struggled with gaining observability. It becomes easily apparent to them just what an astonishing tool DTrace is for them and what it brings to their party. If any doubt at all then, as Jarod says, applying it to their code quickly removes any doubt ... The average sysadmin on the other hand is a different kettle of fish. My experience tells me that many never code at all and a lot have never coded. Amazingly enough there are a reasonable amount that can hardly script! This isn''t the audience that is interested in the kind of thing that you cite because they never need to know these things. The key thing for these kind of guys is up-levelling to something that they do understand and struggle with. The stable abstractions that the sdt based providers bring is just what these guys need. You would have to be dead above the neck to not be amazed the first time you see what the io, sched and proc providers bring to you (especially the io provider as that is probably the biggest source of misery for many of them). If I''ve got a bunch of regular sysadmins in front of me then I''ll just mainly stick around these providers as well as some humourous party trick scripts and maybe some graphical displays (see below).>2. It seems fewer and fewer sysadmins these days know C. Many take a look >and say they''ll never program in DTrace, all they want is examples to use. > > >A healthy stack of examples will help towards that as Alan has talked about (as an example, who learnt perl from a man page anyway?). For me I think the more abstraction and layering we can provide, the better. As I''ve said, the sdt based providers brought an immense source of richness and this leads to greater acceptance. It''s not fair to expect lots of people to understand implementation so as to understand the behaviour they encounter. What we also need to do is to layer graphical tools on top of DTrace. As much as some of us might not like it, people like graphical interfaces and are not happy cranking out scripts. DTrace gives us an astonishingly rich fabric which we can layer tools on to provide as yet unseen views of systemic behaviour. I think that this is where things become very exciting. We are going to provide some Java APIs into DTrace that will hopefully enable lots of cool innovation around visualising behaviour. As an example of what visualisation brings to the whole "impressing people" piece I''ll just relay a quick story. A colleague and I wrote a Java based graphical tool which draws call graphs for fbt based functions (and soon pid based work as well) - I did the JNI consumer part and he did the real Java! I presented DTrace at last years SUPerG conference and after an hour of frantic live coding (it was all demo and no slides) I ended with showing people a dynamically generated view of what a ufs_write() does. There was spontaneous applause from many of the several hundred in the room. The lesson from this last story is not that the tool or the show were good (far from it) but that people generally love actually "seeing" data. For most people the more abstraction the better. It''s only a smaller subset of the whole that will enjoy being knee deep and speculations and fbt - God bless them all. However, the DTrace framework is rich enough for all to live in harmony. Cheers. Jon. _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
> On a completely unrelated side note, does anyone else think it is > stupid that the mail list (see below) is DTrace@www.opensolaris.com. > What the hell is up with the www? Even if the "From" is screwed up, at > least it doesn''t have the www. >I''d second that and this coming from the people that make the ''net work ;-) Nathan _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Jon, On Fri, 26 Nov 2004 dtrace@opensolaris.org wrote: [...]> The two points you raise below are excellent and I come across these all > the time. They are both really a product of the dumbing down that has > happened in the computing industry in recent years. Comments below > and apologies for the length.I''ve had the feeling, not just with DTrace, that the computing industry is certainly dumbing down. If I were to pick a few factors I''d say: 1. The IT Boom. Many people entered the industry who normally would not have. 2. The dissapearance of University Unix servers. I''ve met Comp Sci graduates who have never used Unix/Linux, ever. Now they have landed a job as a Solaris admin. Gone are the days where *everyone* at Uni would use Unix - the days of pine, elm, tin, finger, talk, screen, MUDs, WYSE terminals, Archie, Veronica, etc, etc, ... 3. The appearence of 6 month become-an-IT-professional training courses. Their marketing campaigns go along the lines of - in just 6 easy months you too will be a highly paid IT professional. 4. Cost-cutting. Managers want to believe that IT is easy, as then they don''t need to hire specialists, and it is okay to replace staff with juniors, or to outsource, etc.> Most if not all the developers I''ve ever engaged with around DTrace always > just get it. They mostly have a lot of background and, very importantly, > they > have struggled with gaining observability. It becomes easily apparent to > them > just what an astonishing tool DTrace is for them and what it brings to their > party. If any doubt at all then, as Jarod says, applying it to their > code quickly > removes any doubt ...Yes, DTrace would certainly be best understood by many developers.> The average sysadmin on the other hand is a different kettle of fish. My > experience tells me that many never code at all and a lot have never coded. > Amazingly enough there are a reasonable amount that can hardly script! > This isn''t the audience that is interested in the kind of thing that you > cite > because they never need to know these things. The key thing for these kind > of guys is up-levelling to something that they do understand and > struggle with.The lack of programming skills with sysadmins seems to be becomming worse. I''ve met some who firmly believe that programming is for "lowly" programmers, and that they never intend to ever learn a language - not even scripting. It''s quite worrying!> The stable abstractions that the sdt based providers bring is just what > these > guys need. You would have to be dead above the neck to not be amazed > the first time you see what the io, sched and proc providers bring to you > (especially the io provider as that is probably the biggest source of > misery for > many of them). If I''ve got a bunch of regular sysadmins in front of me then > I''ll just mainly stick around these providers as well as some humourous > party > trick scripts and maybe some graphical displays (see below).Yes, I figured iosnoop.d would have to be the most obvious example to deliver to sysadmins - disk I/O by process has always bugged me in the past with Unix (next on my hit list is networking by process). But still, some sysadmins don''t know of the existance of the iostat command - let alone understand why more info would be useful. This go back to the first point you made! [...]> What we also need to do is to layer graphical tools on top of DTrace. As > much > as some of us might not like it, people like graphical interfaces and > are not > happy cranking out scripts. DTrace gives us an astonishingly rich fabric > which > we can layer tools on to provide as yet unseen views of systemic > behaviour. I > think that this is where things become very exciting. We are going to > provide some Java APIs into DTrace that will hopefully enable lots of cool > innovation around visualising behaviour.Cool! I had the same idea - that plugging the data into a GUI would be great. I looked around core Solaris (actually SUNWCXall), and found no easy way to just plot data. Maybe if Perl''s GD::Graph was core Solaris, or gnuplot. I did find /usr/dt/bin/dtksh however, which seemed like it did plotting but no examples of this on the Internet. I then became sidetracked and wrote a bunch of dtksh scripts to demonstrate plotting, including xplot which works like, vmstat 1 | xplot -f 3 To plot field 3. I had to call them "demos", as they had a kludgey way of doing a timed read. http://www.brendangregg.com/dtkshdemos.html . I hear the timed read will be fixed in Solaris 10 - so they might become useful tools after all!> As an example of what visualisation brings to the whole "impressing people" > piece I''ll just relay a quick story. A colleague and I wrote a Java > based graphical > tool which draws call graphs for fbt based functions (and soon pid based > work > as well) - I did the JNI consumer part and he did the real Java! I > presented DTrace > at last years SUPerG conference and after an hour of frantic live coding > (it was > all demo and no slides) I ended with showing people a dynamically > generated view > of what a ufs_write() does. There was spontaneous applause from many of the > several hundred in the room.Sounds good - Java probably does graphics much nicer that dtksh ever did ;-)> The lesson from this last story is not that the tool or the show were > good (far from it) > but that people generally love actually "seeing" data. For most people[...] Yep - I guess it''s a modern form of the blinkin'' lights DTrace may need similar switch to snoop''s "-a", so users can also *listen* to kernel think! Brendan [Sydney, Australia] _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
From: Brendan Gregg <brendan.gregg@tpg.com.au> On Sat, 27 Nov 2004 dtrace@opensolaris.org wrote:> > On a completely unrelated side note, does anyone else think it is > > stupid that the mail list (see below) is DTrace@www.opensolaris.com. > > What the hell is up with the www? Even if the "From" is screwed up, at > > least it doesn''t have the www. > > > I''d second that and this coming from the people that make the ''net work ;-) > > NathanLine 1 of this email is a hand typed From line, I thought it might be an amusing thing to try. And I have no idea which "Nathan" that is! The mail header timestamp put you in Europe.. :) Brendan [Sydney, Australia] _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
Hello Brendan, dtrace@www.opensolaris.org wrote:>G''Day Jon, > >On Fri, 26 Nov 2004 dtrace@opensolaris.org wrote: >[...] > > >>The two points you raise below are excellent and I come across these all >>the time. They are both really a product of the dumbing down that has >>happened in the computing industry in recent years. Comments below >>and apologies for the length. >> >> > >I''ve had the feeling, not just with DTrace, that the computing industry is >certainly dumbing down. If I were to pick a few factors I''d say: > >I have had a similar conversation to this with Ben Rockwood recently, here is my AU$0.02 on the matter.>1. The IT Boom. > Many people entered the industry who normally would not have. > >Agreed. After recent conversations with a number of large consulting shops in Sydney, they find that it is easy to get people, but that it is impossible to get good people.>2. The dissapearance of University Unix servers. > I''ve met Comp Sci graduates who have never used Unix/Linux, ever. > Now they have landed a job as a Solaris admin. Gone are the days where > *everyone* at Uni would use Unix - the days of pine, elm, tin, finger, > talk, screen, MUDs, WYSE terminals, Archie, Veronica, etc, etc, ... > ><RANT> Who the hell thought it would be a good idea to get rid of UNIX from Universities? We at CSU (Charles Sturt University) are suffering from this problem due to the students'' total lack of ability to use anything but Windows. It saddens me to see two perfectly good Mac labs, running recent hardware and OSX, almost completely unused except for the odd graphic design student or two. Having recently finished a Masters of Information Technology (I handed in my final research report yesterday[0]), I have a few things to say about this. When I started back in 1997, the IT students (we don''t have a comp sci degree) would check their mail with pine, program on UNIX using vi and so on. I was recently in a class to do with Linux Networking and Security (bloody prerequisites). The majority of the people in the class had problems setting up Linux boxes, let alone using them. Entire classes were dedicated to learning vi !!!! These people are postgraduate students, I expect postgrads to be comfortable using whatever environment they are in; UNIX, Windows, Mainframe ... it''s all just variations on the same theme. Being "forced" to use a command-line is considered too difficult for these people !!! God forbid they actually have to work outside of a Windows world one day. But I digress. Of the students I had met, I doubt that half of them would be useful once they graduate. I was able to line up a job in Sydney for one of the useful ones though.>3. The appearence of 6 month become-an-IT-professional training courses. > Their marketing campaigns go along the lines of - in just 6 easy months > you too will be a highly paid IT professional. > >This just plain sh*ts me. The industry itself is suffering at the hands of these so called "professionals". I come back to my previous comment; It''s easy to find people, but the good ones are hard to come by. </RANT>>4. Cost-cutting. > Managers want to believe that IT is easy, as then they don''t need to > hire specialists, and it is okay to replace staff with juniors, or to > outsource, etc. > >One thing I came across in my research[0] is the concept that "quality is free"[1] If you take into account the costs of the bumbling about by the incompetent 6-month IT professional types, measured against the benefits that a good sys-admin can bring to the company, then paying a good sys-admin AU$80-120K a year is nothing.> > >>Most if not all the developers I''ve ever engaged with around DTrace always >>just get it. They mostly have a lot of background and, very importantly, >>they >>have struggled with gaining observability. It becomes easily apparent to >>them >>just what an astonishing tool DTrace is for them and what it brings to their >>party. If any doubt at all then, as Jarod says, applying it to their >>code quickly >>removes any doubt ... >> >> > >Yes, DTrace would certainly be best understood by many developers. > > > >>The average sysadmin on the other hand is a different kettle of fish >>Ben has more to say on the "average sys-admin"; However, I will summarise his words here. Ben claimed that the average sys-admin is concerned simply with keeping the system running and not with improving it. A disk breaks, they put in a new one; Need a user added, they can do it. Anything beyond that is outside of their abilities.>>. My >>experience tells me that many never code at all and a lot have never coded. >>Amazingly enough there are a reasonable amount that can hardly script! >> >>These are the people I would call tape-monkeys.>>This isn''t the audience that is interested in the kind of thing that you >>cite >>because they never need to know these things. The key thing for these kind >>of guys is up-levelling to something that they do understand and >>struggle with. >> >> > >The lack of programming skills with sysadmins seems to be becomming worse. >I''ve met some who firmly believe that programming is for "lowly" >programmers, and that they never intend to ever learn a language - >not even scripting. It''s quite worrying! > >This is compounded by many developer''s opinions of system administrators. Many developers believe that system administrators simply get in the way; However, there are good developers and there are good sys-admins. Conversely, there are bad developers and bad sys-admins. To be good at either of these two fields, you need an appreciation of the other. By this I mean that a good developer needs to understand the effects of his code on the system and this is where DTrace can help. On the other hand, a good sys-admin needs to understand the aspects of development to properly manage the demands of the applications on the system, this is where DTrace can help. Both fields gain from this truly excellent tool. I take my hat off to Sun for this.> > >>The stable abstractions that the sdt based providers bring is just what >>these >>guys need. You would have to be dead above the neck to not be amazed >>the first time you see what the io, sched and proc providers bring to you >>(especially the io provider as that is probably the biggest source of >>misery for >>many of them). If I''ve got a bunch of regular sysadmins in front of me then >>I''ll just mainly stick around these providers as well as some humourous >>party >>trick scripts and maybe some graphical displays (see below). >> >> > >Yes, I figured iosnoop.d would have to be the most obvious example to >deliver to sysadmins - disk I/O by process has always bugged me in the >past with Unix (next on my hit list is networking by process). But still, >some sysadmins don''t know of the existance of the iostat command - let >alone understand why more info would be useful. This go back to the first >point you made! > >[...] > >iosnoop is really cool. I just wish that I had come across the previous versions of it earlier. Nice work.>>What we also need to do is to layer graphical tools on top of DTrace. As >>much >>as some of us might not like it, people like graphical interfaces and >>are not >>happy cranking out scripts. >>For those of us who are happy cranking out scripts, I found pumping the *stat family''s output into gnuplot quite helpful for creating a graphical representation. I used it once to monitor the I/O used in a number of batch jobs once using vxstat. Little did I know that my manager two levels above me was watching it too. While I didn''t appreciate the bollocking I got when it broke while I was trying to extend it, it was cool to know that the higher-ups were actually using the tool that I had built and the graphs really helped with that. I wish I had Solaris 10 and DTrace at the time of that contract.>>DTrace gives us an astonishingly rich fabric >>which >>we can layer tools on to provide as yet unseen views of systemic >>behaviour. I >>think that this is where things become very exciting. We are going to >>provide some Java APIs into DTrace that will hopefully enable lots of cool >>innovation around visualising behaviour. >> >>Agreed.> >Cool! I had the same idea - that plugging the data into a GUI would be >great. I looked around core Solaris (actually SUNWCXall), and found no >easy way to just plot data. Maybe if Perl''s GD::Graph was core Solaris, or >gnuplot. > >I would suggest gnuplot for shell scripts. I apologise for the somewhat off-topic nature of my post, but my rant was unstoppable once you mentioned the 6 Month IT professionals. Regards, Nathan Dietsch [Wagga Wagga, Australia] [0] Change Management in System Administration [1] Attributed to Phillip Crosby _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
dtrace@opensolaris.org wrote:> Hello Brendan, > > dtrace@www.opensolaris.org wrote: > >> From: Brendan Gregg <brendan.gregg@tpg.com.au> >> >> On Sat, 27 Nov 2004 dtrace@opensolaris.org wrote: >> >> >> >>>> On a completely unrelated side note, does anyone else think it is >>>> stupid that the mail list (see below) is DTrace@www.opensolaris.com. >>>> What the hell is up with the www? Even if the "From" is screwed up, at >>>> least it doesn''t have the www. >>>> >>>> >>> >>> I''d second that and this coming from the people that make the ''net >>> work ;-) >>> >>> Nathan >>> >> >> >> >> Line 1 of this email is a hand typed From line, I thought it might be an >> amusing thing to try. >> >> And I have no idea which "Nathan" that is! The mail header timestamp put >> you in Europe.. :) >> >> Brendan >> >> [Sydney, Australia] >> >> > Sorry about that, I assumed that my name would be in the From header; > A stupid assumption given the topic of conversation (It''s Saturday > morning here, that''s my excuse and I am sticking with it). > > My mail server is in the UK, but I am (sort of) just down the road > from you in Wagga Wagga, NSW. > > Nathan Dietsch > _______________________________________________ > DTrace mailing list > DTrace@www.opensolaris.org > https://www.opensolaris.org/mailman/listinfo/dtraceI sent an email to Derek after the post I made last week about this. Hopefully, it wil get fixed shortly. alan. -- Alan Hargreaves Senior Technical Support Specialist Phone : +61 2 9844 5379 Kernel/VOSJEC/Performance Engineer Mobile : +61 416 207 573 Product Technical Support (APAC) Fax : +61 2 9844 5311 Sun Microsystems http://blogs.sun.com/tpenta 828 Pacific Highway, Gordon NSW 2072 perl -e ''print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);'' _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
On Mon, 29 Nov 2004, Derek Cicero wrote:> dtrace@opensolaris.org wrote: > > > dtrace@opensolaris.org wrote:[...]> Sorry for the late response folks, I was out of the office for a while. > This issue is fixed. > > DerekThanks Derek, dtrace [Sydney, Australia] _______________________________________________ DTrace mailing list DTrace@opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace
G''Day Folks, Well, I''ve been demo''ing DTrace to other sysadmins again which have led to the same conclusions - you would know these already, this might just add weight to the argument. I wrote some DTrace scripts on http://www.brendangregg.com/dtrace.html (and more to come as I get the chance), which I''ve shown to many sysadmins to get them enthusiastic about DTrace. It went along the lines of - see the sort of things you can do, and with a comfy C-like language the sky''s the limit. There''s been two problems with this: 1. Some don''t have the background to understand why DTrace is so fantastic. Even with prompting like "how do we snoop live exec()''s with all their details?", don''t trigger the expected discussions on truss, process accounting and BSM auditing; instead this gets blank looks and shrugs as if to say "why would that be useful?". I find explaining DTrace is like telling a joke - some get it and laugh but others don''t, and having to explain the punchline ruins the impact. (Not that I''m saying DTrace is a joke, of course! ;).. If you''ve rolled around in the sort of problems that DTrace solves well, then it''s obvious that it''s amazing; if you haven''t, then it''s not so obvious... I don''t quite know an answer to this problem... 2. It seems fewer and fewer sysadmins these days know C. Many take a look and say they''ll never program in DTrace, all they want is examples to use. So this brings me to a point (I had a point?), Customers want DTrace examples, I know you have some in the answerbook - they ought to be copied to /usr/lib/dtrace. I''d write a list of ideas of what I''d like to see in /usr/lib/dtrace - but I guess my website could serve as a suggestion list of ideas. Picture /opt/RICHPse/examples/* from the SE Toolkit. It had 80 examples, and is just what many people want. It''s suprising to see how uncomfortable people seem to be to cut-n-paste from the answerbook - I guess people might find DTrace is a little intimidating. ... It seems Sun has a habit of creating wonderfly powerful solutions that customers end up not using. For example, Solaris 8''s Sun::Solaris::Kstat. How cool was that?? A Perl library that made it dead simple to pluck out kernel stats on loads of interesting topics. How many customers used it? Try google''ing for Sun::Solaris::Kstat - not many hits (and I''m quite a few of them). I think DTrace is way cool and deserves every chance at mega success - a /usr/lib/dtrace of examples can''t hurt. Brendan Gregg [Sydney, Australia] _______________________________________________ DTrace mailing list DTrace@www.opensolaris.org https://www.opensolaris.org/mailman/listinfo/dtrace