Peter Schow
2007-Jul-12 22:49 UTC
[dtrace-discuss] University of Toronto''s JIFL OS instrumentation system
DTracers might be interested in reading about other dynamic OS instrumentation technologies, like the University of Toronto''s JIFL system: http://individual.utoronto.ca/marek/jifl-eurosys07.pdf They actually mention DTrace a few times, including in the "Related Work" section but seem to imply that they haven''t tried it.
Adam Leventhal
2007-Jul-29 01:40 UTC
[dtrace-discuss] University of Toronto''s JIFL OS instrumentation system
On Thu, Jul 12, 2007 at 04:49:13PM -0600, Peter Schow wrote:> DTracers might be interested in reading about other dynamic OS > instrumentation technologies, like the University of Toronto''s > JIFL system: > > http://individual.utoronto.ca/marek/jifl-eurosys07.pdf > > They actually mention DTrace a few times, including in the "Related > Work" section but seem to imply that they haven''t tried it.They did more than not try it -- they pretty much wrote it off. The focus on this paper was efficient instrumention, and they asserted that trap- based instrumentation would be slower than what they had developed; it would have been nice for them to confirm this about DTrace. The paper had some interesting ideas (though I''m not sure how much they went beyond the work done UW-Madison by Barton Miller et al.), but proposed some kernel instrumentation techniques which were a bit too exhilarating for my tastes. Adam -- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
Bryan Cantrill
2007-Jul-31 15:22 UTC
[dtrace-discuss] University of Toronto''s JIFL OS instrumentation system
On Sat, Jul 28, 2007 at 06:40:39PM -0700, Adam Leventhal wrote:> On Thu, Jul 12, 2007 at 04:49:13PM -0600, Peter Schow wrote: > > DTracers might be interested in reading about other dynamic OS > > instrumentation technologies, like the University of Toronto''s > > JIFL system: > > > > http://individual.utoronto.ca/marek/jifl-eurosys07.pdf > > > > They actually mention DTrace a few times, including in the "Related > > Work" section but seem to imply that they haven''t tried it. > > They did more than not try it -- they pretty much wrote it off. The focus > on this paper was efficient instrumention, and they asserted that trap- > based instrumentation would be slower than what they had developed; it > would have been nice for them to confirm this about DTrace. > > The paper had some interesting ideas (though I''m not sure how much they > went beyond the work done UW-Madison by Barton Miller et al.), but proposed > some kernel instrumentation techniques which were a bit too exhilarating for > my tastes.I think Adam is being too charitable; at best, this paper represents dated, pre-DTrace notions of instrumentation. In particular, I believe that our most important contribution is the importance of instrumenting systems in _production_. The performance ramifications of this are clear, and they are exactly the opposite of the prevailing wisdom prior to DTrace: a dynamic instrumentation framework must have zero _disabled_ probe effect. The _enabled_ probe effect is actually not nearly as important as getting the disabled probe effect to zero, for it is the zero disabled probe effect that allows you to have the instrumentation that you need when you''re going to need it. (Like this work, virtually all work prior to DTrace focussed exclusively on minimizing the enabled probe effect, while cheerfully ignoring a non-zero disabled probe effect.) Indeed, this paragraph from the paper sums up their misunderstanding of DTrace''s fundamental contribution: JIFL incurs a performance cost of approximately 2% running with no instrumentation due to JIT overhead and cost of executing from the code cache. As noted earlier, this result it included to illustrate the cost of the JIFL framework -- if no instrumentation is wanted JIFL would be disabled, incurring no overhead. This represents an old developer mindset around dynamic instrumentation -- that one should make an upfront, non-reversible (for the lifetime of the running application/system) decision about whether instrumentation is "wanted." As many of you already know, you often don''t know that you want instrumentation until you need it -- and with their dated approach, it''s too late. I hasten to add that there are all also safety ramifications of running in production, none of which they seem to understand or appreciate. They blow off these ramifications as "straightforward to add" in their section discussing DTrace -- which is pretty rich considering that they acknowledge in the next section that their work is not even preemption-safe. Will researchers ever start looking at the many new problems that DTrace has opened instead of the old problems that DTrace has solved? One hopes. - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
陶捷 Tao Jie
2007-Jul-31 15:55 UTC
[dtrace-discuss] University of Toronto''s JIFL OS instrumentation system
Hi, Bryan: 2007/7/31, Bryan Cantrill <bmc at eng.sun.com>:> > > On Sat, Jul 28, 2007 at 06:40:39PM -0700, Adam Leventhal wrote: > > On Thu, Jul 12, 2007 at 04:49:13PM -0600, Peter Schow wrote: > > > DTracers might be interested in reading about other dynamic OS > > > instrumentation technologies, like the University of Toronto''s > > > JIFL system: > > > > > > http://individual.utoronto.ca/marek/jifl-eurosys07.pdf > > > > > > They actually mention DTrace a few times, including in the "Related > > > Work" section but seem to imply that they haven''t tried it. > > > > They did more than not try it -- they pretty much wrote it off. The > focus > > on this paper was efficient instrumention, and they asserted that trap- > > based instrumentation would be slower than what they had developed; it > > would have been nice for them to confirm this about DTrace. > > > > The paper had some interesting ideas (though I''m not sure how much they > > went beyond the work done UW-Madison by Barton Miller et al.), but > proposed > > some kernel instrumentation techniques which were a bit too exhilarating > for > > my tastes. > > I think Adam is being too charitable; at best, this paper represents > dated, pre-DTrace notions of instrumentation. In particular, I believe > that our most important contribution is the importance of instrumenting > systems in _production_. The performance ramifications of this are clear, > and they are exactly the opposite of the prevailing wisdom prior to > DTrace: > a dynamic instrumentation framework must have zero _disabled_ probe > effect. > The _enabled_ probe effect is actually not nearly as important as getting > the disabled probe effect to zero, for it is the zero disabled probe > effect > that allows you to have the instrumentation that you need when you''re > going > to need it. (Like this work, virtually all work prior to DTrace focussed > exclusively on minimizing the enabled probe effect, while cheerfully > ignoring a non-zero disabled probe effect.) Indeed, this paragraph from > the paper sums up their misunderstanding of DTrace''s fundamental > contribution: > > JIFL incurs a performance cost of approximately 2% running with > no instrumentation due to JIT overhead and cost of executing from > the code cache. As noted earlier, this result it included to > illustrate the cost of the JIFL framework -- if no instrumentation > is wanted JIFL would be disabled, incurring no overhead. > > This represents an old developer mindset around dynamic instrumentation -- > that one should make an upfront, non-reversible (for the lifetime of the > running application/system) decision about whether instrumentation is > "wanted." As many of you already know, you often don''t know that you > want instrumentation until you need it -- and with their dated approach, > it''s too late. > > I hasten to add that there are all also safety ramifications of running > in production, none of which they seem to understand or appreciate. They > blow off these ramifications as "straightforward to add" in their section > discussing DTrace -- which is pretty rich considering that they > acknowledge > in the next section that their work is not even preemption-safe. > > Will researchers ever start looking at the many new problems that DTrace > has opened instead of the old problems that DTrace has solved? One hopes.Where do these new problems list? Regards, TJ - Bryan> > -------------------------------------------------------------------------- > Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070731/9bddaeb0/attachment.html>
Matt Ingenthron
2007-Jul-31 17:05 UTC
[dtrace-discuss] University of Toronto''s JIFL OS instrumentation system
?? Tao Jie wrote:> Hi, Bryan: > > 2007/7/31, Bryan Cantrill <bmc at eng.sun.com <mailto:bmc at eng.sun.com>>:(snip...)> > > Will researchers ever start looking at the many new problems that > DTrace > has opened instead of the old problems that DTrace has > solved? One hopes. > > > Where do these new problems list? >I believe Bryan is referring to the problems that you can now see in the system using DTrace. There isn''t a specific list of problems. In nearly any given area, one could find room for improvement. Another instrumentation tool which gives you the same data doesn''t help fix these issues. Pointing DTrace at other parts of the system to get data on the cause can help one find solutions to these age-old problems. - Matt -- Matt Ingenthron - Web Infrastructure Solutions Architect Sun Microsystems, Inc. - Global Systems Practice http://blogs.sun.com/mingenthron/ email: matt.ingenthron at sun.com Phone: 310-242-6439
陶捷 Tao Jie
2007-Jul-31 17:13 UTC
[dtrace-discuss] University of Toronto''s JIFL OS instrumentation system
2007/8/1, Matt Ingenthron <Matt.Ingenthron at sun.com>:> > ?? Tao Jie wrote: > > Hi, Bryan: > > > > 2007/7/31, Bryan Cantrill <bmc at eng.sun.com <mailto:bmc at eng.sun.com>>: > > (snip...) > > > > > > > Will researchers ever start looking at the many new problems that > > DTrace > > has opened instead of the old problems that DTrace has > > solved? One hopes. > > > > > > Where do these new problems list? > > > > I believe Bryan is referring to the problems that you can now see in the > system using DTrace. There isn''t a specific list of problems. In > nearly any given area, one could find room for improvement. > > Another instrumentation tool which gives you the same data doesn''t help > fix these issues. Pointing DTrace at other parts of the system to get > data on the cause can help one find solutions to these age-old problems.Oh, I see. err... I want to say, in fact, like business is business, academy is academy as well. Regards, TJ - Matt> > -- > Matt Ingenthron - Web Infrastructure Solutions Architect > Sun Microsystems, Inc. - Global Systems Practice > http://blogs.sun.com/mingenthron/ > email: matt.ingenthron at sun.com Phone: 310-242-6439 > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070801/5d01657c/attachment.html>
Bryan Cantrill
2007-Jul-31 17:18 UTC
[dtrace-discuss] University of Toronto''s JIFL OS instrumentation system
> > Will researchers ever start looking at the many new problems that > > DTrace > > has opened instead of the old problems that DTrace has > > solved? One hopes. > > > > > > Where do these new problems list? > > > > I believe Bryan is referring to the problems that you can now see in the > system using DTrace. There isn''t a specific list of problems. In > nearly any given area, one could find room for improvement. > > Another instrumentation tool which gives you the same data doesn''t help > fix these issues. Pointing DTrace at other parts of the system to get > data on the cause can help one find solutions to these age-old problems.Yes, exactly. The problem has transitioned from "how do we get data on a running system?" to "given our ability to gather essentially arbitrary data on a running system, what data do we gather and how do we make sense of it?" This is a much larger and much harder problem. We''re seeing some exciting efforts to attack this from the developer tools side (e.g., Apple''s Xray, Sun''s Project D-Light), but there is still much to do to build on the data that DTrace can gather from production systems. I would love to see researchers start to attack these problems if only because (in contrast to the pre-DTrace era) they needn''t (re)invent the wheel before working on the automobile (or the motorcycle or the skateboard or the roller skate). - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
Hi, We are evaluating products (such as Confio''s Ignite) to instrument our webapps (Java and Oracle) In the case of Ignite, the unified app and DB view is very valuable. Is there a comparable presentation layer for Dtrace (something to generate graphical reports of time spent in Java methods)? - DLite? The results are used on the dev side to improve code efficiency BEFORE deployment to production, then it would be great to have the ability to run ad hoc reports in production while performance issues/questions are occurring. Many thanks for any tips, Fletcher
On Tue, Jul 31, 2007 at 10:31:19AM -0700, Fletcher Cocquyt wrote:> Hi, > We are evaluating products (such as Confio''s Ignite) to instrument our > webapps (Java and Oracle) > In the case of Ignite, the unified app and DB view is very valuable. > > Is there a comparable presentation layer for Dtrace (something to generate > graphical reports of time spent in Java methods)? - DLite? > > The results are used on the dev side to improve code efficiency BEFORE > deployment to production, then it would be great to have the ability to run > ad hoc reports in production while performance issues/questions are > occurring. > > Many thanks for any tips, > Fletcher >Fletcher, If you can run Solaris 11 (or Solaris 10 update 4) you can try the Chime Visualization Tool for DTrace, available for download here: http://www.opensolaris.org/os/project/dtrace-chime/install/ Chime displays aribitrary DTrace aggregations using live bar graphs and line graphs. It has a JMX layer to support remote tracing, and it supports record/playback. I attached a display of time spent in Java methods, sortable by class name and/or method name (including signature). You can copy the attached method_times.xml into /opt/OSOL0chime/displays, run /opt/OSOL0chime/bin/chime, then select "Method Times ..." from the Traces list in the Chime GUI to try it yourself. For some helpful pointers on how to trace Java apps, check out these displays from Steven Reynolds'' JavaOne presentation, Performance Statistics of Java(TM) Applications Using DTrace and Chime: http://www.opensolaris.org/os/project/dtrace-chime/misc/javaone2007_screens/ Tom -------------- next part -------------- <?xml version="1.0" encoding="UTF-8"?> <!-- * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END --> <!-- Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. --> <java version="1.6.0_01" class="java.beans.XMLDecoder"> <object class="org.opensolaris.chime.DisplayDescription"> <void property="clearedAggregations"> <array class="java.lang.String" length="0"/> </void> <void property="columns"> <array class="org.opensolaris.chime.ColumnDescription" length="6"> <void index="0"> <object class="org.opensolaris.chime.ColumnDescription"> <void property="hidden"> <boolean>true</boolean> </void> <void property="name"> <string>Class Name</string> </void> <void property="tupleIndex"> <int>0</int> </void> </object> </void> <void index="1"> <object class="org.opensolaris.chime.ColumnDescription"> <void property="name"> <string>Class</string> </void> <void property="sourceColumnName"> <string>Class Name</string> </void> <void property="totallerDescription"> <object class="org.opensolaris.chime.TotallerDescription"> <void property="displayFormat"> <string>{0, number, integer} {0, choice, 0#classes|1#class|2#classes}</string> </void> <void property="totalType"> <string>UNIQUE_COUNT</string> </void> </object> </void> <void property="valueType"> <string>GROUP</string> </void> </object> </void> <void index="2"> <object class="org.opensolaris.chime.ColumnDescription"> <void property="hidden"> <boolean>true</boolean> </void> <void property="name"> <string>Method Name</string> </void> <void property="tupleIndex"> <int>1</int> </void> </object> </void> <void index="3"> <object class="org.opensolaris.chime.ColumnDescription"> <void property="hidden"> <boolean>true</boolean> </void> <void property="name"> <string>Signature</string> </void> <void property="tupleIndex"> <int>2</int> </void> </object> </void> <void index="4"> <object class="org.opensolaris.chime.ColumnDescription"> <void property="name"> <string>Method</string> </void> <void property="sourceColumnName"> <string>Method Name</string> </void> <void property="totallerDescription"> <object class="org.opensolaris.chime.TotallerDescription"> <void property="displayFormat"> <string>{0, number, integer} {0, choice, 0#methods|1#method|2#methods}</string> </void> <void property="totalType"> <string>COUNT</string> </void> </object> </void> <void property="tupleConversion"> <object class="org.opensolaris.chime.TupleConversionDescription"> <void property="conversionName"> <string>JOIN</string> </void> <void property="tupleIndexes"> <array class="int" length="2"> <void index="0"> <int>1</int> </void> <void index="1"> <int>2</int> </void> </array> </void> <void property="unconvertedIndex"> <int>1</int> </void> <void property="userSpec"> <string>{0}{1}</string> </void> </object> </void> </object> </void> <void index="5"> <object class="org.opensolaris.chime.ColumnDescription"> <void property="aggregationName"> <string>method_times</string> </void> <void property="displayFormat"> <string>###0.0</string> </void> <void property="minimumValueRange"> <long>10000000</long> </void> <void property="name"> <string>Time</string> </void> <void property="plottable"> <boolean>true</boolean> </void> <void property="unitsDescription"> <object class="org.opensolaris.chime.util.DisplayUnits$Description"> <void property="unitLabels"> <array class="java.lang.String" length="4"> <void index="0"> <string>ns</string> </void> <void index="1"> <string>us</string> </void> <void index="2"> <string>ms</string> </void> <void index="3"> <string>s</string> </void> </array> </void> <void property="unitMultiple"> <int>1000</int> </void> </object> </void> </object> </void> </array> </void> <void property="longDescription"> <string><html>Displays cumulative time spent in Java(TM) methods. The target JVM must be run with the <b><font face="Courier">-XX:+ExtendedDTraceProbes</font></b> option on a Java 6 runtime.<br> <br> Method times are overlaid when one method calls another. Time spent in a called method is also time spent in the calling method, so it does not make sense to total the values in the <b>Time</b> column. Time is traced using the DTrace built-in <i>timestamp</i> variable. If a method runs in more than one thread, the time spent in that method is summed across all threads and may exceed the time spent running the program.<br> <br> The display supports drilldown by thread ID (right-click one or more selected rows and access drilldown from the popup menu). Because of a limitation specific to the hotspot provider, starting a new display with drilldown stops the original display.</html></string> </void> <void property="menuItems"> <array class="org.opensolaris.chime.MenuItemDescription" length="1"> <void index="0"> <object class="org.opensolaris.chime.MenuItemDescription"> <void property="multiSelectionName"> <string>by Thread</string> </void> <void property="newColumns"> <array class="org.opensolaris.chime.ColumnDescription" length="1"> <void index="0"> <object class="org.opensolaris.chime.ColumnDescription"> <void property="name"> <string>Thread</string> </void> <void property="totallerDescription"> <object class="org.opensolaris.chime.TotallerDescription"> <void property="displayFormat"> <string>{0, number, integer} {0, choice, 0#threads|1#thread|2#threads}</string> </void> <void property="totalType"> <string>COUNT</string> </void> </object> </void> <void property="tupleIndex"> <int>0</int> </void> <void property="tupleMember"> <string>tid</string> </void> </object> </void> </array> </void> <void property="singleSelectionNameFormat"> <string>{0} by Thread</string> </void> </object> </void> </array> </void> <void property="options"> <array class="org.opensolaris.os.dtrace.Option" length="4"> <void index="0"> <object class="org.opensolaris.os.dtrace.Option"> <string>zdefs</string> <string>set</string> </object> </void> <void index="1"> <object class="org.opensolaris.os.dtrace.Option"> <string>dynvarsize</string> <string>10m</string> </object> </void> <void index="2"> <object class="org.opensolaris.os.dtrace.Option"> <string>aggsize</string> <string>2m</string> </object> </void> <void index="3"> <object class="org.opensolaris.os.dtrace.Option"> <string>bufsize</string> <string>2m</string> </object> </void> </array> </void> <void property="programString"> <string>/* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright 2007 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ hotspot$target:::method-entry { /* class name */ self->str_ptr = (char*)copyin(arg1, arg2 + 1); self->str_ptr[arg2] = '\0'; self->class = (string)self->str_ptr; /* method name */ self->str_ptr = (char*)copyin(arg3, arg4 + 1); self->str_ptr[arg4] = '\0'; self->method = (string)self->str_ptr; /* signature */ self->str_ptr = (char*)copyin(arg5, arg6 + 1); self->str_ptr[arg6] = '\0'; self->signature = (string)self->str_ptr; self->ptr = 0; } hotspot$target:::method-entry ??P0?? { self->times[??T??] = timestamp; } hotspot$target:::method-entry { self->class_name = 0; self->method_name = 0; self->signature = 0; } hotspot$target:::method-return { /* class name */ self->str_ptr = (char*)copyin(arg1, arg2 + 1); self->str_ptr[arg2] = '\0'; self->class = (string)self->str_ptr; /* method name */ self->str_ptr = (char*)copyin(arg3, arg4 + 1); self->str_ptr[arg4] = '\0'; self->method = (string)self->str_ptr; /* signature */ self->str_ptr = (char*)copyin(arg5, arg6 + 1); self->str_ptr[arg6] = '\0'; self->signature = (string)self->str_ptr; self->str_ptr = 0; } hotspot$target:::method-return ??P1?? { @method_times[??T??] sum(timestamp - self->times[??T??]); self->times[??T??] = 0; } hotspot$target:::method-return { self->class_name = 0; self->method_name = 0; self->signature = 0; } ??T0 self->class ??T1 self->method ??T2 self->signature ??P0 ??P1 self->times[??T??]</string> </void> <void property="promptTarget"> <boolean>true</boolean> </void> <void property="size"> <object class="java.awt.Dimension"> <int>642</int> <int>452</int> </object> </void> <void property="stopOnDrilldown"> <boolean>true</boolean> </void> <void property="title"> <string>Method Times</string> </void> <void property="version"> <int>2</int> </void> </object> </java>