dtrace discuss - Sep 2005 - Can I use printa() for printing multiple aggregations?

Hi,

in my dtrace programs (using sax provider), I usually collect data on 
number of entries (@ce) and returns (@cr) for each probefunc as well as 
time spent there (@t), so I have 3 aggregations:

 	@ce, @cr, @t

and when I use

END{
 	printa(@ce);
 	printa(@cr);
 	printa(@t);
}

to print them, they come one after another which is not really helpful, 
if the listings are long. What would be ideal is to print a table similar 
to (in perl-like notation assuming that @ce, @cr and @t are associate 
arrays):

 	foreach $f (keys(%ce)) {
 		printf "%s %d %d %d\n", $f, $ce{$f}, $cr{$f}, $t{$f};
 	}

Is this possible? How? If not, can we open an RFE for e.g.
 	printa(format_strings, @agr [, @agr, ...])

?

TIA and bye,

Dragan

-- 
Dragan Cvetkovic,

To be or not to be is true. G. Boole    No it isn''t.  L. E. J. Brouwer

Hi Dragan,

That''s not currently possible, but we''ve been considering
adding some
new syntax to allow either multiple aggregating actions for a given
aggregation or a way of formatting multiple aggregations with the same
key signatures.

Adam

On Thu, Sep 08, 2005 at 10:08:42AM -0400, Dragan Cvetkovic
wrote:
> 	Hi,
> 
> in my dtrace programs (using sax provider), I usually collect data on 
> number of entries (@ce) and returns (@cr) for each probefunc as well as 
> time spent there (@t), so I have 3 aggregations:
> 
> 	@ce, @cr, @t
> 
> and when I use
> 
> END{
> 	printa(@ce);
> 	printa(@cr);
> 	printa(@t);
> }
> 
> to print them, they come one after another which is not really helpful, 
> if the listings are long. What would be ideal is to print a table similar 
> to (in perl-like notation assuming that @ce, @cr and @t are associate 
> arrays):
> 
> 	foreach $f (keys(%ce)) {
> 		printf "%s %d %d %d\n", $f, $ce{$f}, $cr{$f}, $t{$f};
> 	}
> 
> Is this possible? How? If not, can we open an RFE for e.g.
> 	printa(format_strings, @agr [, @agr, ...])
> 
> ?
> 
> TIA and bye,
> 
> Dragan
> 
> -- 
> Dragan Cvetkovic,
> 
> To be or not to be is true. G. Boole    No it isn''t.  L. E. J.
Brouwer
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Thu, 8 Sep 2005, Adam Leventhal wrote:
> On Thu, Sep 08, 2005 at 10:08:42AM -0400, Dragan Cvetkovic wrote:
>> If not, can we open an RFE for e.g.
>> 	printa(format_strings, @agr [, @agr, ...])
> Hi Dragan,
> That''s not currently possible, but we''ve been considering
adding some
> new syntax to allow either multiple aggregating actions for a given
> aggregation or a way of formatting multiple aggregations with the same
> key signatures.
Thanks for the answer Adam. I am looking forward to these extenstions.

In the mean time, is the piping dtrace output to awk/perl/you-name-it the 
only option I have?

Bye, Dragan

-- 
Dragan Cvetkovic,

To be or not to be is true. G. Boole    No it isn''t.  L. E. J. Brouwer

On Thu, Sep 08, 2005 at 01:41:53PM -0400, Dragan Cvetkovic
wrote:
> In the mean time, is the piping dtrace output to awk/perl/you-name-it the 
> only option I have?
That''s what I''d do, yeah. Not perfect, but you can make it
work.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

Maybe we should post some sample awk perl scripts for novice awk/perl  
programmers like me :-)

Thanks.
Regards,
Jignesh


Adam Leventhal wrote:
>On Thu, Sep 08, 2005 at 01:41:53PM -0400, Dragan Cvetkovic wrote:
>  
>
>>In the mean time, is the piping dtrace output to awk/perl/you-name-it
the
>>only option I have?
>>    
>>
>
>That''s what I''d do, yeah. Not perfect, but you can make it
work.
>
>Adam
>
>  
>
-- 
______________________________

Jignesh K. Shah
MTS Software Engineer, 
MDE - Horizontal Technologies 
Sun Microsystems, Inc
Phone: (781) 442 3052
Email: J.K.Shah at sun.com
______________________________

On Thu, Sep 08, 2005 at 10:35:46AM -0700, Adam Leventhal
wrote:
> Hi Dragan,
> 
> That''s not currently possible, but we''ve been considering
adding some
> new syntax to allow either multiple aggregating actions for a given
> aggregation or a way of formatting multiple aggregations with the same
> key signatures.
And for whatever it''s worth, after seeing your mail this morning, I
spent
my morning commute thinking about how to allow printa() to take multiple
aggregations with the same key signatures.  (This is an RFE that we''ve
had for a while -- the first time this was formally logged as a request
was on April 18th, 2003; it''s time to address it.)   It
shouldn''t be _too_
bad; we''ll keep you posted on our progress... (And apologies for
forcing
you to postprocess the data in the meantime.)

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

On Thu, 8 Sep 2005, Bryan Cantrill wrote:
> On Thu, Sep 08, 2005 at 10:35:46AM -0700, Adam Leventhal wrote:
>> Hi Dragan,
>>
>> That''s not currently possible, but we''ve been
considering adding some
>> new syntax to allow either multiple aggregating actions for a given
>> aggregation or a way of formatting multiple aggregations with the same
>> key signatures.
>
> And for whatever it''s worth, after seeing your mail this morning,
I spent
> my morning commute thinking about how to allow printa() to take multiple
> aggregations with the same key signatures.  (This is an RFE that
we''ve
> had for a while -- the first time this was formally logged as a request
> was on April 18th, 2003; it''s time to address it.)   It
shouldn''t be _too_
> bad; we''ll keep you posted on our progress... (And apologies for
forcing
> you to postprocess the data in the meantime.)
Hi Bryan, no problems about postprocessing. I don''t want to forget my
awk
and perl skills just because I have such a fantastic tool (dtrace) at my 
disposal! I guess Jignesh and I would manage to put something together :-)

Btw, do you happen to know the bugid of that RFE?

Bye, Dragan

-- 
Dragan Cvetkovic,

To be or not to be is true. G. Boole    No it isn''t.  L. E. J. Brouwer

On Thu, 8 Sep 2005, Jignesh K. Shah wrote:
> Adam Leventhal wrote:
>> On Thu, Sep 08, 2005 at 01:41:53PM -0400, Dragan Cvetkovic wrote:
>> 
>>> In the mean time, is the piping dtrace output to
awk/perl/you-name-it the
>>> only option I have?
>>> 
>> 
>> That''s what I''d do, yeah. Not perfect, but you can
make it work.
>> 
>> Adam
> Maybe we should post some sample awk perl scripts for novice awk/perl 
> programmers like me :-)
>
In my case

nawk ''{a[$1] = a[$1] " " $2} END{for(i in a) {print i,
a[i]}'' <dtrace_output_file

does the trick. OK, it needs some polishing, but you get the idea.

Bye, Dragan

-- 
Dragan Cvetkovic,

To be or not to be is true. G. Boole    No it isn''t.  L. E. J. Brouwer

> >And for whatever it''s worth, after seeing your mail this
morning, I spent
> >my morning commute thinking about how to allow printa() to take
multiple
> >aggregations with the same key signatures.  (This is an RFE that
we''ve
> >had for a while -- the first time this was formally logged as a request
> >was on April 18th, 2003; it''s time to address it.)   It
shouldn''t be _too_
> >bad; we''ll keep you posted on our progress... (And apologies
for forcing
> >you to postprocess the data in the meantime.)
> 
> Hi Bryan, no problems about postprocessing. I don''t want to forget
my awk
> and perl skills just because I have such a fantastic tool (dtrace) at my 
> disposal! I guess Jignesh and I would manage to put something together :-)
> 
> Btw, do you happen to know the bugid of that RFE?
Embarrassingly, we never filed a formal bug on this -- it was only tracked
as a pre-integration project bug.  I''ll get a bug filed and put you
down
as the call record...

Question for you (plural):  when you''re doing a printa() of multiple
aggregations, what should the sort order be?  We could obviously sum across
the values to derive a single sort value, but I''m not sure if
that''s what''s
desired -- should we be printing the aggregation in key-order instead of
value-order for this case?  

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

Bryan Cantrill wrote:
>>>And for whatever it''s worth, after seeing your mail this
morning, I spent
>>>my morning commute thinking about how to allow printa() to take
multiple
>>>aggregations with the same key signatures.  (This is an RFE that
we''ve
>>>had for a while -- the first time this was formally logged as a
request
>>>was on April 18th, 2003; it''s time to address it.)   It
shouldn''t be _too_
>>>bad; we''ll keep you posted on our progress... (And
apologies for forcing
>>>you to postprocess the data in the meantime.)
>>
>>Hi Bryan, no problems about postprocessing. I don''t want to
forget my awk
>>and perl skills just because I have such a fantastic tool (dtrace) at my
>>disposal! I guess Jignesh and I would manage to put something together
:-)
>>
>>Btw, do you happen to know the bugid of that RFE?
> 
> 
> Embarrassingly, we never filed a formal bug on this -- it was only tracked
> as a pre-integration project bug.  I''ll get a bug filed and put
you down
> as the call record...
> 
> Question for you (plural):  when you''re doing a printa() of
multiple
> aggregations, what should the sort order be?  We could obviously sum across
> the values to derive a single sort value, but I''m not sure if
that''s what''s
> desired -- should we be printing the aggregation in key-order instead of
> value-order for this case?  
> 
> 	- Bryan
> 
> --------------------------------------------------------------------------
> Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
Would you do a union of all keys or the intersection?  Eg if two aggregations
had some differences in their keys, how would this work?

- Bart

-- 
Bart Smaalders			Solaris Kernel Performance
barts at cyber.eng.sun.com		http://blogs.sun.com/barts

> Would you do a union of all keys or the intersection?  Eg if two 
> aggregations
> had some differences in their keys, how would this work?
It would be the union.  If a key is in one aggregation but not another,
the value would be zero for that aggregation''s column in the row 
corresponding to that key.

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

Bryan Cantrill wrote:
>>>And for whatever it''s worth, after seeing your mail this
morning, I spent
>>>my morning commute thinking about how to allow printa() to take
multiple
>>>aggregations with the same key signatures.  (This is an RFE that
we''ve
>>>had for a while -- the first time this was formally logged as a
request
>>>was on April 18th, 2003; it''s time to address it.)   It
shouldn''t be _too_
>>>bad; we''ll keep you posted on our progress... (And
apologies for forcing
>>>you to postprocess the data in the meantime.)
>>>      
>>>
>>Hi Bryan, no problems about postprocessing. I don''t want to
forget my awk
>>and perl skills just because I have such a fantastic tool (dtrace) at my
>>disposal! I guess Jignesh and I would manage to put something together
:-)
>>
>>Btw, do you happen to know the bugid of that RFE?
>>    
>>
>
>Embarrassingly, we never filed a formal bug on this -- it was only tracked
>as a pre-integration project bug.  I''ll get a bug filed and put you
down
>as the call record...
>
>Question for you (plural):  when you''re doing a printa() of
multiple
>aggregations, what should the sort order be?  We could obviously sum across
>the values to derive a single sort value, but I''m not sure if
that''s what''s
>desired -- should we be printing the aggregation in key-order instead of
>value-order for this case?  
>  
>
Bryan,

Any chance of making this more general to allow access to
aggregation values in DTrace expressions? I had a case
where I wanted to do that recently, though I can''t remember
the specifics.

So something like:

    @mx = max(this->response_time);
    ...

Then in another clause:

tick-1sec
/@mx > 100/
{
    printf("max response time hit threshold:%d\n", at mx);
    ...
}

Jim

> >
> >Question for you (plural):  when you''re doing a printa() of
multiple
> >aggregations, what should the sort order be?  We could obviously sum
across
> >the values to derive a single sort value, but I''m not sure if
that''s what''s
> >desired -- should we be printing the aggregation in key-order instead
of
> >value-order for this case?  
> > 
> >
> Bryan,
> 
> Any chance of making this more general to allow access to
> aggregation values in DTrace expressions? I had a case
> where I wanted to do that recently, though I can''t remember
> the specifics.
> 
> So something like:
> 
>    @mx = max(this->response_time);
>    ...
> 
> Then in another clause:
> 
> tick-1sec
> /@mx > 100/
> {
>    printf("max response time hit threshold:%d\n", at mx);
>    ...
> }
That we can''t do.  Part of the magic of aggregations is that their
results are kept per-CPU -- so we have no way of knowing the true value
of the aggregation in probe context.  So for this kind of use, you should
use a global variable...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

On Fri, 9 Sep 2005, Bryan Cantrill wrote:
>
> Question for you (plural):  when you''re doing a printa() of
multiple
> aggregations, what should the sort order be?  We could obviously sum across
> the values to derive a single sort value, but I''m not sure if
that''s what''s
> desired -- should we be printing the aggregation in key-order instead of
> value-order for this case?
How about the last aggregation order? In that way, if I am to print for 
example two aggregations e.g count (in @c) and CPU time (in @t), I can 
sort simply by choosing relative positions of printa() arguments. I.e.

 	printa(format, @c, at t)

sorts by CPU time, whereas

 	printa(format, @t, @c)

sorts by the number of occurences.

Bye, Dragan

-- 
Dragan Cvetkovic,

To be or not to be is true. G. Boole    No it isn''t.  L. E. J. Brouwer

On Sun, Sep 11, 2005 at 11:10:51PM -0400, Dragan Cvetkovic
wrote:
> >Question for you (plural):  when you''re doing a printa() of
multiple
> >aggregations, what should the sort order be?  We could obviously sum
across
> >the values to derive a single sort value, but I''m not sure if
that''s what''s
> >desired -- should we be printing the aggregation in key-order instead
of
> >value-order for this case?
> 
> How about the last aggregation order? In that way, if I am to print for 
> example two aggregations e.g count (in @c) and CPU time (in @t), I can 
> sort simply by choosing relative positions of printa() arguments. I.e.
> 
> 	printa(format, @c, at t)
> 
> sorts by CPU time, whereas
> 
> 	printa(format, @t, @c)
> 
> sorts by the number of occurences.
I think we''d probably use the order the aggregations appear in the
printa()
call to specify how they''re filled into the given format. It might look
something like this:

    printa("%d %@u %s %@u", @a, @b);

Where @a and @b were two aggregations with keys of the form [int, string].

Just a possibility.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Sun, Sep 11, 2005 at 08:22:12PM -0700, Adam Leventhal
wrote:
> On Sun, Sep 11, 2005 at 11:10:51PM -0400, Dragan Cvetkovic wrote:
> > >Question for you (plural):  when you''re doing a printa()
of multiple
> > >aggregations, what should the sort order be?  We could obviously
sum across
> > >the values to derive a single sort value, but I''m not
sure if that''s what''s
> > >desired -- should we be printing the aggregation in key-order
instead of
> > >value-order for this case?
> > 
> > How about the last aggregation order? In that way, if I am to print
for
> > example two aggregations e.g count (in @c) and CPU time (in @t), I can
> > sort simply by choosing relative positions of printa() arguments. I.e.
> > 
> > 	printa(format, @c, at t)
> > 
> > sorts by CPU time, whereas
> > 
> > 	printa(format, @t, @c)
> > 
> > sorts by the number of occurences.
> 
> I think we''d probably use the order the aggregations appear in the
printa()
> call to specify how they''re filled into the given format. It might
look
> something like this:
> 
>     printa("%d %@u %s %@u", @a, @b);
> 
> Where @a and @b were two aggregations with keys of the form [int, string].
Exactly.  We don''t want to use the position to denote the ordering (or
at
least, I''d rather not) because it violates the principle of
least-surprise.
I''m leaning torwards always doing a value-based ordering (and coming up
with the value for a given key by summing across aggregations), and
then giving the option for a key-based ordering.  This option would 
apply more generally to all aggregation processing -- several have said
that the need for key-based ordering has come up before, and I for one
would like to use it to generate an easy flat profile of (say) cycle
counts within a function.  Is "#pragma D option aggsort=key" okay for
the
new option name?  Better suggestions?

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

How about giving capability to store values as follows?

@function[probefunction,tid,pid] = [ count(), sum(timestamp - self->ts)];

Then it will sort as it is being done correctly (also this allows more 
simpler representation of how you want to aggregate).

Regards,
Jignesh


Bryan Cantrill wrote:
>>>And for whatever it''s worth, after seeing your mail this
morning, I spent
>>>my morning commute thinking about how to allow printa() to take
multiple
>>>aggregations with the same key signatures.  (This is an RFE that
we''ve
>>>had for a while -- the first time this was formally logged as a
request
>>>was on April 18th, 2003; it''s time to address it.)   It
shouldn''t be _too_
>>>bad; we''ll keep you posted on our progress... (And
apologies for forcing
>>>you to postprocess the data in the meantime.)
>>>      
>>>
>>Hi Bryan, no problems about postprocessing. I don''t want to
forget my awk
>>and perl skills just because I have such a fantastic tool (dtrace) at my
>>disposal! I guess Jignesh and I would manage to put something together
:-)
>>
>>Btw, do you happen to know the bugid of that RFE?
>>    
>>
>
>Embarrassingly, we never filed a formal bug on this -- it was only tracked
>as a pre-integration project bug.  I''ll get a bug filed and put you
down
>as the call record...
>
>Question for you (plural):  when you''re doing a printa() of
multiple
>aggregations, what should the sort order be?  We could obviously sum across
>the values to derive a single sort value, but I''m not sure if
that''s what''s
>desired -- should we be printing the aggregation in key-order instead of
>value-order for this case?  
>
>	- Bryan
>
>--------------------------------------------------------------------------
>Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>  
>
-- 
______________________________

Jignesh K. Shah
MTS Software Engineer, 
MDE - Horizontal Technologies 
Sun Microsystems, Inc
Phone: (781) 442 3052
Email: J.K.Shah at sun.com
______________________________

Hi Bryan,

Maybe I''m in low power mode today but I''m having a job
seeing how value-based odering helps a lot here (maybe my
use cases are just non representative).

A standard case is where I have two aggregations with exactly
the same keys, one storing a count of something (i.e syscalls) and
one storing the time taken to do these operations. Sometimes I may
want to order my results by total calls, sometimes by time taken
and sometimes by key but I never want to order by the sum of the
count and time (as it means nothing useful). For this kind of
use I personally like position based ordering.

If I''m way off the mark could you post a small example of
what you mean?


Cheers.

Jon.
> Exactly.  We don''t want to use the position to denote the ordering
(or at
> least, I''d rather not) because it violates the principle of
least-surprise.
> I''m leaning torwards always doing a value-based ordering (and
coming up
> with the value for a given key by summing across aggregations), and
> then giving the option for a key-based ordering.  This option would 
> apply more generally to all aggregation processing -- several have said
> that the need for key-based ordering has come up before, and I for one
> would like to use it to generate an easy flat profile of (say) cycle
> counts within a function.  Is "#pragma D option aggsort=key" okay
for the
> new option name?  Better suggestions?
> 
> 	- Bryan
> 
> --------------------------------------------------------------------------
> Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

On Sun, 11 Sep 2005, Bryan Cantrill wrote:
> then giving the option for a key-based ordering.  This option would
> apply more generally to all aggregation processing -- several have said
> that the need for key-based ordering has come up before, and I for one
> would like to use it to generate an easy flat profile of (say) cycle
> counts within a function.  Is "#pragma D option aggsort=key" okay
for the
> new option name?  Better suggestions?
Isn''t that one global i.e. wouldn''t it then apply to all
printa()''s? What
if I would like to use different ordering in different printa()s?

Bye, Dragan

P.S. Sorry if I misunderstood you, I am currently upgrading my computer 
(to snv22) and am telneting to our mail server to read the mail.

-- 
Dragan Cvetkovic,

To be or not to be is true. G. Boole    No it isn''t.  L. E. J. Brouwer

Hey Jon,
> A standard case is where I have two aggregations with exactly
> the same keys, one storing a count of something (i.e syscalls) and
> one storing the time taken to do these operations. Sometimes I may
> want to order my results by total calls, sometimes by time taken
> and sometimes by key but I never want to order by the sum of the
> count and time (as it means nothing useful). 
Hmmm...touche...
> For this kind of use I personally like position based ordering.
Okay.  Perhaps all of the cases where I want to sum across them will
be handled when we support multiple aggregating actions (which won''t
be a part of this work, but which we plan to ultimately do).  An example
of this might be aggregating by both sum of bytes and a quantization
of the size -- to save me from having to do the mental calculation when
looking at quantize output.  But I suppose even that case could just
be handled by position-based ordering.  Does position-based ordering
_not_ violate the principle of least-surprise?  (And we could always
default to position-based and then provide an option to have some other
ordering.)

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

>Okay.  Perhaps all of the cases where I want to sum across them will
>be handled when we support multiple aggregating actions (which
won''t
>be a part of this work, but which we plan to ultimately do).  An example
>of this might be aggregating by both sum of bytes and a quantization
>of the size -- to save me from having to do the mental calculation when
>looking at quantize output.  But I suppose even that case could just
>be handled by position-based ordering.  Does position-based ordering
>_not_ violate the principle of least-surprise?  (And we could always
>default to position-based and then provide an option to have some other
>ordering.)
>  
>
I''ll have a further think about it but, at the moment, I think that 
position-based
provides good functionality and also a least-surprise default behaviour.

I guess you could always have 3 options to the aggsort #pragma; 
position(default),
value and key?


Cheers.

Jon.

On Tue, Sep 13, 2005 at 11:00:21PM +0100, Jonathan Haslam
wrote:
> 
> >Okay.  Perhaps all of the cases where I want to sum across them will
> >be handled when we support multiple aggregating actions (which
won''t
> >be a part of this work, but which we plan to ultimately do).  An
example
> >of this might be aggregating by both sum of bytes and a quantization
> >of the size -- to save me from having to do the mental calculation when
> >looking at quantize output.  But I suppose even that case could just
> >be handled by position-based ordering.  Does position-based ordering
> >_not_ violate the principle of least-surprise?  (And we could always
> >default to position-based and then provide an option to have some other
> >ordering.)
> > 
> >
> 
> I''ll have a further think about it but, at the moment, I think
that
> position-based
> provides good functionality and also a least-surprise default behaviour.
> 
> I guess you could always have 3 options to the aggsort #pragma; 
> position(default),
> value and key?
I think I''m going to introduce "#pragma D option
aggsort=key", but I
would _really_ rather not have an additional position sorting argument.
Not only would it be arcane, it wouldn''t even be complete:  presumably
one would want to additionally define an arbitrary positioning order
for breaking ties.  It''s doable, but it''s a mess (mainly,
it''s a mess
to test).  So I''m just going to leave it as strict positioning, with
ties resolved in positioning-order.  (That is, if the first two
aggregation values for two keys are equal, the second two aggregation
values are checked and so on.  If (and only if) all aggregation values
are equal, the keys will be checked as the final arbiter.

Does that sit well with everyone?

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc