dtrace discuss - Jun 2009 - Possible to read current value of aggregate?

Hi all, I''m trying to gather some statistics which dtrace can return to
the traced app upon request, and it would be really nice to be able to 
do something like this:

syscall::read:entry /pid == $target/ { @syscalls["read"] = count(); }
pid$target::ask_dtrace_for_stats:entry {
    this->read_buf = (int*) alloca(sizeof(int));
    *this->read_buf = @syscalls["read"];
    copyout(this->read_buf, arg0, sizeof(int));
}

Unfortunately, compiling the script gives the error that
> operands have incompatible types: "int" = "aggregation"
So, three questions:

   1. Is there a way to read the aggregate''s value directly from a
      script, as opposed to parsing the output of a printa() at some
      later point?
   2. Would the script perform well if I replaced
      @syscalls["read"]=count() with global_reads[cpu]++ and manually
      summed up the array entries when needed? The dire warnings in the
      documentation suggest the answer is "no" ... and it would be a
      royal pain in the neck just to reinvent the wheel.
   3. If the answers to (1) and (2) are both "no" could dtrace
      reasonably be extended to allow reading from aggregates in
      scripts? It seems that whatever printa() does to extract the value
      in a cheap/scalable/no-control-flow way would work just as well
      for direct access.

Since my app already forks off the dtrace script anyway, the alternative 
would be to make the script call printa() when requested, then scanf() 
the result from dtrace''s stdout, but again that seems like taking the 
long way around, with four trips between userland and the kernel: 
app->dtrace, dtrace->dtrace_client, dtrace_client->syscall_write, 
app->syscall_read.

Thoughts?
Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20090626/f8b4159a/attachment.html>

Hi Ryan,

You need to use libdtrace(3LIB) to get this done in a clean manner.
There are a bunch of tools in Solaris that you can look at for examples
of how to use the interfaces. These include plockstat(1M), instrat(1M),
lockstat(1M) and powertop(1M) to name a few.

Jon.
> Hi all, I''m trying to gather some statistics which dtrace can
return to
> the traced app upon request, and it would be really nice to be able to 
> do something like this:
> 
> syscall::read:entry /pid == $target/ { @syscalls["read"] =
count(); }
> pid$target::ask_dtrace_for_stats:entry {
>     this->read_buf = (int*) alloca(sizeof(int));
>     *this->read_buf = @syscalls["read"];
>     copyout(this->read_buf, arg0, sizeof(int));
> }
> 
> Unfortunately, compiling the script gives the error that
>> operands have incompatible types: "int" =
"aggregation"
> So, three questions:
> 
>    1. Is there a way to read the aggregate''s value directly from a
>       script, as opposed to parsing the output of a printa() at some
>       later point?
>    2. Would the script perform well if I replaced
>       @syscalls["read"]=count() with global_reads[cpu]++ and
manually
>       summed up the array entries when needed? The dire warnings in the
>       documentation suggest the answer is "no" ... and it would
be a
>       royal pain in the neck just to reinvent the wheel.
>    3. If the answers to (1) and (2) are both "no" could dtrace
>       reasonably be extended to allow reading from aggregates in
>       scripts? It seems that whatever printa() does to extract the value
>       in a cheap/scalable/no-control-flow way would work just as well
>       for direct access.
> 
> Since my app already forks off the dtrace script anyway, the alternative 
> would be to make the script call printa() when requested, then scanf() 
> the result from dtrace''s stdout, but again that seems like taking
the
> long way around, with four trips between userland and the kernel: 
> app->dtrace, dtrace->dtrace_client, dtrace_client->syscall_write, 
> app->syscall_read.
> 
> Thoughts?
> Ryan
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

Hi Jon,

I toyed with using libdtrace earlier, and from what I understand,

   1. Bad Things can happen if I use libdtrace and my process doesn''t
      exit cleanly (e.g. killed by debugger). Scary! This is why my app
      currently forks dtrace(1M) instead of using libdtrace(3LIB) directly.
   2. Using libdtrace just makes me a consumer (like dtrace(1) is), but
      would still require me to parse the output of printa. Granted,
      cutting it down to two kernel round trips is an improvement, but
      still not particularly "clean" from my perspective....
especially
      if I wanted a script to actually use that value in some way
      (though in my current situation that''s not an issue)

Did I miss something?
Ryan

Jon Haslam wrote:
> Hi Ryan,
>
> You need to use libdtrace(3LIB) to get this done in a clean manner.
> There are a bunch of tools in Solaris that you can look at for examples
> of how to use the interfaces. These include plockstat(1M), instrat(1M),
> lockstat(1M) and powertop(1M) to name a few.
>
> Jon.
>
>> Hi all, I''m trying to gather some statistics which dtrace can
return
>> to the traced app upon request, and it would be really nice to be 
>> able to do something like this:
>>
>> syscall::read:entry /pid == $target/ { @syscalls["read"] =
count(); }
>> pid$target::ask_dtrace_for_stats:entry {
>>     this->read_buf = (int*) alloca(sizeof(int));
>>     *this->read_buf = @syscalls["read"];
>>     copyout(this->read_buf, arg0, sizeof(int));
>> }
>>
>> Unfortunately, compiling the script gives the error that
>>> operands have incompatible types: "int" =
"aggregation"
>> So, three questions:
>>
>>    1. Is there a way to read the aggregate''s value directly
from a
>>       script, as opposed to parsing the output of a printa() at some
>>       later point?
>>    2. Would the script perform well if I replaced
>>       @syscalls["read"]=count() with global_reads[cpu]++ and
manually
>>       summed up the array entries when needed? The dire warnings in the
>>       documentation suggest the answer is "no" ... and it
would be a
>>       royal pain in the neck just to reinvent the wheel.
>>    3. If the answers to (1) and (2) are both "no" could
dtrace
>>       reasonably be extended to allow reading from aggregates in
>>       scripts? It seems that whatever printa() does to extract the
value
>>       in a cheap/scalable/no-control-flow way would work just as well
>>       for direct access.
>>
>> Since my app already forks off the dtrace script anyway, the 
>> alternative would be to make the script call printa() when requested, 
>> then scanf() the result from dtrace''s stdout, but again that
seems
>> like taking the long way around, with four trips between userland and 
>> the kernel: app->dtrace, dtrace->dtrace_client, 
>> dtrace_client->syscall_write, app->syscall_read.
>>
>> Thoughts?
>> Ryan
>>
>>
>>
------------------------------------------------------------------------
>>
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>

>   1. Bad Things can happen if I use libdtrace and my process
doesn''t
>      exit cleanly (e.g. killed by debugger). Scary! This is why my app
>      currently forks dtrace(1M) instead of using libdtrace(3LIB) directly.
Can you provide any specifics regarding these bad things?
>   2. Using libdtrace just makes me a consumer (like dtrace(1) is), but
>      would still require me to parse the output of printa. Granted,
>      cutting it down to two kernel round trips is an improvement, but
>      still not particularly "clean" from my perspective....
especially
>      if I wanted a script to actually use that value in some way
>      (though in my current situation that''s not an issue)
Using libdtrace gives you the cleanest method to do this with. There
are other mechanisms which you could use to periodically import
performance data which would work and I''m sure others may well
chime in with suggestions. Yes, with libdtrace you would have to do
some leg work yourself with coding but the joy and pleasure it
will give you will far outweigh any pain you may have encountered
along the way :-) .

Have a go. You know you want to.

Jon.
> 
> Did I miss something?
> Ryan
> 
> Jon Haslam wrote:
>> Hi Ryan,
>>
>> You need to use libdtrace(3LIB) to get this done in a clean manner.
>> There are a bunch of tools in Solaris that you can look at for examples
>> of how to use the interfaces. These include plockstat(1M), instrat(1M),
>> lockstat(1M) and powertop(1M) to name a few.
>>
>> Jon.
>>
>>> Hi all, I''m trying to gather some statistics which dtrace
can return
>>> to the traced app upon request, and it would be really nice to be 
>>> able to do something like this:
>>>
>>> syscall::read:entry /pid == $target/ { @syscalls["read"]
= count(); }
>>> pid$target::ask_dtrace_for_stats:entry {
>>>     this->read_buf = (int*) alloca(sizeof(int));
>>>     *this->read_buf = @syscalls["read"];
>>>     copyout(this->read_buf, arg0, sizeof(int));
>>> }
>>>
>>> Unfortunately, compiling the script gives the error that
>>>> operands have incompatible types: "int" =
"aggregation"
>>> So, three questions:
>>>
>>>    1. Is there a way to read the aggregate''s value
directly from a
>>>       script, as opposed to parsing the output of a printa() at
some
>>>       later point?
>>>    2. Would the script perform well if I replaced
>>>       @syscalls["read"]=count() with global_reads[cpu]++
and manually
>>>       summed up the array entries when needed? The dire warnings in
the
>>>       documentation suggest the answer is "no" ... and it
would be a
>>>       royal pain in the neck just to reinvent the wheel.
>>>    3. If the answers to (1) and (2) are both "no" could
dtrace
>>>       reasonably be extended to allow reading from aggregates in
>>>       scripts? It seems that whatever printa() does to extract the
value
>>>       in a cheap/scalable/no-control-flow way would work just as
well
>>>       for direct access.
>>>
>>> Since my app already forks off the dtrace script anyway, the 
>>> alternative would be to make the script call printa() when
requested,
>>> then scanf() the result from dtrace''s stdout, but again
that seems
>>> like taking the long way around, with four trips between userland
and
>>> the kernel: app->dtrace, dtrace->dtrace_client, 
>>> dtrace_client->syscall_write, app->syscall_read.
>>>
>>> Thoughts?
>>> Ryan
>>>
>>>
>>>
------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> dtrace-discuss mailing list
>>> dtrace-discuss at opensolaris.org
>>
>

Jon Haslam wrote:
>
>>   1. Bad Things can happen if I use libdtrace and my process
doesn''t
>>      exit cleanly (e.g. killed by debugger). Scary! This is why my app
>>      currently forks dtrace(1M) instead of using libdtrace(3LIB) 
>> directly.
>
> Can you provide any specifics regarding these bad things?
Well, libdtrace is rather underdocumented, but according to the folks 
behind TclDTrace, "Note that *dtrace_close* is needed to prevent an epic 
fail. It does a major cleanup, that involves restoring the state of all 
grabbed processes, which may be all processes in the system..."

Now, if somebody with internal knowledge could reassure us all that, 
say, the kernel always calls dtrace_close during process cleanup, then 
libdtrace becomes much more attractive.
>
>>   2. Using libdtrace just makes me a consumer (like dtrace(1) is), but
>>      would still require me to parse the output of printa. Granted,
>>      cutting it down to two kernel round trips is an improvement, but
>>      still not particularly "clean" from my perspective....
especially
>>      if I wanted a script to actually use that value in some way
>>      (though in my current situation that''s not an issue)
>
> Using libdtrace gives you the cleanest method to do this with. There
> are other mechanisms which you could use to periodically import
> performance data which would work and I''m sure others may well
> chime in with suggestions. Yes, with libdtrace you would have to do
> some leg work yourself with coding but the joy and pleasure it
> will give you will far outweigh any pain you may have encountered
> along the way :-) .
>
> Have a go. You know you want to.
For now I agree it''s the cleanest approach... but is it feasible to 
implement the ability to just read the aggregate directly inside a 
script? You know you want to ;)

Ryan

>> Can you provide any specifics regarding these bad things?
> Well, libdtrace is rather underdocumented, but according to the folks 
> behind TclDTrace, "Note that *dtrace_close* is needed to prevent an
epic
> fail. It does a major cleanup, that involves restoring the state of all 
> grabbed processes, which may be all processes in the system..."
> 
> Now, if somebody with internal knowledge could reassure us all that, 
> say, the kernel always calls dtrace_close during process cleanup, then 
> libdtrace becomes much more attractive.
It is true that you need to call dtrace_close() to release children
you are controlling. Look at the source for dtrace(1M) or plockstat(1M)
and you''ll see that the have signal handlers setup which call into
dtrace_close for SIGTERM and SIGINT.

However, why are you worried about leaving children in an odd state?
By the look of the example you gave you''re not controlling children
are you? You just want to compile a D script and read the data
periodically and do something with that. If so, you''re just doing
regular consumer activities - it doesn''t matter that the script you
injected is  interested in the consumer. You''re not dealing with victim
processes there''s no need to worry about the subtleties of that.

Jon.

On Fri, Jun 26, 2009 at 03:15:16PM +0100, Jon Haslam
wrote:
> 
> >>Can you provide any specifics regarding these bad things?
> >Well, libdtrace is rather underdocumented, but according to the folks 
> >behind TclDTrace, "Note that *dtrace_close* is needed to prevent
an epic
> >fail. It does a major cleanup, that involves restoring the state of all
> >grabbed processes, which may be all processes in the system..."
> >
> >Now, if somebody with internal knowledge could reassure us all that, 
> >say, the kernel always calls dtrace_close during process cleanup, then 
> >libdtrace becomes much more attractive.
> 
> It is true that you need to call dtrace_close() to release children
> you are controlling. Look at the source for dtrace(1M) or plockstat(1M)
> and you''ll see that the have signal handlers setup which call into
> dtrace_close for SIGTERM and SIGINT.
> 
> However, why are you worried about leaving children in an odd state?
> By the look of the example you gave you''re not controlling
children
> are you? You just want to compile a D script and read the data
> periodically and do something with that. If so, you''re just doing
> regular consumer activities - it doesn''t matter that the script
you
> injected is  interested in the consumer. You''re not dealing with
victim
> processes there''s no need to worry about the subtleties of that.
In particular, this shouldn''t be an issue unless you are enabling
foo12345:::
probes (that is, any providers with PIDs in them)

Cheers,
- jonathan

Jonathan Adams wrote:

On Fri, Jun 26, 2009 at 03:15:16PM +0100, Jon Haslam wrote:

Can you provide any specifics regarding these bad things?

Well, libdtrace is rather underdocumented, but according to the folks 
behind TclDTrace, "Note that *dtrace_close* is needed to prevent an epic 
fail. It does a major cleanup, that involves restoring the state of all 
grabbed processes, which may be all processes in the system..."

Now, if somebody with internal knowledge could reassure us all that, 
say, the kernel always calls dtrace_close during process cleanup, then 
libdtrace becomes much more attractive.

It is true that you need to call dtrace_close() to release children
you are controlling. Look at the source for dtrace(1M) or plockstat(1M)
and you''ll see that the have signal handlers setup which call into
dtrace_close for SIGTERM and SIGINT.

However, why are you worried about leaving children in an odd state?
By the look of the example you gave you''re not controlling children
are you? You just want to compile a D script and read the data
periodically and do something with that. If so, you''re just doing
regular consumer activities - it doesn''t matter that the script you
injected is  interested in the consumer. You''re not dealing with victim
processes there''s no need to worry about the subtleties of that.

In particular, this shouldn''t be an issue unless you are enabling
foo12345:::
probes (that is, any providers with PIDs in them)

Ah, but I am... that was the way for my app to tell dtrace it wants the
latest stats... the question was how to get the answer back since I
can''t copyout() the aggregation''s value.

DTrace isn''t "controlling" the process, though, at least not
by the
pid-grabbing-proc-locking-debugger-excluding definition. Is it
controlling a process that requires cleanup or use of the pid provider?
If it''s only the latter, what happens if the dtrace consumer and the
process being traced are the same? A not-clean process exit would take
both the consumer and the process with, so unless there''s stuff to
clean up in the kernel...

Ryan

>> In particular, this shouldn''t be an issue unless you are
enabling foo12345:::
>> probes (that is, any providers with PIDs in them)
>>   
> Ah, but I am... that was the way for my app to tell dtrace it wants the 
> latest stats... the question was how to get the answer back since I 
> can''t copyout() the aggregation''s value.
But I don''t think you are. If all you are doing is:

syscall:::entry
/pid == 1234/
{
	@[probefunc] = count();
}

then all you need to do is call into dtrace periodically from
your consumer and consume the data. No need to have a victim to
do that.

I think the confusion may be that, originally, you were wanting to
have some mechanism where you had a probe (pid$target::foo:entry)
from which you could extract data that was being collected via
a copyout? I''ve got the heebie-jeebies just thinking about it.
This just isn''t the way to do it and it''s not going to work.
The
consumer approach is what you need if the clause above is the
kind of thing you''re after. Honest.

Jon.

On Fri, Jun 26, 2009 at 04:42:11PM +0200, Ryan Johnson
wrote:
> 
>    Ah, but I am... that was the way for my app to tell dtrace it wants the
>    latest stats... the question was how to get the answer back since I
can''t
>    copyout() the aggregation''s value.
> 
>    DTrace isn''t "controlling" the process, though, at
least not by the
>    pid-grabbing-proc-locking-debugger-excluding definition. Is it
controlling
>    a process that requires cleanup or use of the pid provider? If
it''s only
>    the latter, what happens if the dtrace consumer and the process being
>    traced are the same? A not-clean process exit would take both the
consumer
>    and the process with, so unless there''s stuff to clean up in
the kernel...
Why not make your application link against libdtrace?  Why all the additional
work?

Cheers,
- jonathan

Jon Haslam wrote:
>>> In particular, this shouldn''t be an issue unless you are
enabling
>>> foo12345:::
>>> probes (that is, any providers with PIDs in them)
>>>   
>> Ah, but I am... that was the way for my app to tell dtrace it wants 
>> the latest stats... the question was how to get the answer back since 
>> I can''t copyout() the aggregation''s value.
>
> But I don''t think you are. If all you are doing is:
>
> syscall:::entry
> /pid == 1234/
> {
>     @[probefunc] = count();
> }
>
> then all you need to do is call into dtrace periodically from
> your consumer and consume the data. No need to have a victim to
> do that.
>
> I think the confusion may be that, originally, you were wanting to
> have some mechanism where you had a probe (pid$target::foo:entry)
> from which you could extract data that was being collected via
> a copyout? I''ve got the heebie-jeebies just thinking about it.
> This just isn''t the way to do it and it''s not going to
work. The
> consumer approach is what you need if the clause above is the
> kind of thing you''re after. Honest.
So let me get this straight... if I link against libdtrace and become a 
first-class consumer, I can just ask for the latest value of @ at any 
time and get it? I thought consumers could only consume -- that the 
script had to printa() first -- and the pid$target was the way for my 
app to trigger the printa().

Hmm... re-reading the wiki entry on aggregates, it looks like the 
aggregate data is transferred to userland at a rate controlled by 
''aggrate'' -- so I guess I would set aggrate to
"trigger" output instead
of using a hi-res timer. Is that right?

Also, is there any documentation for libdtrace other than the programs 
that use it?

Ryan

Sorry, one more question... is it safe to fire up multiple dtrace_hdl_t 
within a process?

Ryan

Jon Haslam wrote:
>
>
>>> In particular, this shouldn''t be an issue unless you are
enabling
>>> foo12345:::
>>> probes (that is, any providers with PIDs in them)
>>>   
>> Ah, but I am... that was the way for my app to tell dtrace it wants 
>> the latest stats... the question was how to get the answer back since 
>> I can''t copyout() the aggregation''s value.
>
> But I don''t think you are. If all you are doing is:
>
> syscall:::entry
> /pid == 1234/
> {
>     @[probefunc] = count();
> }
>
> then all you need to do is call into dtrace periodically from
> your consumer and consume the data. No need to have a victim to
> do that.
>
> I think the confusion may be that, originally, you were wanting to
> have some mechanism where you had a probe (pid$target::foo:entry)
> from which you could extract data that was being collected via
> a copyout? I''ve got the heebie-jeebies just thinking about it.
> This just isn''t the way to do it and it''s not going to
work. The
> consumer approach is what you need if the clause above is the
> kind of thing you''re after. Honest.
>
> Jon.

On Fri, Jun 26, 2009 at 11:55 AM, Ryan Johnson<ryanjohn at ece.cmu.edu>
wrote:
>
> Also, is there any documentation for libdtrace other than the programs that
> use it?
The tools are a good start.  Especially intrstat for what you want to
to, as it consumes aggregation data to print out.

The second place is /usr/include/dtrace.h, look for "DTrace Program
Interface".  Not really documentation per se, but pretty useful once
you start looking at the code for the utilities.  Look for
dtrace_aggregate_snap to get a feel for what libdtrace provides you
for handling aggregates.

If you''d like, I could send you a stripped down consumer so you
don''t
have to worry much about all the baggage in the code for other
utilities.

Chad

> Sorry, one more question... is it safe to fire up multiple dtrace_hdl_t 
> within a process?
Sure, a consumer can do multiple opens.

Chad''s advice is good (as always). Especially the bit about
offering to send you a stripped down consumer :-) .

Jon.

> 
> Ryan
> 
> Jon Haslam wrote:
>>
>>
>>>> In particular, this shouldn''t be an issue unless you
are enabling
>>>> foo12345:::
>>>> probes (that is, any providers with PIDs in them)
>>>>   
>>> Ah, but I am... that was the way for my app to tell dtrace it wants
>>> the latest stats... the question was how to get the answer back
since
>>> I can''t copyout() the aggregation''s value.
>>
>> But I don''t think you are. If all you are doing is:
>>
>> syscall:::entry
>> /pid == 1234/
>> {
>>     @[probefunc] = count();
>> }
>>
>> then all you need to do is call into dtrace periodically from
>> your consumer and consume the data. No need to have a victim to
>> do that.
>>
>> I think the confusion may be that, originally, you were wanting to
>> have some mechanism where you had a probe (pid$target::foo:entry)
>> from which you could extract data that was being collected via
>> a copyout? I''ve got the heebie-jeebies just thinking about it.
>> This just isn''t the way to do it and it''s not going
to work. The
>> consumer approach is what you need if the clause above is the
>> kind of thing you''re after. Honest.
>>
>> Jon.
>

Chad Mynhier wrote:
> If you''d like, I could send you a stripped down consumer so you
don''t
> have to worry much about all the baggage in the code for other
> utilities.
>   
That would be really helpful! The dtrace code is a bit bewildering to 
take on all at once...

Thanks,
Ryan

On Fri, Jun 26, 2009 at 2:49 PM, Ryan Johnson<ryanjohn at ece.cmu.edu>
wrote:
> Chad Mynhier wrote:
>>
>> If you''d like, I could send you a stripped down consumer so
you don''t
>> have to worry much about all the baggage in the code for other
>> utilities.
>>
>
> That would be really helpful! The dtrace code is a bit bewildering to take
> on all at once...
>
I''ll post it to the list for posterity''s sake, and so anyone
else can
point out any flaws.  Compile with "cc consumer.c -ldtrace".

-----
#include <dtrace.h>
#include <stdarg.h>
#include <stdlib.h>
#include <strings.h>

static char *g_prog = ""
"profile-997"
"/arg0 && curthread->t_pri != -1/"
"{"
"       @c[stack()] = count();"
"}"
"END"
"{"
"       trunc(@c,30);"
"}";

static dtrace_hdl_t *g_dtp;

static void
fatal(const char *fmt, ...)
{
        va_list ap;

        va_start(ap, fmt);

        (void) vfprintf(stderr, fmt, ap);

        if (fmt[strlen(fmt) - 1] != ''\n'')
                (void) fprintf(stderr, ": %s\n",
                    dtrace_errmsg(g_dtp, dtrace_errno(g_dtp)));

        exit(EXIT_FAILURE);
}

static int
chewrec(const dtrace_probedata_t *data, const dtrace_recdesc_t *rec, void *arg)
{
        return (DTRACE_CONSUME_THIS);
}

static int
chew(const dtrace_probedata_t *data, void *arg)
{
        return (DTRACE_CONSUME_THIS);
}

int
main()
{
        dtrace_prog_t *prog;
        dtrace_proginfo_t info;
        dtrace_optval_t statustime;
        int err;

        if ((g_dtp = dtrace_open(DTRACE_VERSION, 0, &err)) == NULL)
                fatal("cannot open dtrace library: %s\n",
                    dtrace_errmsg(NULL, err));

        if ((prog = dtrace_program_strcompile(g_dtp, g_prog,
            DTRACE_PROBESPEC_NAME, 0, 0, NULL)) == NULL)
                fatal("invalid program");

        if (dtrace_program_exec(g_dtp, prog, &info) == -1)
                fatal("failed to enable probes");

        if (dtrace_setopt(g_dtp, "aggsize", "512k") == -1)
                fatal("failed to set aggsize");

        if (dtrace_go(g_dtp) != 0)
                fatal("dtrace_go()");

        for (int i = 0; i < 10; i++) {

                dtrace_sleep(g_dtp);

                switch (dtrace_work(g_dtp, stdout, chew, chewrec, NULL)) {
                case DTRACE_WORKSTATUS_DONE:
                        break;
                case DTRACE_WORKSTATUS_OKAY:
                        break;
                default:
                        fatal("processing aborted");
                }
        }

        if (dtrace_aggregate_print(g_dtp, stdout, NULL) == -1)
                fatal("failed to print aggregation");

        dtrace_close(g_dtp);
        return (0);
}
-----

Some notes:
- This only sets aggsize, it doesn''t set bufsize.  If it were
generating any non-aggregation data, it would need bufsize to be set.
- The loop calls dtrace_work().  If you look at intrstat(1M), you''ll
see that it calls dtrace_status() and dtrace_aggregate_snap() instead.
 dtrace_work() does it all for you, or you can break out the steps
that it takes like intrstat(1M) is doing.  It''s the responsibility of
the consumer to check the status and consume data on a regular basis.
For non-aggregation data you''d want dtrace_consume() if you
don''t just
use dtrace_work().
- I''m doing the dtrace_aggregate_print() at the end, similar to
dtrace(1M).  The code for intrstat(1M) is a good example of handling
the aggregation data incrementally.  This is in the loop at the bottom
of intrstat.c, where it calls dtrace_aggregate_snap(),
dtrace_aggregate_walk_keyvarsorted() and dtrace_aggregate_clear().
-  If you haven''t run across it yet,
http://src.opensolaris.org/source/ is a source browser for the
OpenSolaris source.

Chad