dtrace discuss - Oct 2008 - C preprocessor issues - how can I help?

I am using the "-C" option to invoke the C preprocessor and include
Apache''s httpd.h in a dtrace script.  This lets me dereference
Apache''s request_rec struct and pull some great per-request details.

But it won''t work "out of the box."  Dtrace (or the D
compiler in libdtrace) fails to parse the cpp output.  I have to run cpp
manually on httpd.h, saving the output to a monolithic dot-h file and then
iteratively remove lines from that file when Dtrace complains about them until I
have a "working" file.  After that, it''s just awesome.

I guess my question is, is there any Dtrace debug flag that will give me more
information on the parse error?  Where are the "-x" (compiler and
tracing) options defined?  If I understood the parse error (found ABC but
expecting XYZ), maybe I could contribute something back.

Here''s all I see now:

$ sudo dtrace -32 -C -I /export/home/amitchel/include -s
watchApacheRequestsTestingCPP.d -p 16770
dtrace: failed to compile script watchApacheRequestsTestingCPP.d:
"/export/home/amitchel/include/apr_thread_mutex.h", line 100: syntax
error near "#"


Line 100 is:
     APR_POOL_DECLARE_ACCESSOR(thread_mutex);


My script is:

$ cat watchApacheRequestsTestingCPP.d 
#!/usr/sbin/dtrace

#include <httpd.h>

pid$target:a.out:ap_process_request:entry
{

        self->ts = timestamp;
	self->r = (request_rec*) copyin(arg0, sizeof(request_rec));
        self->uri = copyinstr((uintptr_t) self->r->uri);

}


pid$target:a.out:ap_process_request:return
/self->ts/
{
        @request_uris[self->uri] = quantize((timestamp - self->ts) /
1000000);
        self->ts = 0;
        self->uri = "err";
}

tick-5s 
{
	printa(@request_uris);
}
--
This message posted from opensolaris.org

On Fri, Oct 24, 2008 at 01:04:41PM -0700, Adam Mitchell
wrote:
> I am using the "-C" option to invoke the C preprocessor and
include
> Apache''s httpd.h in a dtrace script.  This lets me dereference
> Apache''s request_rec struct and pull some great per-request
details.
>
> But it won''t work "out of the box."  Dtrace (or the D
compiler in
> libdtrace) fails to parse the cpp output.  I have to run cpp manually
> on httpd.h, saving the output to a monolithic dot-h file and then
> iteratively remove lines from that file when Dtrace complains about
> them until I have a "working" file.  After that, it''s
just awesome.
>
> I guess my question is, is there any Dtrace debug flag that will give
> me more information on the parse error?  Where are the "-x"
(compiler
> and tracing) options defined?  If I understood the parse error (found
> ABC but expecting XYZ), maybe I could contribute something back.
The D compiler defaults to the pre-ansi-C precompiler /usr/ccs/lib/cpp,
since that is the only one guaranteed to be on the system.  Since it is
pre-ANSI C, constructs like "#" and "##" will confuse it.


You can specify a different preprocessor using the -xcpppath=/path/to/cpp

Cheers,
- jonathan
>
> Here''s all I see now:
> 
> $ sudo dtrace -32 -C -I /export/home/amitchel/include -s
watchApacheRequestsTestingCPP.d -p 16770
> dtrace: failed to compile script watchApacheRequestsTestingCPP.d:
"/export/home/amitchel/include/apr_thread_mutex.h", line 100: syntax
error near "#"
> 
> 
> Line 100 is:
>      APR_POOL_DECLARE_ACCESSOR(thread_mutex);
> 
> 
> My script is:
> 
> $ cat watchApacheRequestsTestingCPP.d 
> #!/usr/sbin/dtrace
> 
> #include <httpd.h>
> 
> pid$target:a.out:ap_process_request:entry
> {
> 
>         self->ts = timestamp;
> 	self->r = (request_rec*) copyin(arg0, sizeof(request_rec));
>         self->uri = copyinstr((uintptr_t) self->r->uri);
> 
> }
> 
> 
> pid$target:a.out:ap_process_request:return
> /self->ts/
> {
>         @request_uris[self->uri] = quantize((timestamp - self->ts) /
1000000);
>         self->ts = 0;
>         self->uri = "err";
> }
> 
> tick-5s 
> {
> 	printa(@request_uris);
> }
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

Thanks, Jonathan.  I''ll try that on Monday with a different cpp.
--
This message posted from opensolaris.org

Thanks for that tip.  I can use the "-x cpppath=/usr/sfw/bin/cpp"
option and get further.

I see that option defined in:
http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/dt_options.c

Is there a way to pass generic arguments to cpp?  It looks like the
dt_opt_cpp_opts function is only called for: -D, -I, and -U

Thanks again,
Adam
--
This message posted from opensolaris.org

On Mon, Oct 27, 2008 at 01:23:19PM -0700, Adam Mitchell
wrote:
> Thanks for that tip.  I can use the "-x cpppath=/usr/sfw/bin/cpp"
option and get further.
> 
> I see that option defined in:
http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/dt_options.c
> 
> Is there a way to pass generic arguments to cpp?  It looks like the
dt_opt_cpp_opts function is only called for: -D, -I, and -U
> 
> Thanks again,
> Adam
There is no generic option at present (though easy to add)
but you can of course set cpppath to some shell script
which calls sfw/bin/cpp with some lengthy list of arguments
other than -D/I/U

-Mike

-- 
Mike Shapiro, Sun Microsystems Fishworks. blogs.sun.com/mws/

Okay - using GNU cpp gets me further, but now the D compiler complains here:

[ full output attached as dtrace.out ]
libdtrace DEBUG: typedef __builtin_va_list added as id 510
dtrace: failed to compile script watchApacheRequests.d:
"/usr/sfw/lib/gcc/i386-pc-solaris2.11/3.4.3/include/stdarg.h", line
43: syntax error near "__gnuc_va_list"

Mike - I found a post from 2005 where you were helping someone with the same
issue, but I think you guys went to private email to finish up.  Did you all
ever figure this out?: 
http://www.opensolaris.org/jive/thread.jspa?messageID=180530

Thanks,
Adam
-- 
This message posted from opensolaris.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dtrace.out
Type: application/octet-stream
Size: 32456 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20081028/cf5a2869/attachment.obj>

I know that "syntax error near" message comes from dt_lex.l, but I
can''t figure out what''s wrong with the input line...

typedef __builtin_va_list __gnuc_va_list;
-- 
This message posted from opensolaris.org

On Tue, Oct 28, 2008 at 12:24:03PM -0700, Adam Mitchell
wrote:
> I know that "syntax error near" message comes from dt_lex.l, but
I can''t figure out what''s wrong with the input line...
> 
> typedef __builtin_va_list __gnuc_va_list;
The problem is that __builtin_va_list is a predefined type in GCC, but not in
D.  You might be able to get away with something like:

	-D__builtin_va_list=long
or
	-D__builtin_va_list=''void*''

Thanks.  I''m getting ever closer.  I had to deal with __attribute__,
and now I think I''m using a reserved keyword:

# cat watchApacheRequests.d 
#!/usr/sbin/dtrace

#define __builtin_va_list long
#define __attribute__(x)
#include <httpd.h>
...


# dtrace -32 -C -I /export/home/amitchel/include -x cpppath=/usr/sfw/bin/cpp -Xs
-s watchApacheRequests.d -p 843
cc1: warning: /dev/fd/6 is shorter than expected
dtrace: failed to compile script watchApacheRequests.d:
"/export/home/amitchel/include/ap_regex.h", line 117: syntax error
near "string"


The offending line is:

AP_DECLARE(int) ap_regexec(const ap_regex_t *preg, const char *string,
                           apr_size_t nmatch, ap_regmatch_t *pmatch, int
eflags);



I know from
[http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/dt_lex.l]
that ''string'' is reserved.  Can you think of a trick that will
let me use it?  Obviously, I''m trying to avoid touching
Apache''s header files.

Thanks a lot for all the help.
-- 
This message posted from opensolaris.org

BTW, I replaced "string" with "Xstring" and it works.  If I
could avoid touching Apache''s files, that would be nice.  But I can
live with this.

Thanks for the help everybody!

I want to share this with anyone who will come looking for this later:

Copy the Apache header files to a work directory:

# cp -R /opt/apache/include /export/home/amitchel 


My command line:


dtrace -32 -C -I /export/home/amitchel/include -x cpppath=/usr/sfw/bin/cpp -Xs
-s watchApacheRequests.d -p 843

My script:
#!/usr/sbin/dtrace

#define __builtin_va_list long
#define __attribute__(x)
#include <httpd.h>

pid$target:a.out:ap_process_request:entry
{

        self->ts = timestamp;
        self->r = (request_rec*) copyin(arg0, sizeof(request_rec));
        self->uri = copyinstr((uintptr_t) self->r->uri);

}


pid$target:a.out:ap_process_request:return
/self->ts/
{
        @request_uris[self->uri] = quantize((timestamp - self->ts) /
1000000);
        self->ts = 0;
        self->uri = "err";
}

tick-5s 
{
        printa(@request_uris);
}



You''ll get some errors like this because of the
''string'' reserved keyword:

cc1: warning: /dev/fd/6 is shorter than expected
dtrace: failed to compile script watchApacheRequests.d:
"/export/home/amitchel/include/ap_regex.h", line 117: syntax error
near "string"


Edit the file, replace ''string'' with
''Xstring'' and try again.  I had to edit these files:

/export/home/amitchel/include/ap_regex.h
/export/home/amitchel/include/httpd.h


And here''s the script output:

  0  49008                         :tick-5s 
  /hello                                            
           value  ------------- Distribution ------------- count    
               1 |                                         0        
               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
               4 |                                         0        

  /page1                                            
           value  ------------- Distribution ------------- count    
               1 |                                         0        
               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
               4 |                                         0        

  /pageTwo                                          
           value  ------------- Distribution ------------- count    
               1 |                                         0        
               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
               4 |                                         0
-- 
This message posted from opensolaris.org

I did this on a dev machine.  We won''t have /usr/sfw/bin/cpp on prod
hosts.  Can someone point me to some general instructions on using the
''-G'' option to reuse this compiled dtrace program on another
host?
-- 
This message posted from opensolaris.org

On Tue, Oct 28, 2008 at 01:53:30PM -0700, Adam Mitchell
wrote:
> BTW, I replaced "string" with "Xstring" and it works. 
If I could avoid touching Apache''s files, that would be nice.  But I
can live with this.
No need to.  Just do:

#!/usr/sbin/dtrace -Cs

#define __builtin_va_list long
#define __attribute__(x)
#define string Xstring
#include <httpd.h>
#undef string
...

> Thanks for the help everybody!
> 
> I want to share this with anyone who will come looking for this later:
> 
> Copy the Apache header files to a work directory:
> 
> # cp -R /opt/apache/include /export/home/amitchel 
> 
> My command line:
> 
> 
> dtrace -32 -C -I /export/home/amitchel/include -x cpppath=/usr/sfw/bin/cpp
-Xs -s watchApacheRequests.d -p 843
> 
> My script:
> #!/usr/sbin/dtrace
> 
> #define __builtin_va_list long
> #define __attribute__(x)
> #include <httpd.h>
> 
> pid$target:a.out:ap_process_request:entry
> {
> 
>         self->ts = timestamp;
>         self->r = (request_rec*) copyin(arg0, sizeof(request_rec));
>         self->uri = copyinstr((uintptr_t) self->r->uri);
> 
> }
> 
> 
> pid$target:a.out:ap_process_request:return
> /self->ts/
> {
>         @request_uris[self->uri] = quantize((timestamp - self->ts) /
1000000);
>         self->ts = 0;
>         self->uri = "err";
> }
> 
> tick-5s 
> {
>         printa(@request_uris);
> }
> 
> 
> 
> You''ll get some errors like this because of the
''string'' reserved keyword:
> 
> cc1: warning: /dev/fd/6 is shorter than expected
> dtrace: failed to compile script watchApacheRequests.d:
"/export/home/amitchel/include/ap_regex.h", line 117: syntax error
near "string"
> 
> 
> Edit the file, replace ''string'' with
''Xstring'' and try again.  I had to edit these files:
> 
> /export/home/amitchel/include/ap_regex.h
> /export/home/amitchel/include/httpd.h
> 
> 
> And here''s the script output:
> 
>   0  49008                         :tick-5s 
>   /hello                                            
>            value  ------------- Distribution ------------- count    
>                1 |                                         0        
>                2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
>                4 |                                         0        
> 
>   /page1                                            
>            value  ------------- Distribution ------------- count    
>                1 |                                         0        
>                2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
>                4 |                                         0        
> 
>   /pageTwo                                          
>            value  ------------- Distribution ------------- count    
>                1 |                                         0        
>                2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1        
>                4 |                                         0
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

On Tue, Oct 28, 2008 at 02:00:23PM -0700, Adam Mitchell
wrote:
> I did this on a dev machine.  We won''t have /usr/sfw/bin/cpp on
prod
> hosts.  Can someone point me to some general instructions on using the
> ''-G'' option to reuse this compiled dtrace program on
another host?
Generally, you can''t.  But you could save the output of cpp:

truss -aft exec ./script

Pick out the "cpp" line, which might look like:

4056:  argv: cpp -D__sun -D__unix -D__SVR4 -D__SUNW_D=1 -D__SUNW_D_64
4056:   -D__SunOS_5_11 -D__amd64 -D__SUNW_D_VERSION=0x01006002
4056:   -D__STDC__=0 /dev/fd/4 /dev/fd/5

and transform that into:

tail +2 script | 
    /path/to/gnu/cpp -D__sun -D__unix -D__SVR4 -D__SUNW_D=1 -D__SUNW_D_64 \
    -D__SunOS_5_11 -D__amd64 -D__SUNW_D_VERSION=0x01006002 \
    -D__STDC__=0 /dev/stdin > script.preprocessed

Then, you can just do dtrace -s ./script.preprocessed on any machine of the
same architecture.

Note that if you''re targetting a 32-bit Apache, you should compile with
the ''-32'' argument to dtrace(1M).

Cheers,
- jonathan

> -- 
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

Thanks.  This works on my dev box running [dtrace: Sun D 1.3].  I ran the whole
DTrace script through cpp (with all the preserved flags that dtrace would have
used to invoke cpp) and the output will run stand-alone now (no cpp needed).

Our prod hosts are a little behind.  They have [dtrace: Sun D 1.1] and choke on
this section of the script:

typedef struct apr_bucket_refcount apr_bucket_refcount;

struct apr_bucket_refcount {

    int refcount;
};

struct apr_bucket_heap {

    apr_bucket_refcount refcount;  // I don''t think the typedef is
applied correctly here.
                                                   // I can prepend
''struct'' to get past it.


    char *base;

    apr_size_t alloc_len;

    void (*free_func)(void *data);
};


If this is a parser issue that was fixed between the two versions, I''m
wondering what my options are.  Is there any way to take the compiled code from
one host to another?
-- 
This message posted from opensolaris.org