dtrace discuss - Jul 2006 - access to errno when using pid provider

I would like to know how to get access to errno when using pid provider to probe
calls to
libc functions like fopen(). The built-in errno appears to be only for system
calls. What I''d
like to be able to do is investigate where in an application I''m
encountering EMFILE and
what the stack looks like at the time.
 
 
This message posted from opensolaris.org

Hi Richard,

I don''t believe fopen sets errno directly.  When fopen calls the open 
system call to actually open the file, then errno gets set.  So the 
builtin errno is the right variable to use, but you''ll have to wait 
until "syscall::open:return" fires to access it.

In order to narrow your scope to just fopens, I suppose you could set a 
thread local variable using the pid provider when fopen is invoked, and 
then check that variable in the predicate for the syscall::open probes.

Chip

Richard Smith wrote:
>I would like to know how to get access to errno when using pid provider to
probe calls to
>libc functions like fopen(). The built-in errno appears to be only for
system calls. What I''d
>like to be able to do is investigate where in an application I''m
encountering EMFILE and
>what the stack looks like at the time.
> 
> 
>This message posted from opensolaris.org
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>  
>

I think fopen() must assign to errno, directly or indirectly, at least for
32-bit code
due to the stdio limit of 255 open FILEs. Suppose I increase the file descriptor
limit from its default.
----------------------------------------
ulimit -n
=> 256
ulimit -n 260
ulimit -n
=> 260
----------------------------------------
And then run a silly program like:
----------------------------------------
#include <stdio.h>
#include <errno.h>

int main(int argc, char *argv[])
{
        FILE *f;
        int i;

        for (i = 0; i < 265; i++) {
                f = fopen("junk.c", "rb");
                if (f == NULL)
                        printf("errno=%d\n", errno);
        }
        return 0;
}
----------------------------------------
truss -topen,close -u libc:fopen junk
...
/1 at 1:   -> libc:fopen(0x10c80, 0x10c88, 0x0, 0xff36c7da)
/1:     open("junk.c", O_RDONLY)                        = 256
/1:     close(256)                                      = 0
/1 at 1:   <- libc:fopen() = 0
errno=24
----------------------------------------
So in this case an open() was issued, although maybe the library already knew it
had reached its FILE limit, the open was successful but got closed immediately,
and fopen() returned EMFILE. My challenge is to find a way of getting access to
errno via DTrace when using pid provider to so I can take specific actions on
encountering EMFILE.
 
 
This message posted from opensolaris.org

Richard Smith wrote:
> I think fopen() must assign to errno, directly or indirectly, at least for
32-bit code
> due to the stdio limit of 255 open FILEs. 
Yes, it does; if you haven''t yet seen enable_extended_FILE_stdio(3C),
check it
out; that''s the underlying reason (as you may know).
> Suppose I increase the file descriptor
> limit from its default.
> ----------------------------------------
> ulimit -n
> => 256
> ulimit -n 260
> ulimit -n
> => 260
> ----------------------------------------
> And then run a silly program like:
> ----------------------------------------
> #include <stdio.h>
> #include <errno.h>
> 
> int main(int argc, char *argv[])
> {
>         FILE *f;
>         int i;
> 
>         for (i = 0; i < 265; i++) {
>                 f = fopen("junk.c", "rb");
>                 if (f == NULL)
>                         printf("errno=%d\n", errno);
>         }
>         return 0;
> }
> ----------------------------------------
> truss -topen,close -u libc:fopen junk
> ...
> /1 at 1:   -> libc:fopen(0x10c80, 0x10c88, 0x0, 0xff36c7da)
> /1:     open("junk.c", O_RDONLY)                        = 256
> /1:     close(256)                                      = 0
> /1 at 1:   <- libc:fopen() = 0
> errno=24
> ----------------------------------------
> So in this case an open() was issued, although maybe the library already
knew it
> had reached its FILE limit, the open was successful but got closed
immediately,
> and fopen() returned EMFILE. My challenge is to find a way of getting
access to
> errno via DTrace when using pid provider to so I can take specific actions
on
> encountering EMFILE.
I guess you could echo its calculation and setting, by looking for returns of 
 >=256 from open(2) (assuming that you''re not using a last character
of F in the
mode string to bypass that check, or running on a 64-bit kernel where that check
is already ignored)

Richard Smith wrote:
> I think fopen() must assign to errno, directly or indirectly, at least for
32-bit code
> due to the stdio limit of 255 open FILEs. Suppose I increase the file
descriptor
> limit from its default.
> ----------------------------------------
> ulimit -n
> => 256
> ulimit -n 260
> ulimit -n
> => 260
> ----------------------------------------
> And then run a silly program like:
> ----------------------------------------
> #include <stdio.h>
> #include <errno.h>
> 
> int main(int argc, char *argv[])
> {
>         FILE *f;
>         int i;
> 
>         for (i = 0; i < 265; i++) {
>                 f = fopen("junk.c", "rb");
>                 if (f == NULL)
>                         printf("errno=%d\n", errno);
>         }
>         return 0;
> }
> ----------------------------------------
> truss -topen,close -u libc:fopen junk
> ...
> /1 at 1:   -> libc:fopen(0x10c80, 0x10c88, 0x0, 0xff36c7da)
> /1:     open("junk.c", O_RDONLY)                        = 256
> /1:     close(256)                                      = 0
> /1 at 1:   <- libc:fopen() = 0
> errno=24
> ----------------------------------------
> So in this case an open() was issued, although maybe the library already
knew it
> had reached its FILE limit, the open was successful but got closed
immediately,
> and fopen() returned EMFILE. My challenge is to find a way of getting
access to
> errno via DTrace when using pid provider to so I can take specific actions
on
> encountering EMFILE.
Richard,

Yes, fopen() for 32-bit apps will open, then close the file. You can hack
something
because fopen() calls _endopen(), which does the check for fd <=
_FILE_FD_MAX. If fd
 > than that value, it calls _file_set() which sets errno to EMFILE.

So something like this should work:

# cat fopen.d

pid$target::fopen:entry
{
         self->in_fopen = 1;
}

pid$target::_file_set:return
/self->in_fopen && (int)arg1 == -1/
{
         self->emfile = 1;
}

pid$target::fopen:return
/(int) arg1 == 0 && self->emfile /
{
         printf("too many files in fopen\n");
         self->emfile = 0;
}
pid$target::fopen:return
{
         self->in_fopen = 0;
}

# dtrace -s fopen.d -c ./a.out
CPU     ID                    FUNCTION:NAME
   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

   0  46961                     fopen:return too many files in fopen

Jim

Hey Richard,

Getting at the errno value in libc is a little tricky, but it can be done.
I''ve attached a script that shows an example of this might work. This
script
defines two inline variables: uwlp and uerrno. I''ve tried it on 32-bit
and
64-bit processes on an AMD64 system -- it may or may not work on other systems.

Unfortunately, there aren''t any header files in /usr/include that have
the
ulwp_t structure definition, and DTrace can''t yet make use of user-land
CTF
data so we''re stuck hard-coding the offsets. You may need to modify
them
as described in the attachment.

Adam

On Mon, Jul 17, 2006 at 03:38:33PM -0700, Richard Smith
wrote:
> I would like to know how to get access to errno when using pid provider to
probe calls to
> libc functions like fopen(). The built-in errno appears to be only for
system calls. What I''d
> like to be able to do is investigate where in an application I''m
encountering EMFILE and
> what the stack looks like at the time.
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl
-------------- next part --------------
/*
 * CDDL HEADER START
 *
 * The contents of this file are subject to the terms of the
 * Common Development and Distribution License (the "License").
 * You may not use this file except in compliance with the License.
 *
 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
 * or http://www.opensolaris.org/os/licensing.
 * See the License for the specific language governing permissions
 * and limitations under the License.
 *
 * When distributing Covered Code, include this CDDL HEADER in each
 * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
 * If applicable, add the following below this CDDL HEADER, with the
 * fields enclosed by brackets "[]" replaced with your own identifying
 * information: Portions Copyright [yyyy] [name of copyright owner]
 *
 * CDDL HEADER END
 */

/*
 * Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
 * Use is subject to license terms.
 */

#pragma ident	"%Z%%M%	%I%	%E% SMI"

/*
 * The offsets of the errno value from the start of the thread structure
 * can be determined for your specific installation by doing this:
 *
 *	$ echo ::print -at ulwp_t ul_errnop | mdb /lib/libc.so.1
 *	bc int *ul_errnop 
 *	$ echo ::print -at ulwp_t ul_errnop | mdb /lib/64/libc.so.1
 *	110 int *ul_errnop 
 */

#if defined(__amd64)
inline void *ulwp     (void *)(curpsinfo->pr_dmodel == PR_MODEL_LP64 ?
    curthread->t_lwp->lwp_pcb.pcb_fsbase :
    curthread->t_lwp->lwp_pcb.pcb_gsbase);
inline int uerrno     *(int *)copyin(curpsinfo->pr_dmodel == PR_MODEL_LP64 ?
    *(uint64_t *)copyin((uintptr_t)((char *)ulwp + 0x110), sizeof (uint64_t)) :
    *(uint32_t *)copyin((uintptr_t)((char *)ulwp + 0xbc), sizeof (uint32_t)),
    sizeof (int));
#elif defined(__i386)
inline void *ulwp     (void
*)(curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_lobase |
    (curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_midbase << 16) |
    (curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_hibase << 24));
inline int uerrno = *(int *)copyin(
    *(uintptr_t *)copyin((uintptr_t)((char *)ulwp + 0xbc), sizeof (uintptr_t)),
    sizeof (int));
#elif defined(__sparc)
inline void *ulwp = (void *)uregs[R_G7];
inline int uerrno     *(int *)copyin(curpsinfo->pr_dmodel == PR_MODEL_LP64 ?
    *(uint64_t *)copyin((uintptr_t)((char *)ulwp + 0x128), sizeof (uint64_t)) :
    *(uint32_t *)copyin((uintptr_t)((char *)ulwp + 0xe4), sizeof (uint32_t)),
    sizeof (int));
#endif

pid$target::fopen:return
{
	trace(uerrno);
}

I''ve looking at the code, and I have a suspicion why the definition of 
"errno" has to be buried in a thread structure, rather than what you
see
if you "gcc -E" a file containing <errno.h>, but I
don''t if I should say
before asking.  (OK, I''ll stick my neck out: it won''t be the
first
time.... because the library is reentrant??)

Am I correct in assuming that if "errno" had been defined in the
kernel
as just "extern int", that it would have been available as an
external.
(`errno)?

Chip

Adam Leventhal wrote:
>Hey Richard,
>
>Getting at the errno value in libc is a little tricky, but it can be done.
>I''ve attached a script that shows an example of this might work.
This script
>defines two inline variables: uwlp and uerrno. I''ve tried it on
32-bit and
>64-bit processes on an AMD64 system -- it may or may not work on other
systems.
>
>Unfortunately, there aren''t any header files in /usr/include that
have the
>ulwp_t structure definition, and DTrace can''t yet make use of
user-land CTF
>data so we''re stuck hard-coding the offsets. You may need to modify
them
>as described in the attachment.
>
>Adam
>
>On Mon, Jul 17, 2006 at 03:38:33PM -0700, Richard Smith wrote:
>  
>
>>I would like to know how to get access to errno when using pid provider
to probe calls to
>>libc functions like fopen(). The built-in errno appears to be only for
system calls. What I''d
>>like to be able to do is investigate where in an application
I''m encountering EMFILE and
>>what the stack looks like at the time.
>> 
>> 
>>This message posted from opensolaris.org
>>_______________________________________________
>>dtrace-discuss mailing list
>>dtrace-discuss at opensolaris.org
>>    
>>
>
>  
>
>------------------------------------------------------------------------
>
>/*
> * CDDL HEADER START
> *
> * The contents of this file are subject to the terms of the
> * Common Development and Distribution License (the "License").
> * You may not use this file except in compliance with the License.
> *
> * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
> * or http://www.opensolaris.org/os/licensing.
> * See the License for the specific language governing permissions
> * and limitations under the License.
> *
> * When distributing Covered Code, include this CDDL HEADER in each
> * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
> * If applicable, add the following below this CDDL HEADER, with the
> * fields enclosed by brackets "[]" replaced with your own
identifying
> * information: Portions Copyright [yyyy] [name of copyright owner]
> *
> * CDDL HEADER END
> */
>
>/*
> * Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
> * Use is subject to license terms.
> */
>
>#pragma ident	"%Z%%M%	%I%	%E% SMI"
>
>/*
> * The offsets of the errno value from the start of the thread structure
> * can be determined for your specific installation by doing this:
> *
> *	$ echo ::print -at ulwp_t ul_errnop | mdb /lib/libc.so.1
> *	bc int *ul_errnop 
> *	$ echo ::print -at ulwp_t ul_errnop | mdb /lib/64/libc.so.1
> *	110 int *ul_errnop 
> */
>
>#if defined(__amd64)
>inline void *ulwp >    (void *)(curpsinfo->pr_dmodel == PR_MODEL_LP64
?
>    curthread->t_lwp->lwp_pcb.pcb_fsbase :
>    curthread->t_lwp->lwp_pcb.pcb_gsbase);
>inline int uerrno >    *(int *)copyin(curpsinfo->pr_dmodel ==
PR_MODEL_LP64 ?
>    *(uint64_t *)copyin((uintptr_t)((char *)ulwp + 0x110), sizeof
(uint64_t)) :
>    *(uint32_t *)copyin((uintptr_t)((char *)ulwp + 0xbc), sizeof
(uint32_t)),
>    sizeof (int));
>#elif defined(__i386)
>inline void *ulwp >    (void
*)(curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_lobase |
>    (curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_midbase << 16) |
>    (curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_hibase << 24));
>inline int uerrno = *(int *)copyin(
>    *(uintptr_t *)copyin((uintptr_t)((char *)ulwp + 0xbc), sizeof
(uintptr_t)),
>    sizeof (int));
>#elif defined(__sparc)
>inline void *ulwp = (void *)uregs[R_G7];
>inline int uerrno >    *(int *)copyin(curpsinfo->pr_dmodel ==
PR_MODEL_LP64 ?
>    *(uint64_t *)copyin((uintptr_t)((char *)ulwp + 0x128), sizeof
(uint64_t)) :
>    *(uint32_t *)copyin((uintptr_t)((char *)ulwp + 0xe4), sizeof
(uint32_t)),
>    sizeof (int));
>#endif
>
>pid$target::fopen:return
>{
>	trace(uerrno);
>}
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>

Hi Chip,

If errno were a global kernel symbol you could access with it DTrace by
doing `errno. Of course, since Solaris can''t have a global notion of
errno
since many processes and threads can be active in the kernel at the same
time. When a system call sets an errno value it''s stored in the
kernel''s
LWP structure (klwp_t):

  http://cvs.opensolaris.org/source/xref/on/usr/src/uts/sparc/os/syscall.c#1127
 
http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/sys/klwp.h#lwp_errno

That''s the value you get in the DTrace errno variable which is defined
in
/usr/lib/dtrace/procfs.d:

 
http://cvs.opensolaris.org/source/xref/on/usr/src/lib/libdtrace/common/procfs.d.in#36

In user-land, errno was historically just an int that was defined in libc.
This needed to be changed for the multi-threaded process model. Each LWP
has a ulwp_t structure managed by libc. The ulwp_t structure has a ul_errnop
member which is a pointer to an int, either the global errno int in libc (for
the main thread) or the ul_errno member of ulwp_t (all other threads). Code
that''s compiled with -D_REENTRANT gets a definition for errno that uses
that pointer in the ulwp_t by calling ___errno():

  http://cvs.opensolaris.org/source/xref/on/usr/src/lib/libc/port/gen/errno.c#38

For those of you keeping score at home, a symbol prefixed with a
''_'' is for
internal use only, with ''__'' is even more private and often
has to do with
the implementation of the ''_'' version, and with
''___'' to indicate the some
artifact from the ''70s is hacking its way into the modern era.

The D script I sent out finds the location of the ulwp_t structure in an
ISA-specific way, and then adds the offset to the ul_errnop member.

Adam

On Wed, Jul 19, 2006 at 04:12:50PM -0500, Chip Bennett
wrote:
> I''ve looking at the code, and I have a suspicion why the
definition of
> "errno" has to be buried in a thread structure, rather than what
you see
> if you "gcc -E" a file containing <errno.h>, but I
don''t if I should say
> before asking.  (OK, I''ll stick my neck out: it won''t be
the first
> time.... because the library is reentrant??)
> 
> Am I correct in assuming that if "errno" had been defined in the
kernel
> as just "extern int", that it would have been available as an
external.
> (`errno)?
> 
> Chip
> 
> Adam Leventhal wrote:
> 
> >Hey Richard,
> >
> >Getting at the errno value in libc is a little tricky, but it can be
done.
> >I''ve attached a script that shows an example of this might
work. This
> >script
> >defines two inline variables: uwlp and uerrno. I''ve tried it
on 32-bit and
> >64-bit processes on an AMD64 system -- it may or may not work on other 
> >systems.
> >
> >Unfortunately, there aren''t any header files in /usr/include
that have the
> >ulwp_t structure definition, and DTrace can''t yet make use of
user-land CTF
> >data so we''re stuck hard-coding the offsets. You may need to
modify them
> >as described in the attachment.
> >
> >Adam
> >
> >On Mon, Jul 17, 2006 at 03:38:33PM -0700, Richard Smith wrote:
> > 
> >
> >>I would like to know how to get access to errno when using pid
provider
> >>to probe calls to
> >>libc functions like fopen(). The built-in errno appears to be only
for
> >>system calls. What I''d
> >>like to be able to do is investigate where in an application
I''m
> >>encountering EMFILE and
> >>what the stack looks like at the time.
> >>
> >>
> >>This message posted from opensolaris.org
> >>_______________________________________________
> >>dtrace-discuss mailing list
> >>dtrace-discuss at opensolaris.org
> >>   
> >>
> >
> > 
> >
>
>------------------------------------------------------------------------
> >
> >/*
> >* CDDL HEADER START
> >*
> >* The contents of this file are subject to the terms of the
> >* Common Development and Distribution License (the
"License").
> >* You may not use this file except in compliance with the License.
> >*
> >* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
> >* or http://www.opensolaris.org/os/licensing.
> >* See the License for the specific language governing permissions
> >* and limitations under the License.
> >*
> >* When distributing Covered Code, include this CDDL HEADER in each
> >* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
> >* If applicable, add the following below this CDDL HEADER, with the
> >* fields enclosed by brackets "[]" replaced with your own
identifying
> >* information: Portions Copyright [yyyy] [name of copyright owner]
> >*
> >* CDDL HEADER END
> >*/
> >
> >/*
> >* Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
> >* Use is subject to license terms.
> >*/
> >
> >#pragma ident	"%Z%%M%	%I%	%E% SMI"
> >
> >/*
> >* The offsets of the errno value from the start of the thread structure
> >* can be determined for your specific installation by doing this:
> >*
> >*	$ echo ::print -at ulwp_t ul_errnop | mdb /lib/libc.so.1
> >*	bc int *ul_errnop 
> >*	$ echo ::print -at ulwp_t ul_errnop | mdb /lib/64/libc.so.1
> >*	110 int *ul_errnop 
> >*/
> >
> >#if defined(__amd64)
> >inline void *ulwp > >   (void *)(curpsinfo->pr_dmodel ==
PR_MODEL_LP64 ?
> >   curthread->t_lwp->lwp_pcb.pcb_fsbase :
> >   curthread->t_lwp->lwp_pcb.pcb_gsbase);
> >inline int uerrno > >   *(int *)copyin(curpsinfo->pr_dmodel ==
PR_MODEL_LP64 ?
> >   *(uint64_t *)copyin((uintptr_t)((char *)ulwp + 0x110), sizeof 
> >   (uint64_t)) :
> >   *(uint32_t *)copyin((uintptr_t)((char *)ulwp + 0xbc), sizeof 
> >   (uint32_t)),
> >   sizeof (int));
> >#elif defined(__i386)
> >inline void *ulwp > >   (void
*)(curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_lobase |
> >   (curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_midbase << 16)
|
> >   (curthread->t_lwp->lwp_pcb.pcb_gsdesc.usd_hibase <<
24));
> >inline int uerrno = *(int *)copyin(
> >   *(uintptr_t *)copyin((uintptr_t)((char *)ulwp + 0xbc), sizeof 
> >   (uintptr_t)),
> >   sizeof (int));
> >#elif defined(__sparc)
> >inline void *ulwp = (void *)uregs[R_G7];
> >inline int uerrno > >   *(int *)copyin(curpsinfo->pr_dmodel ==
PR_MODEL_LP64 ?
> >   *(uint64_t *)copyin((uintptr_t)((char *)ulwp + 0x128), sizeof 
> >   (uint64_t)) :
> >   *(uint32_t *)copyin((uintptr_t)((char *)ulwp + 0xe4), sizeof 
> >   (uint32_t)),
> >   sizeof (int));
> >#endif
> >
> >pid$target::fopen:return
> >{
> >	trace(uerrno);
> >}
> > 
> >
>
>------------------------------------------------------------------------
> >
> >_______________________________________________
> >dtrace-discuss mailing list
> >dtrace-discuss at opensolaris.org
> >
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl