dtrace discuss - Apr 2008 - Capability of memory overwritten detection

Hi,
 
I have a problem to sort out a memory overwritten  of our Application on Solaris
10.
I wonder if it is possible to detect memory overwritten by using dtrace. I have
worked for power pc platform before and I have usually used the DABR register to
detect the memory overwritten problem. I would like to perform the following
procedure.
 
1) Set the address which we want to check as watch point.
2) Enable/disable to supervise the watch point
3) Core/stack dump will be generated when the watch point address has been
access to write during enable state.
 
Is it possible to perform the above procedure by the dtrace?
 
Best Regards,
 
/Mitsu

  _____  

Mitsuhiro Nakamura <mailto:F?RNAMN.EFTERNAMN at cinnober.com>  
Partner, Developer 

Cinnober Financial Technology AB <http://www.cinnober.com/>  
Kungsgatan 36 
SE-111 35  STOCKHOLM, Sweden 

Phone:	 +46 8 503 143 06	
Fax:	 +46 8 503 047 01	
Mobile:	 +46 734 14 81 79	

  

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080416/bcf24a0e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 1347 bytes
Desc: cft.jpg
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080416/bcf24a0e/attachment.jpe>

Hi,
> I have a problem to sort out a memory overwritten  of our Application on
Solaris 10.
> I wonder if it is possible to detect memory overwritten by using dtrace. I
have worked for power pc platform before and I have usually used the DABR
register to detect the memory overwritten problem. I would like to perform the
following procedure.
>  
> 1) Set the address which we want to check as watch point.
> 2) Enable/disable to supervise the watch point
> 3) Core/stack dump will be generated when the watch point address has been
access to write during enable state.
>  
> Is it possible to perform the above procedure by the dtrace?
Dtrace does not have watchpoints. If you for example know that the
memory is overwritten by mempcy or strcpy, you could set
probe to those functions and test their parameters.

On the other hand, why not use the memory debugging library
libumem(3LIB) or even dbx debugger which has watchpoints ?

Hope this helps

-- 
	Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 193 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080416/2f7d74d7/attachment.bin>

I think the short answer for you is no, DTrace isn''t suitable here.

Slightly longer answer: If you know that the memory is being overwritten 
through the use of a particular interface (e.g. system call, library 
call etc.) then you could use DTrace to track users of that interface 
and report suspicious activity. However, if you simply have a seemingly 
random act and you have no idea where it is coming from then DTrace is 
not the answer.

Depending upon the circumstances you are seeing a debugger such as dbx 
or gdb could be the answer (this would allow you to set up watchpoints) 
or, if the memory came from malloc and friends, libumem(3LIB) along with 
umem_debug(3MALLOC) may be worth investigating.

Hope this helps,
Peter.

Mitsuhiro Nakamura wrote:
> Hi,
>  
> I have a problem to sort out a memory overwritten  of our Application on 
> Solaris 10.
> I wonder if it is possible to detect memory overwritten by using dtrace. 
> I have worked for power pc platform before and I have usually used the 
> DABR register to detect the memory overwritten problem. I would like to 
> perform the following procedure.
>  
> 1) Set the address which we want to check as watch point.
> 2) Enable/disable to supervise the watch point
> 3) Core/stack dump will be generated when the watch point address has 
> been access to write during enable state.
>  
> Is it possible to perform the above procedure by the dtrace?
>  
> Best Regards,
>  
> /Mitsu
> 
> ------------------------------------------------------------------------
> *Mitsuhiro Nakamura <mailto:F?RNAMN.EFTERNAMN at cinnober.com>*
> Partner, Developer
> 
> *Cinnober Financial Technology AB <http://www.cinnober.com/>*
> Kungsgatan 36
> SE-111 35  STOCKHOLM, Sweden
> 
> Phone: 	+46 8 503 143 06
> Fax: 	+46 8 503 047 01
> Mobile: 	+46 734 14 81 79
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3253 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080416/73049cda/attachment.bin>

Hi Mitsu,
 
> I have a problem to sort out a memory overwritten  of our Application 
> on Solaris 10.
> I wonder if it is possible to detect memory overwritten by using 
> dtrace. I have worked for power pc platform before and I have usually 
> used the DABR register to detect the memory overwritten problem. I 
> would like to perform the following procedure.
>  
> 1) Set the address which we want to check as watch point.
> 2) Enable/disable to supervise the watch point
> 3) Core/stack dump will be generated when the watch point address has 
> been access to write during enable state.
>  
> Is it possible to perform the above procedure by the dtrace?
At the minute you can''t do watchpoints in DTrace. An RFE
exists for a watchpoint provider which you can add yourself to
(although I think you have to have a service agreement to do this):

CR5059507 Watchpoint Provider

Alternatively you could register your interest on the recently created
"Wish List" page on the DTrace wiki:

http://wikis.sun.com/display/DTrace/The+Wish+List

For the minute though, the way of doing this would be to use
the ''::wp'' dcmd in mdb(1) or the ''stop''
command in dbx.

Jon.
**

Hi Jon,

Thanks for your information!!

I will register my interest on the "Wish List" and will check the mdb.

Best Regards,

/Mitsu

-----Original Message-----
From: Jonathan.Haslam at Sun.COM [mailto:Jonathan.Haslam at Sun.COM] 
Sent: den 16 april 2008 13:17
To: Mitsuhiro Nakamura
Cc: dtrace-discuss at opensolaris.org
Subject: Re: [dtrace-discuss] Capability of memory overwritten detection

Hi Mitsu,
 
> I have a problem to sort out a memory overwritten  of our Application 
> on Solaris 10.
> I wonder if it is possible to detect memory overwritten by using 
> dtrace. I have worked for power pc platform before and I have usually 
> used the DABR register to detect the memory overwritten problem. I 
> would like to perform the following procedure.
>  
> 1) Set the address which we want to check as watch point.
> 2) Enable/disable to supervise the watch point
> 3) Core/stack dump will be generated when the watch point address has 
> been access to write during enable state.
>  
> Is it possible to perform the above procedure by the dtrace?
At the minute you can''t do watchpoints in DTrace. An RFE exists for a
watchpoint provider which you can add yourself to (although I think you
have to have a service agreement to do this):

CR5059507 Watchpoint Provider

Alternatively you could register your interest on the recently created
"Wish List" page on the DTrace wiki:

http://wikis.sun.com/display/DTrace/The+Wish+List

For the minute though, the way of doing this would be to use the
''::wp''
dcmd in mdb(1) or the ''stop'' command in dbx.

Jon.
**

Hi Gonzalo,

Thanks!! I will ckeck the link.

BR

/Mitsu

-----Original Message-----
From: Gonzalo.Siero at Sun.COM [mailto:Gonzalo.Siero at Sun.COM] 
Sent: den 16 april 2008 13:22
To: Mitsuhiro Nakamura
Subject: Re: [dtrace-discuss] Capability of memory overwritten detection

Check:

http://www.opensolaris.org/os/community/mdb/

Cheers,
Gonzalo.

Mitsuhiro Nakamura wrote:
>Hi,
>
>Thanks for your information!!
>
>Do you have any recommended link about the mdb?
>
>Best Regards,
>
>/Mitsu
>
>-----Original Message-----
>From: Gonzalo.Siero at Sun.COM [mailto:Gonzalo.Siero at Sun.COM]
>Sent: den 16 april 2008 12:08
>To: Mitsuhiro Nakamura
>Subject: Re: [dtrace-discuss] Capability of memory overwritten 
>detection
>
>I''ve never done that but i think you could do it using mdb...
>
>     addr [ ,len ] :w [ cmd . . . ]
>
>         Set a watchpoint at the specified address. The length in
>         bytes  of the watched region can be set by specifying an
>         optional repeat count preceding the dcmd. If  no  length
>         is  explicitly  set,  the  default is one byte. The ::wp
>         dcmd allows the watchpoint to be configured  to  trigger
>         on  any  combination  of  read  (-r  option),  write (-w
>         option), or execute (-x option) access. The -d, -D,  -e,
>         -s,  -t, -T, -c, and -n options have the same meaning as
>         they do for the ::evset dcmd. The :a dcmd  sets  a  read
>         access  watchpoint at the specified address. The :p dcmd
>         sets an  execute  access  watchpoint  at  the  specified
>         address.  The  :w dcmd sets a write access watchpoint at
>         the specified address. The arguments following  the  :a,
>         :p,  and  :w dcmds are concatenated together to form the
>         callback  string.  If   this   string   contains   meta-
>         characters, it must be quoted.
>
>To start a process with mdb you should so mdb <absolute path to 
>binary>, set the watchpoint and then when process is stoped with gcore 
>get a corefile or with $c get the stack...
>
>mdb has it''s own interest list (see in opensolaris) maybe you can
also
>try there...
>
>Cheers,
>Gonzalo.
>
>
>Mitsuhiro Nakamura wrote:
>
>  
>
>>Hi,
>> 
>>I have a problem to sort out a memory overwritten  of our Application 
>>on Solaris 10.
>>I wonder if it is possible to detect memory overwritten by using 
>>dtrace. I have worked for power pc platform before and I have usually 
>>used the DABR register to detect the memory overwritten problem. I 
>>would like to perform the following procedure.
>> 
>>1) Set the address which we want to check as watch point.
>>2) Enable/disable to supervise the watch point
>>3) Core/stack dump will be generated when the watch point address has 
>>been access to write during enable state.
>> 
>>Is it possible to perform the above procedure by the dtrace?
>> 
>>Best Regards,
>> 
>>/Mitsu
>>
>>----------------------------------------------------------------------
>>-- *Mitsuhiro Nakamura <mailto:F%D6RNAMN.EFTERNAMN at
cinnober.com>*
>>Partner, Developer
>>
>>*Cinnober Financial Technology AB <http://www.cinnober.com/>* 
>>Kungsgatan 36
>>SE-111 35  STOCKHOLM, Sweden
>>
>>Phone: 	+46 8 503 143 06
>>Fax: 	+46 8 503 047 01
>>Mobile: 	+46 734 14 81 79
>>
>>----------------------------------------------------------------------
>>-
>>-
>>
>>_______________________________________________
>>dtrace-discuss mailing list
>>dtrace-discuss at opensolaris.org
>> 
>>
>>    
>>
>
>
>  
>

This might get you going:

dtrace -n ''pid$target::strcpy:entry{printf("From: %lx, To: %lx,
string:''%s''", arg1, arg0, copyinstr(arg1))}'' -c
ls

This traces strcpy function and prints what is being copied where when
you run ''ls'' command. If you want to attach dtrace to running
process,
use ''-p PID'' instead of ''-c binary''.

If you want to see all places where you can attach probe, try

dtrace -l -n ''pid$target::str*:entry'' -c ls

If you want to see stack when the function is executed, add
''ustack();'',
so that you have something like

dtrace -n ''pid$target::strcpy:entry{printf("From: %lx, To: %lx,
string:''%s''", arg1, arg0, copyinstr(arg1));
ustack()}'' -c ls

If you want to check if strcpy starts to write to known buffer
use something like

pid$target::strcpy:entry
/ arg0 > my_buffer && arg0 < (my_buffer+my_buffer_size) /
{
        ...
}

But there are other options you will want to check (strcpy starts to
write before your buffer, but writes too many bytes).

It can be hard to get my_buffer value, as it can be at different place
every program run. Easiest way is to add new function to your code,
where you pass the buffer.

=============================== source.c =============================={
        asm("nop"); /* Just to make sure that compiler won''t
optimize
                       out this function */
}

....
        buf=malloc(1204);
        dtrace_helper(buf);
...
=======================================================================
And later in dtrace

=============================== trace.d
===============================pid$target::dtrace_helper:entry
{
        my_buffer = arg0;
}

pid$target::strcpy:entry
/ my_buffer && ..... /   /* only run the probe if my_buffer has been set
*/
{
}
=======================================================================
Last thing, you don''t need to kill the application, you may sigstop it.

pid$target::strcpy:entry
/ my_buffer && ..... /   /* only run the probe if my_buffer has been set
*/
{
        ustack();
        stop();
}

You can then
$ pstack PID # the same info is obtainable by ustack() dtrace function
$ pmap PID
$ pfiles PID
...
$ prun PID

Let us know how it goes :)


> Do you have any recommended link for libumem?
Adam''s blog has a nice intro. You will need debugger though ...

http://blogs.sun.com/ahl/entry/solaris_10_top_11_20



Ufff ;)

-- 
        Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 193 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080416/e5bcca39/attachment.bin>

Peter Telford - Solaris sustaining writes:
> Depending upon the circumstances you are seeing a debugger such as dbx 
> or gdb could be the answer (this would allow you to set up watchpoints) 
> or, if the memory came from malloc and friends, libumem(3LIB) along with 
> umem_debug(3MALLOC) may be worth investigating.
Another possible answer is proc(4) PCWATCH.  I''ve used this mechanism
in my own applications to catch places where certain dynamically-
allocated structures were getting overwritten.  Because the
application itself is in control of where the watchpoints are being
placed, you can easily implement very detailed and complex rules for
determining when to enable a watchpoint -- something that''s often hard
or impossible to do with a debugger.

Obviously, it means modifying and recompiling the application (likely
more than once or twice), so it doesn''t work for all possible
scenarios.

-- 
James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

Hi all,

Thanks lots for all information!!
We are indeed provide API to the certain customer and they have a
problem at the their production. We are not able to reproduce the
problem at our local test environment. And customer want neither to use
debugger nor to accept rebuild(for debugging) on the production.
Therefore, I will prefer to try Vladimirs suggested dtrace procedure
(attached mail) or mdb. However, I am still missing for DABR register on
Power PC.

I have general question here! How do you manage this kind of problem at
the customer site? I have worked many years for developing of hard and
soft ware. It is often very difficult situation if the problem is
concerning on a interaction between the application and API. Surely, it
is nice to have a good relationship, such customer/user as partner.
However, it is not often the case. Therefore, we must often pinpoint
what is fault, like show the stack-chain in this case. Sorry this kind
of discussion does not belong to this mail-list, but I am very curious!!

Best regards,

/Mitsu

-----Original Message-----
From: James Carlson [mailto:james.d.carlson at Sun.COM] 
Sent: den 16 april 2008 16:09
To: Peter Telford - Solaris sustaining
Cc: Mitsuhiro Nakamura; dtrace-discuss at opensolaris.org
Subject: Re: [dtrace-discuss] Capability of memory overwritten detection

Peter Telford - Solaris sustaining writes:
> Depending upon the circumstances you are seeing a debugger such as dbx
> or gdb could be the answer (this would allow you to set up 
> watchpoints) or, if the memory came from malloc and friends, 
> libumem(3LIB) along with
> umem_debug(3MALLOC) may be worth investigating.
Another possible answer is proc(4) PCWATCH.  I''ve used this mechanism
in
my own applications to catch places where certain dynamically- allocated
structures were getting overwritten.  Because the application itself is
in control of where the watchpoints are being placed, you can easily
implement very detailed and complex rules for determining when to enable
a watchpoint -- something that''s often hard or impossible to do with a
debugger.

Obviously, it means modifying and recompiling the application (likely
more than once or twice), so it doesn''t work for all possible
scenarios.

-- 
James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

-------------- next part --------------
An embedded message was scrubbed...
From: "Vladimir Marek" <Vladimir.Marek at Sun.COM>
Subject: Re: [dtrace-discuss] Capability of memory overwritten detection
Date: Wed, 16 Apr 2008 14:11:31 +0200
Size: 5792
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080416/0dc173c6/attachment.nws>

Mitsuhiro Nakamura writes:
> Thanks lots for all information!!
> We are indeed provide API to the certain customer and they have a
> problem at the their production. We are not able to reproduce the
> problem at our local test environment. And customer want neither to use
> debugger nor to accept rebuild(for debugging) on the production.
> Therefore, I will prefer to try Vladimirs suggested dtrace procedure
> (attached mail) or mdb. However, I am still missing for DABR register on
> Power PC.
Access to the hardware debug features is exactly what the proc(4)
interfaces give you.  (And what the debuggers use.)  I don''t think
you''re missing much here, though I agree that some of the PPC family
members have some *really* slick debug features.  (The 860 family
comes to mind ...)
> I have general question here! How do you manage this kind of problem at
> the customer site? I have worked many years for developing of hard and
> soft ware. It is often very difficult situation if the problem is
> concerning on a interaction between the application and API. Surely, it
> is nice to have a good relationship, such customer/user as partner.
> However, it is not often the case. Therefore, we must often pinpoint
> what is fault, like show the stack-chain in this case. Sorry this kind
> of discussion does not belong to this mail-list, but I am very curious!!
"It depends."

As you note, some customers are interested in getting rapid and
accurate support, and are thus very good about allowing for test runs
and debug to get at the root of an issue.  Other customers clearly are
not -- they sometimes demand that we magically fix a problem without
knowing exactly what it is.

There are a huge number of techniques available -- probably too many
to name here.  A few I''ve seen and/or used:

  - Event logs: just keep a static buffer around in memory to record
    whatever events you feel are significant in your application.  If
    the application crashes, the core file (or system dump, if it''s a
    kernel module) will have that buffer and may provide some clues.

  - Good ol'' printf: sending messages to syslog or to a text log file
    of some sort can be helpful.  It gives you more room and
    flexibility than the typical circular event log, but costs a lot
    more in performance.

  - Assertions: not just assert(3C), but deliberate self-checks in the
    code (verifying the integrity of data structures, and making sure
    that invariants are in fact true) can help isolate problems and
    prove that certain parts of a large complex system are not the
    source of the fault.

  - Intentional damage: deliberately change some part of the system so
    that it either fails intermittently (e.g., make malloc return NULL
    every now and then), or perhaps so it fails completely (e.g.,
    insert a packet filter that drops all application data), or so
    that the timing changes (e.g., insert a delay loop in the middle
    of some block of code that''s running without locks and that you
    suspect looks like a timing hole).  It is sometimes possible to
    reproduce (and diagnose) customer problems by making things worse.

  - Impromptu code review: sometimes, a hard-to-fix bug is just the
    computer''s way of saying that perhaps right now would be a good
    time to stop everyone''s work, print out the code on paper, and go
    through it line-by-line looking for problems.

  - Staring at a core dump until your eyes bleed: the usual way to
    deal with problems at a recalcitrant customer site.

-- 
James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677