dtrace discuss - Oct 2006 - how to debug shared-memory using dtrace/mdb

A complex data structure stored in the shared-memory.
Couple of processes attach to the shared-memory...
Some of them spawn child procs...

Two processes have change notification mechanism encoded... 
Basically the producer timestamps and the consumer compares the timestamp
against it''s snapshot, if they differ then does use a semaphore to lock
and gather critical data and replaces the snapshot with the new timestamp.
Except these two processes none other procs call these routines...
truss''ing the producer doesn''t show it''s hitting the
change routine..
gdb''ing the consumer does hit the notification routine... 
while I''m in that break point , printing the values of the timestamp
couple of times for few minutes..
I noticed approximately 60 seconds interval the timestamp keeps changing..

I tried to monitor the "last pid" that accessed the shared-mem using
the following script.. where all the spowned procs id''s are logged into
the logfile.
0xb40070ff  is the key of the shared-mem.
while true; do 
grep `ipcs -mA | grep 0xb40070ff  | cut -b99-103` logfile; 
done
None of them seem to be suspects... 
Trussing all of the procs does''t look like an option as there are
dynamically forked procs...
Wish to know how can I find the process or the entity which is modifying the
timestamp memory using tools like mdb/dtrace...  something like setting a
watchpoint on this timestamp memory at the kernel debug level and watching who
changed it etc.,
 
 
This message posted from opensolaris.org

Unfortunately, there is no watchpoint provider (but that is a long-standing
RFE that I''d still like to get to some day 5059507) -- and even then,
that
would be at the user-level rather than the kernel level.

How exactly are these processes spawned? Could you use a D script to catch
them as they''re spawned and then use another script to see if that
process
is modifying the data in question?

Adam

On Tue, Oct 24, 2006 at 10:21:10AM -0700, KVM wrote:
> A complex data structure stored in the shared-memory.
> Couple of processes attach to the shared-memory...
> Some of them spawn child procs...
> 
> Two processes have change notification mechanism encoded... 
> Basically the producer timestamps and the consumer compares the timestamp
against it''s snapshot, if they differ then does use a semaphore to lock
and gather critical data and replaces the snapshot with the new timestamp.
> Except these two processes none other procs call these routines...
> truss''ing the producer doesn''t show it''s hitting
the change routine..
> gdb''ing the consumer does hit the notification routine... 
> while I''m in that break point , printing the values of the
timestamp couple of times for few minutes..
> I noticed approximately 60 seconds interval the timestamp keeps changing..
> 
> I tried to monitor the "last pid" that accessed the shared-mem
using the following script.. where all the spowned procs id''s are
logged into the logfile.
> 0xb40070ff  is the key of the shared-mem.
> while true; do 
> grep `ipcs -mA | grep 0xb40070ff  | cut -b99-103` logfile; 
> done
> None of them seem to be suspects... 
> Trussing all of the procs does''t look like an option as there are
dynamically forked procs...
> Wish to know how can I find the process or the entity which is modifying
the timestamp memory using tools like mdb/dtrace...  something like setting a
watchpoint on this timestamp memory at the kernel debug level and watching who
changed it etc.,
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

The consumer process forks off child processes to complete some job...
Other processes at the same level as consumer also fork off child processes... 
Mainly all the child processes will use the parent attached shared-mem further.

At this point we are still in the process of moving to solaris 10... and the
current dev/production is on solaris 9.   Can I use the D-trace on solaris 9?.

I''m thinking of preloading a overloaded _exit function library; where
it collects a gcore and then investigating who had what values for this
timestamp and comparing that against the changed value in the consumer.
Theoritically it sounds workable; any caveats/suggestions?.

Wish I had some understanding of how the shared-mem works at the kernel level;
like how often the change will be made  visible to all the attached processes,
and the mechanism it does. If there is some fault/signal which causes it then
that would be an interesting place to start for me.
 
 
This message posted from opensolaris.org

KVM wrote:
> The consumer process forks off child processes to complete some job...
> Other processes at the same level as consumer also fork off child
processes...
> Mainly all the child processes will use the parent attached shared-mem
further.
> 
> At this point we are still in the process of moving to solaris 10... and
the current dev/production is on solaris 9.   Can I use the D-trace on solaris
9?.
No, the DTrace infrastructure was not backported to Solaris 9.

Michael
-- 
Michael Schuster
Recursion, n.: see ''Recursion''