dtrace discuss - Jan 2007 - How to monitor any access to a specific file using drace

Hi dtrace experts,
I''m a newbie of dtrace, currently I meet a problem about how to monitor
any access(maybe vi, cat, more, rm, cp, ls, even from process, etc) to a
specific file, and then log the access info (user id/process id, date and time,
command).
I tried "iosnoop" in dtrace toolkits but seems only "vi"
operation is detected, can anyone provide some suggestion?

thanks!
 
 
This message posted from opensolaris.org

fds[] array is not supported yet in my system.

Firstly I wonder if I can use the syscall::open:entry to monitor "any
access" to the specific file(or most regular operations), I tried following
script:

#!/usr/sbin/dtrace -qs
syscall::open:entry
{printf("execname=%s,filename=%s\n",execname,copyinstr(arg0));}

from the lots of outputs, I can see some operations such as "cat",
"more", "cp", "vi", "rm",
"touch" ... to the specific file are catched, but how can I filter out
the un-needed outputs? From the outputs, it seems it can''t be done from
comparing the filename with the "specific filename".

for example for "cat" command, I get:
execname=cat,filename=/var/ld/ld.config
execname=cat,filename=/lib/libc.so.1
execname=cat,filename=/platform/SUNW,Ultra-80/lib/libc_psr.so.1

for "vi" command, I get:
execname=vi,filename=test
but there should be relative path/obsolute path issue.

Pls help figure out what enhancements need in the script to implement monitoring
"any access" to the specific file.
 
 
This message posted from opensolaris.org

G''Day David,

On Sun, Jan 21, 2007 at 11:59:28PM -0800, David Chen
wrote:
> Hi dtrace experts,
> I''m a newbie of dtrace, currently I meet a problem about how to
monitor any access(maybe vi, cat, more, rm, cp, ls, even from process, etc) to a
specific file, and then log the access info (user id/process id, date and time,
command).
> I tried "iosnoop" in dtrace toolkits but seems only
"vi" operation is detected, can anyone provide some suggestion?
iosnoop measures actual disk events; most file I/O will be cached, and
not make it to disk. This file I/O is also observable from DTrace using
the syscall provider, the fsinfo provider, and if needed the kernel VFS
interface via the fbt provider. The DTraceToolkit also has tools 
that observe this behaviour from different angles, such as rwsnoop
and pathopens.

Brendan

-- 
Brendan
[CA, USA]

G''Day David,

On Mon, Jan 22, 2007 at 01:14:16AM -0800, David Chen
wrote:
> fds[] array is not supported yet in my system.
> 
> Firstly I wonder if I can use the syscall::open:entry to monitor "any
access" to the specific file(or most regular operations), I tried following
script:
You''ll also want to match open64, eg, syscall::open*:entry.
> #!/usr/sbin/dtrace -qs
> syscall::open:entry
> {printf("execname=%s,filename=%s\n",execname,copyinstr(arg0));}
This will show most file opens. To catch them all, the copyinstr(arg0)
needs to happen on the open:return (once we know the path string has 
been faulted in).

At the syscall layer you can match on the pathname. But what if someone
creates another hardlink to the file? This would allow them to access
the file under a different pathname, and avoid monitoring. There are
a number of options available to solve this, such as,

   - monitor all file activity, including hard link creation.
   - monitor by vnode at the VFS layer, not at the syscall layer
> from the lots of outputs, I can see some operations such as
"cat", "more", "cp", "vi",
"rm", "touch" ... to the specific file are catched, but how
can I filter out the un-needed outputs? From the outputs, it seems it
can''t be done from comparing the filename with the "specific
filename".
Check out predicates in the DTrace guide; apart from a direct match on
a string using "==", an strstr() function has been added which behaves
similar to the C version - and will allow partial matches.

Of course, if you were troubleshooting an issue using DTrace, you could
simply dump all output through grep or egrep.
> for example for "cat" command, I get:
> execname=cat,filename=/var/ld/ld.config
> execname=cat,filename=/lib/libc.so.1
> execname=cat,filename=/platform/SUNW,Ultra-80/lib/libc_psr.so.1
> 
> for "vi" command, I get:
> execname=vi,filename=test
> but there should be relative path/obsolute path issue.
There are a number of ways to get absolute paths; I wrote one technique
in pathopens.d for the DTraceToolkit - although that was some time ago,
there may be better ways to do this these days.
> Pls help figure out what enhancements need in the script to implement
monitoring "any access" to the specific file.
DTrace is a fantastic troubleshooting tool; but "monitoring" is a
different
task that needs consideration. Is this security monitoring? Solaris''s
BSM auditing (aka SunSHIELD) may already solve your monitoring needs - 
and so while DTrace may work (and is much more customisable), you may
be reinventing some functionality.

And if it is security monitoring -- what should happen if there were
more events that the system could record. Should the system stop?
continue but drop events? drop and log the fact events were dropped? ...

Brendan

-- 
Brendan
[CA, USA]

Hi Brendan,
Thanks a lot for your information.
>From your mail, I understand that syscall provider can''t fulfill my
requirement even if I can get the full path name, since hardlink may be used, is
that right?
As you suggested, can I monitor "any access" by vnode at the VFS
layer?  How to do it and could you pls give some example?
It''s a security monitoring as you said, I just record the "access
events" and log them into a file ( I know it''s amusing since
"hacker" can easily delete his access log from the file, or even
delete the log file, but it''s the "requirement", :-(  ). On
the other hand, I''m quickly going through SunSHILED user guide, I see
it can monitor any user''s any action, but not sure if it can monitor
"any access" to a file, any experience that you can share about
setting it?

Best Regards
David
 
 
This message posted from opensolaris.org

Hello David,
    Not familiar with SunSHILED, but I guess you can monitor the disired 
access via VFS functions.
I''ve attached a simple D-script which can monitor that.It is far from 
enough at this moment,and certainly needs further polishing.
Just try to give an example.
BTW:thanks Brendan for explanations on dtrace :-)

Regards,
Zhijun

David Chen wrote:
> Hi Brendan,
> Thanks a lot for your information.
> >From your mail, I understand that syscall provider can''t
fulfill my requirement even if I can get the full path name, since hardlink may
be used, is that right?
> As you suggested, can I monitor "any access" by vnode at the VFS
layer?  How to do it and could you pls give some example?
> It''s a security monitoring as you said, I just record the
"access events" and log them into a file ( I know it''s
amusing since "hacker" can easily delete his access log from the file,
or even delete the log file, but it''s the "requirement", :-( 
). On the other hand, I''m quickly going through SunSHILED user guide, I
see it can monitor any user''s any action, but not sure if it can
monitor "any access" to a file, any experience that you can share
about setting it?
>
> Best Regards
> David
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>   
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fop.d
Type: text/x-dsrc
Size: 501 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070123/0ea10838/attachment.bin>

Hi ZhiJun,
Wow! Great, it''s almost what I want. Thanks a lot.
I tried the script and found a little problem, when I did "mv file
file.bak", i.e. rename the file under same directory, it can''t be
detected, I believe it dues to the internal implementation of "mv",
but I don''t know which VFS function I should use to detect it. BTW, I
can''t find VFS functions in solaris dtrace guide, where can I look for
the reference of VFS?
Additionally, I want to add some other info into the output, for example, the
user id, command name, date and time, could you pls help for a example?
I''m totally a newbie on dtrace and I only know using
"execname" to get command info, it works fine for commands such as
"cp", "cat",
"more","touch","vi" etc, but for some other
command ("crypt" as I tried ), the output command info is
"ksh", is there any way to get the correct command info?

Regards
David
 
 
This message posted from opensolaris.org

G''Day Zhijun, David,

On Tue, Jan 23, 2007 at 01:28:34PM +0800, Zhijun Fu
wrote:
> Hello David,
>    Not familiar with SunSHILED, but I guess you can monitor the disired 
> access via VFS functions.
> I''ve attached a simple D-script which can monitor that.It is far
from
> enough at this moment,and certainly needs further polishing.
> Just try to give an example.
Yes, this script is an example that shows what it can be like to trace at
the VFS layer. There is still much work to turn this into something
that is suitable for *security* monitoring. For example, the vnode->v_path
cached string had been provided for observability value, not security
auditing. In earlier Solaris 10 there were situations where v_path was
not correct; a discussion about this was at:

http://www.opensolaris.org/jive/thread.jspa?messageID=31616&#31616

AFAIK on newer builds of Solaris the file rename issue has been fixed; 
although additional hard link activity will still need to be traced.

Getting this figured out is still only one part of the problem; the
steps I would take are:

   1) write a script that monitors all activity to a given pathname.
   2) verify that this works by writing a test suite.
   3) enhance the script to log dropped events under load and to do
      something appropriate if the logging file system fills up.
   4) write a test suite to verify that the script behaves as expected
      under pressure and when the file system fills.

To achieve 1), it would help to have a good understanding of the VFS layer;
Chapter 14 of Solaris Internals 2nd edition is a great reference.
Also, in the long run it can save some time to write the test suites first.

If this seems like a lot of work, then try SunSHIELD which already 
provides file monitoring (although not for specific files), and can
already deal with issues described under 3).

no worries,

Brendan

-- 
Brendan
[CA, USA]

G''Day David,

On Mon, Jan 22, 2007 at 07:11:55PM -0800, David Chen
wrote:
> Hi Brendan,
> Thanks a lot for your information.
> >From your mail, I understand that syscall provider can''t
fulfill my requirement even if I can get the full path name, since hardlink may
be used, is that right?
By tracing syscall::link:entry, you can trace the creation of new hard links.
So long a the hard links were created after logging was enabled, you can
still try the syscall provider approach.
> As you suggested, can I monitor "any access" by vnode at the VFS
layer?  How to do it and could you pls give some example?
While VFS is an option, I''d certainly try this from the syscall layer
first, by tracing at least:

   syscall::open*:entry
   syscall::link:entry
   syscall::unlink:entry
> It''s a security monitoring as you said, I just record the
"access events" and log them into a file ( I know it''s
amusing since "hacker" can easily delete his access log from the file,
or even delete the log file, but it''s the "requirement", :-( 
).
hackers/crackers will have a tough time modifying the log file if it were
owned and modifiable only as root; patched Solaris 10 is extreamly secure.
Even more secure if the cracker has only breached a Solaris Zone, and your
monitoring script is running from the global zone.

With security monitoring it is best to either do it properly, or, do the
best you can and let the powers that be know what the flaws are. If the
logs are ever needed for a security incident - then you want something
you can rely on.
> On the other hand, I''m quickly going through SunSHILED user guide,
I see it can monitor any user''s any action, but not sure if it can
monitor "any access" to a file, any experience that you can share
about setting it?
A while back I setup and helped setup BSM auditing (SunSHIELD) on several
systems - it is a powerful tool that can solve many security monitoring
needs. I remember being told that the log files were inscruitable to anyone
without a PhD in computer science -- but after spending some time
with the SunSHIELD manual, I found the log files weren''t so bad at all.
They are very suitable for post processing in Perl.

If I get a chance I''ll blog about how I like to configure BSM, and 
explain the roles of BSM and DTrace side by side...

no worries,

Brendan

-- 
Brendan
[CA, USA]

David Chen wrote:
> Hi ZhiJun,
> Wow! Great, it''s almost what I want. Thanks a lot.
>   
Glad to be able to help.
> I tried the script and found a little problem, when I did "mv file
file.bak", i.e. rename the file under same directory, it can''t be
detected, I believe it dues to the internal implementation of "mv",
but I don''t know which VFS function I should use to detect it.
It is fop_rename() in this case
/fop_rename:entry
/ args[0]->v_path == "/export/home/tmp" && args[1] ==
"file" /
{
    printf("%s", stringof(args[0]->v_path));
    printf("/%s", stringof(args[1]));
}/
> BTW, I can''t find VFS functions in solaris dtrace guide, where can
I look for the reference of VFS?
>   
<<it would help to have a good understanding of the VFS layer;
<<Chapter 14 of Solaris Internals 2nd edition is a great reference.

As Brendan has pointed out,"Solaris Internals 2nd" is a good reference
Also you can reference the source code via
opensolaris.org
> Additionally, I want to add some other info into the output, for example,
the user id, command name, date and time, could you pls help for a example?
Please refer to "Solaris Dynamic Tracing Guide" Chapter 3, buidtin 
variables id,execname,timestamp/vtimestamp would likely to be your
choice
> I''m totally a newbie on dtrace and I only know using
"execname" to get command info, it works fine for commands such as
"cp", "cat",
"more","touch","vi" etc, but for some other
command ("crypt" as I tried ), the output command info is
"ksh", is there any way to get the correct command info?
>   
I don''t have much point on this issue.
To get the correct command info,I guess,as you can access thread 
structure via some builtin variables in D-script, perhaps you can get 
the command name from these structures.But currently I cannot remember 
where the command name is stored...

Regards,
Zhijun
> Regards
> David
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070123/aba20eab/attachment.html>

Hi Brendan, ZhiJun,
Really appreciate your kind help. I''ll go on with my job using
ZhiJun''s script for schedule, although it may be some far away from
"security monitoring", it does provide "limit"
functionality.
It''s good experience for me to discuss with you and learn a lot from
you, many thanks again.

Regards
David
 
 
This message posted from opensolaris.org

>
>> I''m totally a newbie on dtrace and I only know using
"execname" to
>> get command info, it works fine for commands such as "cp",
"cat",
>> "more","touch","vi" etc, but for some
other command ("crypt" as I
>> tried ), the output command info is "ksh", is there any way
to get
>> the correct command info?
>>   
> I don''t have much point on this issue.
> To get the correct command info,I guess,as you can access thread 
> structure via some builtin variables in D-script, perhaps you can get 
> the command name from these structures.But currently I cannot remember 
> where the command name is stored...
>
After looking up in the code,I found that the disired information can be 
found via user structure.You can try adding the below two lines into the 
simple D-script and watch the output.This may help to explain why output 
command info is "ksh" in your case.

/printf("%s\n", curthread->t_procp->p_user.u_comm);
printf("%s\n", curthread->t_procp->p_user.u_psargs);
/
Best,
Zhijun
/
/

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20070124/fed6fee7/attachment.html>

Hi ZhiJun,
I''ve tried the user structure, for a "crypt" command I
entered, the output was "ksh -o vi", I think maybe it''s
forked by "crypt" that actually touches the file. Anyway, an access
was detected although the "real command" is not known, it''s
acceptable :P.
I meet another issue, I try to monitor several files (for example, /tmp/core*): 

fop_open:entry
/ strstr((*args[0])->v_path, "/tmp/core") != NULL /
{
        printf("\n%Y\n",walltimestamp);
       
printf("userid=%d,operation=%s,file=%s\n",uid,curthread->t_procp->p_user
.u_psargs,stringof((*args[0])->v_path));
}

but during script compilation, it says "strstr" is undefined. 
Any other way can do it?

thanks and regards
David
 
 
This message posted from opensolaris.org

It works fine to me.
What version of Solaris you are using?

Best,
Zhijun

David Chen wrote:
> Hi ZhiJun,
> I''ve tried the user structure, for a "crypt" command I
entered, the output was "ksh -o vi", I think maybe it''s
forked by "crypt" that actually touches the file. Anyway, an access
was detected although the "real command" is not known, it''s
acceptable :P.
> I meet another issue, I try to monitor several files (for example,
/tmp/core*):
>
> fop_open:entry
> / strstr((*args[0])->v_path, "/tmp/core") != NULL /
> {
>         printf("\n%Y\n",walltimestamp);
>        
printf("userid=%d,operation=%s,file=%s\n",uid,curthread->t_procp->p_user
> .u_psargs,stringof((*args[0])->v_path));
> }
>
> but during script compilation, it says "strstr" is undefined. 
> Any other way can do it?
>
> thanks and regards
> David
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

David Chen wrote:
> but during script compilation, it says "strstr" is undefined. 
I think this is because strstr was introduced in Nevada. You''re
probably
using S10, right?

Sorry, I have no solution handy :-(

Michael
-- 
Michael Schuster
Sun Microsystems, Inc.

Hi Mike,
Yes I''m using Solaris 10, do you know any other string function to do a
"partial match"?
I know stardand C library has "strstr" function, is there a way for
dtrace script to call a C function?

thanks
David
 
 
This message posted from opensolaris.org

David Chen wrote:
> Hi Mike,
> Yes I''m using Solaris 10, do you know any other string function to
do a
> "partial match"? I know stardand C library has "strstr"
function, is
> there a way for dtrace script to call a C function?
you can use system() to call a *program*, but not, AFAIK, in a predicate. 
system() doesn''t return a value, so that''s be problematic in a
predicate
anyway ;-)

Michael
-- 
Michael Schuster
Sun Microsystems, Inc.

On Thu, Jan 25, 2007 at 01:02:08AM -0800, David Chen
wrote:
> Hi Mike,
> Yes I''m using Solaris 10, do you know any other string function to
do a "partial match"?
> I know stardand C library has "strstr" function, is there a way
for dtrace script to call a C function?
There''s no way to invoke a C function. There are a bunch of reasons for
this
but the short answer is: safety. The strstr() DTrace subroutine will appear
in an update to Solaris 10 probably within the next several weeks.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Thu, Jan 25, 2007 at 10:14:11AM +0100, Michael Schuster
wrote:
> David Chen wrote:
> >Hi Mike,
> >Yes I''m using Solaris 10, do you know any other string
function to do a
> >"partial match"? I know stardand C library has
"strstr" function, is
> >there a way for dtrace script to call a C function?
> 
> you can use system() to call a *program*, but not, AFAIK, in a predicate. 
> system() doesn''t return a value, so that''s be problematic
in a predicate
> anyway ;-)
There''s some confusion here that''s worth clearing up. In
DTrace when you
trace() something, that datum is recorded to an in-kernel buffer which is
later read and displayed by the user-land client (often dtrace(1M)). The
system action works similarly: a string is traced to the in-kernel buffer
and the client then executes that string rather than simply printing it
to the screen.

This program, for example, will print a bunch of stack traces:

syscall:::entry
{
	system("pstack %d", pid);
}

But those stacks will happen well after the event has occurred, whereas the
ustack() action records the stack _at that time_. The scope of what can be
done in "probe context" is limited due to issues of safety, security,
and
ubiquity of tracing.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

This is what I thought of when you mentioned file activity.  But I''m
old...

  http://www.lecb.ncifcrf.gov/~toms/atchange.html

Sorry for off-topic.

Dale
 
 
This message posted from opensolaris.org