dtrace discuss - Sep 2008 - how to use dtrace to find pid not release the file deleted ?

Hello,

file system was filled up and files were deleled by someone while there still
had processes referencing those deleted files,
the disk space was not released,   the result of du was different from output
of df.
now how to use dtrace to  find the pids that were still referencing those
deleted files ?

# dtrace -n ''io:::start { @files[pid, execname,
args[2]->fi_pathname] sum(args[0]->b_bcount);} tick-5sec
{exit(0);}''

this will find out the processes that have write IO, such as tape backup.
if  cmd of more or tail still were referencing the deleted file,  the above
dtrace could not file the process of more or tail,
if there were no read IO from cmd of more or tail.

so do I still IO provider to trace the process ? how ?

Thanks,

James Yang
Deutsche Bank US
---

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080920/dea6715d/attachment.html>

On Sat, Sep 20, 2008 at 4:19 PM, Jianhua Yang <jianhua.yang at db.com>
wrote:
> Hello,
>
> file system was filled up and files were deleled by someone while there
> still had processes referencing those deleted files,
> the disk space was not released, the result of du was different from output
> of df.
> now how to use dtrace to find the pids that were still referencing those
> deleted files ?
I don''t use dtrace for this - I use find.  For example:

# find /proc/*/fd -type f -links 0 \! -size 0 -ls | sort -n +1

You can view the (e.g) log file with commands like:

# tail -f /proc/<pid>/fd/<fd>

If you find that there is a process that you can''t kill that has a
huge file, you can truncate it with:

# cp /dev/null /proc/<pid>/fd/<fd>

Be sure you have the right one.  See the tail command above.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

Hi Mike,

thanks a lot for your kind reply !!!
> I don''t use dtrace for this - I use find.  For example:
>
> # find /proc/*/fd -type f -links 0 \! -size 0 -ls | sort -n +1
      this will list the inode number, but does not tell the file name, use ps
to list
      and check the pid cost a lot of time.

      so I''d like to write one line dtrace script to get all the pids
who
still reference
      those deleted files
> You can view the (e.g) log file with commands like:
>
> # tail -f /proc/<pid>/fd/<fd>
      most of the time we do not know the pid and the file
name.
>
> If you find that there is a process that you can''t kill that has a
> huge file, you can truncate it with:
>
> # cp /dev/null /proc/<pid>/fd/<fd>
>
> Be sure you have the right one.  See the tail command above.
>
Thanks,

James Yang
Global Unix Support,  IES,  GTO
Deutsche Bank US
Phone:   201-593-1360
Email :   jianhua.yang at db.com
Pager :  1-800-946-4646  PIN# 6105618
CR: NYC_UNIX_ES_US_UNIX_SUPPORT
http://dcsupport.ies.gto.intranet.db.com/


"Mike Gerdts" <mgerdts at gmail.com> wrote on 09/20/2008
08:43:49 PM:
> On Sat, Sep 20, 2008 at 4:19 PM, Jianhua Yang <jianhua.yang at
db.com> wrote:
> > Hello,
> >
> > file system was filled up and files were deleled by someone while
there
> > still had processes referencing those deleted files,
> > the disk space was not released, the result of du was different from
output
> > of df.
> > now how to use dtrace to find the pids that were still referencing
those
> > deleted files ?
>
> I don''t use dtrace for this - I use find.  For example:
>
> # find /proc/*/fd -type f -links 0 \! -size 0 -ls | sort -n +1
>
> You can view the (e.g) log file with commands like:
>
> # tail -f /proc/<pid>/fd/<fd>
>
> If you find that there is a process that you can''t kill that has a
> huge file, you can truncate it with:
>
> # cp /dev/null /proc/<pid>/fd/<fd>
>
> Be sure you have the right one.  See the tail command above.
>
> --
> Mike Gerdts
> http://mgerdts.blogspot.com/
---

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080921/ea504de1/attachment.html>

Jianhua Yang wrote:
> Hi Mike,
> 
> thanks a lot for your kind reply !!!
> 
>> I don''t use dtrace for this - I use find.  For example:
>>
>> # find /proc/*/fd -type f -links 0 \! -size 0 -ls | sort -n +1
>       this will list the inode number, but does not tell the file name, use
ps
> to list
>       and check the pid cost a lot of time.
> [...]
     Maybe I''m missing something, but what do you mean by the
"name" of a file that has been deleted?  Remember, a file''s
name is just some text in a directory entry, and once the
directory entry is gone ...

-- 
Eric.Sosman at sun.com

Eric,

my original question:

file system was filled up and files were deleled by someone while there still
had processes referencing those deleted files,
the disk space was not released,   the result of du was different from output
of df.
now how to use dtrace to  find the pids that were still referencing those
deleted files ?

# dtrace -n ''io:::start { @files[pid, execname,
args[2]->fi_pathname] sum(args[0]->b_bcount);} tick-5sec
{exit(0);}''

this will find out the processes that have write IO, such as tape backup.
if  cmd of more or tail still were referencing the deleted file,  the above
dtrace could not file the process of more or tail,
if there were no read IO from cmd of more or tail.

so do I still IO provider to trace the process ? how ?


Eric.Sosman at Sun.COM wrote on 09/21/2008 10:39:08 AM:
> Jianhua Yang wrote:
> > Hi Mike,
> >
> > thanks a lot for your kind reply !!!
> >
> >> I don''t use dtrace for this - I use find.  For example:
> >>
> >> # find /proc/*/fd -type f -links 0 \! -size 0 -ls | sort -n +1
> >       this will list the inode number, but does not tell the file
> name, use ps
> > to list
> >       and check the pid cost a lot of time.
> > [...]
>
>      Maybe I''m missing something, but what do you mean by the
> "name" of a file that has been deleted?  Remember, a
file''s
> name is just some text in a directory entry, and once the
> directory entry is gone ...
      # find /proc/*/fd -type f -links 0 \! -size 0 -ls | sort -n +1
      20 5245448 -r-------T   0 root     root   5368709120 Sep 20 20:59
/proc/1425/fd/0
      the cmd shows the inode # of 0 only,  I want to know the filename of
this inode.
>
> --
> Eric.Sosman at sun.com
---

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080921/3c257ffb/attachment.html>

On Sun, Sep 21, 2008 at 8:55 AM, Jianhua Yang <jianhua.yang at db.com>
wrote:
> Hi Mike,
>
> thanks a lot for your kind reply !!!
>
>> I don''t use dtrace for this - I use find.  For example:
>>
>> # find /proc/*/fd -type f -links 0 \! -size 0 -ls | sort -n +1
> this will list the inode number, but does not tell the file name
If the file has been removed, nothing will tell you the file name - it
no longer has a name.
> use ps to list and check the pid cost a lot of time.
The "find" command that I listed will take a lot more time than ps.
On an Ultra 2 (great machine in 1996) running a fairly default
installation of snv_90, find takes about 0.10 sec of CPU time.  In
comparison, the following takes on average about 5.9 seconds (59 times
the CPU time of find):

dtrace -n ''BEGIN { printf("Hello World!\n"); exit(0);
}''

If you have a system with tens of thousands of processes or with some
processes with many thousands of open files and those processes are
sensitive to a millisecond or so of a pause, then you have something
to worry about.

Even the following is a lot more efficient than a minimal dtrace
script that doesn''t even do what you need:

# time ksh -c ''find /proc/*/fd -type f -links 0 \! -size 0 \
      | while IFS=/ read j1 j2 pid j3 fd ; do \
            echo "PID: $pid FD $fd"; \
            echo "Command: $(ps -o args= -p $pid)" ; \
            ls -l /proc/$pid/fd/$fd ; \
      done''
PID: 9 FD: 4
Command: /lib/svc/bin/svc.configd
-rw-------   0 root     root        2048 Sep 20 16:29 /proc/9/fd/4
PID: 9 FD: 6
Command: /lib/svc/bin/svc.configd
-rw-------   0 root     root        2048 Sep 20 16:29 /proc/9/fd/6

real    0m0.210s
user    0m0.042s
sys     0m0.168s

> so I''d like to write one line dtrace script to get all the pids
who still reference
> those deleted files
With dtrace, you will need to wait until a process does something
(e.g. writes) to the file descriptor. If the process that has the
large, deleted, open file never does anything with it then dtrace will
never see it.  Perhaps it is an administrator doing "more biglog" then
realized it was time to leave for a vacation and didn''t quit more or
log out.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/