dtrace discuss - Feb 2009 - df -k takes too long only at random times

How do I find out why df -k is taking soo long to respond only sometimes?


-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Asif Iqbal skrev:
> How do I find out why df -k is taking soo long to respond only sometimes?
> 
> 
sunsolve? ;-)

I assume that was not the answer you were looking for, but CR 6499704 
might be your answer.

Asif Iqbal wrote:
> How do I find out why df -k is taking soo long to respond only sometimes?
apply some analysing techniques:
- define "soo long".
- what are you running ''df -k'' on? the whole system or only a
specific
filesystem?
- is an NFS-mounted FS among the list of FSs examined?
- are there any messages in /var/adm/messages corresponding to the times 
where you see the slowdown?
- describe the behaviour in a little more detail ...
- what OS are you using? (this is not a joke question)
- what''s the machine setup? how many users, etc...
- ... (you get the idea I hope)

is this something related to DTrace, or is dtrace-discuss the first alias 
that came to mind?

HTH
Michael
-- 
Michael Schuster        http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

> How do I find out why df -k is taking soo long to respond only
> sometimes?
> 
truss df -k

and watch for where the delay happens.  If the "pause" is pronounced,
you should see a system call where it hangs for a moment.  Then break it
down from there.

Chip

On Thu, Feb 12, 2009 at 4:59 PM, Michael Schuster
<Michael.Schuster at sun.com> wrote:
> Asif Iqbal wrote:
>>
>> How do I find out why df -k is taking soo long to respond only
sometimes?
>
> apply some analysing techniques:
> - define "soo long".
more than a min. however I will try the truss method, like Chip
Bennett suggesting,
first to see where in the system call it is delaying
> - what are you running ''df -k'' on? the whole system or
only a specific
> filesystem?
just `df -k'' , so yes the whole system
> - is an NFS-mounted FS among the list of FSs examined?
there is no nfs mounted fs in this host
> - are there any messages in /var/adm/messages corresponding to the times
> where you see the slowdown?
I am not sure exactly when it is slow. It happens at random time and delays some
monitor script which calls `df -k'' hourly and considers `df
-k'' failed
if it does not
respond in 60 secs. May be add a logic to the script to run some
`truss'' or `dtrace''
at that time to collect some system state info
> - describe the behaviour in a little more detail ...
df -k does takes longer than 60 secs to result with an output, sometimes.
> - what OS are you using? (this is not a joke question)
Solaris 10 sparc update 3 on Sun Fire 420R with 4G mem
> - what''s the machine setup? how many users, etc...
Usually one or two user login
> - ... (you get the idea I hope)
>
> is this something related to DTrace, or is dtrace-discuss the first alias
> that came to mind?
No. I am looking for the system calls when df -k is slow. Dtrace can be run
in daemon mode and if crafted correctly can collect tons of output *only* when
`df -k'' takes longer than 60 sec to output

>
> HTH
> Michael
> --
> Michael Schuster        http://blogs.sun.com/recursion
> Recursion, n.: see ''Recursion''
>


-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Hi,

I''ve seen this as well in a specific circumstance on my server.  I run
an M4000, Solaris 10 U4, Oracle 10.2.0.3, UFS filesystems managed by
SVM (RAID 5), talking to Storedge 3510''s and 6140''s.  I use
RMAN to
make backups of the database to a 3Tb filesystem mounted on /backup.

In particular, when I delete obsolete RMAN backups using the command
RMAN>   delete obsolete noprompt;

For about 2 minutes after the delete returns control to the terminal,
if I run df -k in another terminal, it will pause or hang while trying
to report on /backup.  I can do other operations on /backup.  I can
CD, ls files, du -sk directories.  It''s only the df -k that causes any
problems.  After about 2 minutes, the df -k completes, and fron them
after I can run subsequent df -k with no problems.

It''s not dtrace or truss data, but I hope it helps.  I''m off
to run
truss myself.  Thanks for the tip

Glen Parker





On Fri, Feb 13, 2009 at 7:20 AM, Asif Iqbal <vadud3 at gmail.com>
wrote:
> On Thu, Feb 12, 2009 at 4:59 PM, Michael Schuster
> <Michael.Schuster at sun.com> wrote:
>> Asif Iqbal wrote:
>>>
>>> How do I find out why df -k is taking soo long to respond only
sometimes?
>>
>> apply some analysing techniques:
>> - define "soo long".
>
> more than a min. however I will try the truss method, like Chip
> Bennett suggesting,
> first to see where in the system call it is delaying
>
>> - what are you running ''df -k'' on? the whole system
or only a specific
>> filesystem?
>
> just `df -k'' , so yes the whole system
>
>> - is an NFS-mounted FS among the list of FSs examined?
>
> there is no nfs mounted fs in this host
>
>> - are there any messages in /var/adm/messages corresponding to the
times
>> where you see the slowdown?
>
> I am not sure exactly when it is slow. It happens at random time and delays
some
> monitor script which calls `df -k'' hourly and considers `df
-k'' failed
> if it does not
> respond in 60 secs. May be add a logic to the script to run some
> `truss'' or `dtrace''
> at that time to collect some system state info
>
>> - describe the behaviour in a little more detail ...
>
> df -k does takes longer than 60 secs to result with an output, sometimes.
>
>> - what OS are you using? (this is not a joke question)
>
> Solaris 10 sparc update 3 on Sun Fire 420R with 4G mem
>
>> - what''s the machine setup? how many users, etc...
>
> Usually one or two user login
>
>> - ... (you get the idea I hope)
>>
>> is this something related to DTrace, or is dtrace-discuss the first
alias
>> that came to mind?
>
> No. I am looking for the system calls when df -k is slow. Dtrace can be run
> in daemon mode and if crafted correctly can collect tons of output *only*
when
> `df -k'' takes longer than 60 sec to output
>
>
>>
>> HTH
>> Michael
>> --
>> Michael Schuster        http://blogs.sun.com/recursion
>> Recursion, n.: see ''Recursion''
>>
>
>
>
> --
> Asif Iqbal
> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

Glen Parker skrev:
> Hi,
> 
> I''ve seen this as well in a specific circumstance on my server.  I
run
> an M4000, Solaris 10 U4, Oracle 10.2.0.3, UFS filesystems managed by
> SVM (RAID 5), talking to Storedge 3510''s and 6140''s.  I
use RMAN to
> make backups of the database to a 3Tb filesystem mounted on /backup.
> 
> In particular, when I delete obsolete RMAN backups using the command
> RMAN>   delete obsolete noprompt;
> 
> For about 2 minutes after the delete returns control to the terminal,
> if I run df -k in another terminal, it will pause or hang while trying
> to report on /backup.  I can do other operations on /backup.  I can
> CD, ls files, du -sk directories.  It''s only the df -k that causes
any
> problems.  After about 2 minutes, the df -k completes, and fron them
> after I can run subsequent df -k with no problems.
Thats is almost certainly the CR is mentioned before, CR 6499704.

It is statvfs64 that takes a long time after deleting large files on 
logging ufs.
> 
> It''s not dtrace or truss data, but I hope it helps.  I''m
off to run
> truss myself.  Thanks for the tip
> 
> Glen Parker
> 
> 
> 
> 
> 
> On Fri, Feb 13, 2009 at 7:20 AM, Asif Iqbal <vadud3 at gmail.com>
wrote:
>> On Thu, Feb 12, 2009 at 4:59 PM, Michael Schuster
>> <Michael.Schuster at sun.com> wrote:
>>> Asif Iqbal wrote:
>>>> How do I find out why df -k is taking soo long to respond only
sometimes?
>>> apply some analysing techniques:
>>> - define "soo long".
>> more than a min. however I will try the truss method, like Chip
>> Bennett suggesting,
>> first to see where in the system call it is delaying
>>
>>> - what are you running ''df -k'' on? the whole
system or only a specific
>>> filesystem?
>> just `df -k'' , so yes the whole system
>>
>>> - is an NFS-mounted FS among the list of FSs examined?
>> there is no nfs mounted fs in this host
>>
>>> - are there any messages in /var/adm/messages corresponding to the
times
>>> where you see the slowdown?
>> I am not sure exactly when it is slow. It happens at random time and
delays some
>> monitor script which calls `df -k'' hourly and considers `df
-k'' failed
>> if it does not
>> respond in 60 secs. May be add a logic to the script to run some
>> `truss'' or `dtrace''
>> at that time to collect some system state info
>>
>>> - describe the behaviour in a little more detail ...
>> df -k does takes longer than 60 secs to result with an output,
sometimes.
>>
>>> - what OS are you using? (this is not a joke question)
>> Solaris 10 sparc update 3 on Sun Fire 420R with 4G mem
>>
>>> - what''s the machine setup? how many users, etc...
>> Usually one or two user login
>>
>>> - ... (you get the idea I hope)
>>>
>>> is this something related to DTrace, or is dtrace-discuss the first
alias
>>> that came to mind?
>> No. I am looking for the system calls when df -k is slow. Dtrace can be
run
>> in daemon mode and if crafted correctly can collect tons of output
*only* when
>> `df -k'' takes longer than 60 sec to output
>>
>>
>>> HTH
>>> Michael
>>> --
>>> Michael Schuster        http://blogs.sun.com/recursion
>>> Recursion, n.: see ''Recursion''
>>>
>>
>>
>> --
>> Asif Iqbal
>> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

On 02/13/09 04:20, Asif Iqbal wrote:
>> - what OS are you using? (this is not a joke question)
> 
> Solaris 10 sparc update 3 on Sun Fire 420R with 4G mem
you should be aware that you''re posting on opensolaris.org about an OS
that
is NOT OpenSolaris. Sun sells maintainance contracts for Solaris 10, please 
follow up by filing a case with Services.
>> is this something related to DTrace, or is dtrace-discuss the first
alias
>> > that came to mind?
> 
> No. I am looking for the system calls when df -k is slow
... especially as your request is NOT DTrace-related.

Michael
-- 
Michael Schuster 	http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

On Fri, Feb 13, 2009 at 12:20 PM, Michael Schuster
<Michael.Schuster at sun.com> wrote:
> On 02/13/09 04:20, Asif Iqbal wrote:
>
>>> - what OS are you using? (this is not a joke question)
>>
>> Solaris 10 sparc update 3 on Sun Fire 420R with 4G mem
>
> you should be aware that you''re posting on opensolaris.org about
an OS that
> is NOT OpenSolaris. Sun sells maintainance contracts for Solaris 10, please
dtrace is on solaris 10 as well, in case you are not aware of that.
> follow up by filing a case with Services.
>
>>> is this something related to DTrace, or is dtrace-discuss the first
alias
>>> > that came to mind?
>>
>> No. I am looking for the system calls when df -k is slow
>
> ... especially as your request is NOT DTrace-related.
It is since I am trying to craft a dtrace to catch why df -k is slow.

I actually found a dtrace script / tool that I will give a try

http://blogs.sun.com/gregns/entry/solaris_appcrash_discussion_and_updates
>
> Michael
> --
> Michael Schuster        http://blogs.sun.com/recursion
> Recursion, n.: see ''Recursion''
>

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

On 02/13/09 10:04, Asif Iqbal wrote:
> On Fri, Feb 13, 2009 at 12:20 PM, Michael Schuster
> <Michael.Schuster at sun.com> wrote:
>> On 02/13/09 04:20, Asif Iqbal wrote:
>>
>>>> - what OS are you using? (this is not a joke question)
>>> Solaris 10 sparc update 3 on Sun Fire 420R with 4G mem
>> you should be aware that you''re posting on opensolaris.org
about an OS that
>> is NOT OpenSolaris. Sun sells maintainance contracts for Solaris 10,
please
> 
> dtrace is on solaris 10 as well, in case you are not aware of that.
that is not the point. If you have a DTrace-specific question, you''re
at
the right place. If you have a generic problem with your Solaris 10 
machine, only the DTrace-specific parts belong here.
>> follow up by filing a case with Services.
>>
>>>> is this something related to DTrace, or is dtrace-discuss the
first alias
>>>>> that came to mind?
>>> No. I am looking for the system calls when df -k is slow
>> ... especially as your request is NOT DTrace-related.
> 
> It is since I am trying to craft a dtrace to catch why df -k is slow.
that''s not how I read the emails so far.
> I actually found a dtrace script / tool that I will give a try
> 
> http://blogs.sun.com/gregns/entry/solaris_appcrash_discussion_and_updates
that''s good then, good luck.

Michael
-- 
Michael Schuster 	http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''