dtrace discuss - May 2008 - [FWD: Slow pkginstalls due to long door_calls to nscd]

Hi Justin,
Attached is a link to the dtrace script we wrote for the sparks project to
monitor
requests going to/from nscd.  We have a couple of variations on the theme,
plus a few others that we still need to post to the sparks /dtrace page.

We also are working with someone who is working on a set of USDT probes
for some of the key internal components of our nss_ldap name service switch.

The script is here:
http://www.opensolaris.org/os/project/sparks/dtrace/nscd.d

Current sparks dtrace news, scripts or other updates can be found here:
http://www.opensolaris.org/os/project/sparks/dtrace/
the Sparks project is:
http://www.opensolaris.org/os/project/sparks/

If you gather any performance information that may be helpful, please let us
know
on the sparks-discuss alias or forward it to me.  I will see that the
performance
problem gets looked into.

Thanks again,
Doug.


Nicolas Williams wrote:
> ----- Forwarded message from Justin Lloyd <jlloyd at
digitalglobe.com> -----
> 
> Date: Wed, 21 May 2008 11:50:50 -0600
> From: Justin Lloyd <jlloyd at digitalglobe.com>
> Subject: [dtrace-discuss] Slow pkginstalls due to long door_calls to nscd
> To: dtrace-discuss at opensolaris.org
> 
> Hi all,
> 
> I am installing a zone onto two different V445s running S10U4 and the
> zones are taking hours to install (about 1000 packages), that is, the
> problem is identical on both systems. A bit of trussing and dtracing has
> shown that the pkginstalls being run by the zoneadm install are making
> door_call calls to nscd that are taking very long, so far observed to be
> 5 to 40 seconds, but always in times divisible by 5 seconds (5, 10, 15,
> etc.), leading to the thought that it is one or more name service
> timeouts. Eventually, once the door_calls finish, the actual package
> install is very quick.
> 
> There may be a simple misconfiguration on both systems, but I
can''t
> figure out what that might be. Additionally, I don''t know why the
> pkginstalls are making door_calls to begin with, and I''m not sure
how to
> figure that out. So that''s one my main question:
> 
>    *** How can I find out exactly what the pkginstalls are asking nscd
> to resolve? ***
> 
> Hopefully that will lead me (and the Sun engineer with whom I have a
> case open) to an understanding of what is leading to the presumed
> timeouts.
> 
> FWIW, here''s a partial truss with timestamps of an existing
pkginstall
> process. Once it hit the ioctl(3,...) it flew through the package
> installation. Note the times around the door_calls: 13, 10 and 15
> seconds (the first isn''t a multiple of 5 since I started the truss
> during the call).
> 
> Base time stamp:  1210181159.9172  [ Wed May  7 17:25:59 GMT 2008 ]
> door_call(4, 0xFFBFD030)        (sleeping...)
> 1210472.1302    door_call(4, 0xFFBFD030)                        = 0
> 1210472.1306    open("/dev/udp6", O_RDONLY)                     =
5
> 1210472.1310    ioctl(5, SIOCGLIFNUM, 0xFFBFD2DC)               = 0
> 1210472.1312    close(5)                                        = 0
> 1210472.1312    getuid()                                        = 0 [0]
> 1210472.1313    getuid()                                        = 0 [0]
> 1210472.1314    door_info(4, 0xFFBFCF88)                        = 0
> door_call(4, 0xFFBFD030)        (sleeping...)
> 1210482.1503    door_call(4, 0xFFBFD030)                        = 0
> 1210482.1505    getuid()                                        = 0 [0]
> 1210482.1505    getuid()                                        = 0 [0]
> 1210482.1506    door_info(4, 0xFFBFCF88)                        = 0
> door_call(4, 0xFFBFD030)        (sleeping...)
> 1210497.1802    door_call(4, 0xFFBFD030)                        = 0
> 1210497.1804    ioctl(3, ((''m''<<8)|7), 0xFFBFD8EC) 
= 0
> 
> Thanks,
> Justin
> 
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
> 
> 
> ----- End forwarded message -----

I''ve replied to Doug privately, but this could be of interest to others
here. It appears that on my 08/07 system (and previous builds back to 03/05),
nscd is stripped and thus cannot resolve the symbols to enable probes in the pid
provider. So this script won''t work since it attempts to trace the
switcher() function in nscd.

Justin


--
This message posted from opensolaris.org

Boy, doesn''t that just scream out "static probes"? If there
was ever
an executable that wants static probes, it''s nscd.

Justin Lloyd wrote:
> I''ve replied to Doug privately, but this could be of interest to
others here. It appears that on my 08/07 system (and previous builds back to
03/05), nscd is stripped and thus cannot resolve the symbols to enable probes in
the pid provider. So this script won''t work since it attempts to trace
the switcher() function in nscd.
> 
> Justin
> 
> 
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
blu

There are two rules in life:
Rule 1- Don''t tell people everything you know
----------------------------------------------------------------------
Brian Utterback - Solaris RPE, Sun Microsystems, Inc.
Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom