Hola, I am trying to isolate the memory leak I suspect in a mailman installation ? I found: http://blogs.sun.com/sanjeevb/date/200506 It gives an error: god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 dtrace: failed to compile script ./memleak.d: line 3: probe description pid10312:libc.so.1:malloc:entry does not match any probes I am on SunOS 5.10 Generic_127112-07 i86pc i386 i86pc Are there some better scripts for isolating memory leaks? thanks Fletch. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080701/a786b8db/attachment.html>
Fletcher Cocquyt wrote:> Hola, I am trying to isolate the memory leak I suspect in a mailman > installation ? I found: > http://blogs.sun.com/sanjeevb/date/200506 > > It gives an error: > > god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 > dtrace: failed to compile script ./memleak.d: line 3: probe description > pid10312:libc.so.1:malloc:entry does not match any probesthis begs the question: is there a process with pid 10312? Michael -- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''
Yes: god at irt-smtp-02:~ 10:02am 52 # ps -ef | grep 10312 mailman 10312 22726 0 09:13:19 ? 0:05 /bin/python /opt/mailman-2.1.9/bin/qrunner --runner=VirginRunner:0:1 -s This is the error for no such process: god at irt-smtp-02:~ 10:04am 53 # ./memleak.d 666 dtrace: failed to compile script ./memleak.d: line 3: failed to grab process 666 god at irt-smtp-02:~ 10:04am 54 # ps -ef | grep 666 root 20386 19893 0 10:04:49 pts/1 0:00 grep 666 god at irt-smtp-02:~ 10:04am 55 # I''m hoping there is a "fresher" script than this 3yr old one I found via the top google hit for: " dtrace script for memory leak" The 2nd and third hits are now this thread - gah! I know memory leaks are a non-trivial problem - but the rate of this one is so egregious as to require twice daily restarts of mailman - I like the logic behind checking the alloc/free calls and matching them up... Any tips appreciated - Thanks, Fletcher On 7/1/08 9:41 AM, "Michael Schuster" <Michael.Schuster at Sun.COM> wrote:> Fletcher Cocquyt wrote: >> Hola, I am trying to isolate the memory leak I suspect in a mailman >> installation ? I found: >> http://blogs.sun.com/sanjeevb/date/200506 >> >> It gives an error: >> >> god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 >> dtrace: failed to compile script ./memleak.d: line 3: probe description >> pid10312:libc.so.1:malloc:entry does not match any probes > > this begs the question: > is there a process with pid 10312? > > Michael-- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt at stanford.edu Phone: (650) 724-7485
Fletcher Cocquyt wrote:> Yes: > god at irt-smtp-02:~ 10:02am 52 # ps -ef | grep 10312 > mailman 10312 22726 0 09:13:19 ? 0:05 /bin/python > /opt/mailman-2.1.9/bin/qrunner --runner=VirginRunner:0:1 -soh ... I hadn''t realised you were talking about python. I seem to remember that there''s a dtrace provider for python ... but I may be mistaken - if I''m not, that may be your best bet. HTH Michael> > This is the error for no such process: > > god at irt-smtp-02:~ 10:04am 53 # ./memleak.d 666 > dtrace: failed to compile script ./memleak.d: line 3: failed to grab process > 666 > god at irt-smtp-02:~ 10:04am 54 # ps -ef | grep 666 > root 20386 19893 0 10:04:49 pts/1 0:00 grep 666 > god at irt-smtp-02:~ 10:04am 55 # > > I''m hoping there is a "fresher" script than this 3yr old one I found via the > top google hit for: " dtrace script for memory leak" > > The 2nd and third hits are now this thread - gah! > > I know memory leaks are a non-trivial problem - but the rate of this one is > so egregious as to require twice daily restarts of mailman - I like the > logic behind checking the alloc/free calls and matching them up... > > Any tips appreciated - > Thanks, > Fletcher > > > > On 7/1/08 9:41 AM, "Michael Schuster" <Michael.Schuster at Sun.COM> wrote: > >> Fletcher Cocquyt wrote: >>> Hola, I am trying to isolate the memory leak I suspect in a mailman >>> installation ? I found: >>> http://blogs.sun.com/sanjeevb/date/200506 >>> >>> It gives an error: >>> >>> god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 >>> dtrace: failed to compile script ./memleak.d: line 3: probe description >>> pid10312:libc.so.1:malloc:entry does not match any probes >> this begs the question: >> is there a process with pid 10312? >> >> Michael >-- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''
Hello Fletcher, From the error looks like dtrace is not able recognize it as probe. DTrace needs a signature for the function to be detected as probe. Probably this is missing in case of malloc. Just to double check this you could disassemble malloc and check if we have a "push'' instruction at the beginning. Thanks and regards, Sanjeev. Fletcher Cocquyt wrote:> Hola, I am trying to isolate the memory leak I suspect in a mailman > installation ? I found: > http://blogs.sun.com/sanjeevb/date/200506 > > It gives an error: > > god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 > dtrace: failed to compile script ./memleak.d: line 3: probe > description pid10312:libc.so.1:malloc:entry does not match any probes > > I am on SunOS 5.10 Generic_127112-07 i86pc i386 i86pc > > Are there some better scripts for isolating memory leaks? > > thanks > Fletch. > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >-- Solaris Revenue Products Engineering, India Engineering Center, Sun Microsystems India Pvt Ltd. Tel: x27521 +91 80 669 27521
Can you please provide a reference for disassembling malloc on Solaris 10? I am also pursuing the previous suggestion of a Python provider - this one seems to be against Python 2.5: http://blogs.sun.com/binujp/resource/pydtrace/diffs Thanks, Fletcher On 7/1/08 9:48 PM, "Sanjeev Bagewadi" <Sanjeev.Bagewadi at Sun.COM> wrote:> Hello Fletcher, > > From the error looks like dtrace is not able recognize it as probe. > DTrace needs a signature for the function to be detected as probe. > Probably this is missing in case of malloc. > > Just to double check this you could disassemble malloc and check if we > have a "push'' instruction at the beginning. > > Thanks and regards, > Sanjeev. > > Fletcher Cocquyt wrote: >> Hola, I am trying to isolate the memory leak I suspect in a mailman >> installation ? I found: >> http://blogs.sun.com/sanjeevb/date/200506 >> >> It gives an error: >> >> god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 >> dtrace: failed to compile script ./memleak.d: line 3: probe >> description pid10312:libc.so.1:malloc:entry does not match any probes >> >> I am on SunOS 5.10 Generic_127112-07 i86pc i386 i86pc >> >> Are there some better scripts for isolating memory leaks? >> >> thanks >> Fletch. >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org >> >
Fletcher, You could attach mdb to the running process and disassemble the routine in question : -- snip -- # mdb -p <pid> > malloc::dis libc.so.1`malloc: pushl %ebp libc.so.1`malloc+1: movl %esp,%ebp libc.so.1`malloc+3: pushl %ebx libc.so.1`malloc+4: pushl %esi libc.so.1`malloc+5: pushl %edi libc.so.1`malloc+6: call +0x5 <libc.so.1`malloc+0xb> libc.so.1`malloc+0xb: popl %ebx libc.so.1`malloc+0xc: addl $0x88fe1,%ebx -- snip -- So, in my case notice that the first instruction is "pushl". Thanks and regards, Sanjeev. Fletcher Cocquyt wrote:> Can you please provide a reference for disassembling malloc on Solaris 10? > I am also pursuing the previous suggestion of a Python provider - this one > seems to be against Python 2.5: > http://blogs.sun.com/binujp/resource/pydtrace/diffs > > Thanks, > Fletcher > > On 7/1/08 9:48 PM, "Sanjeev Bagewadi" <Sanjeev.Bagewadi at Sun.COM> wrote: > > >> Hello Fletcher, >> >> From the error looks like dtrace is not able recognize it as probe. >> DTrace needs a signature for the function to be detected as probe. >> Probably this is missing in case of malloc. >> >> Just to double check this you could disassemble malloc and check if we >> have a "push'' instruction at the beginning. >> >> Thanks and regards, >> Sanjeev. >> >> Fletcher Cocquyt wrote: >> >>> Hola, I am trying to isolate the memory leak I suspect in a mailman >>> installation ? I found: >>> http://blogs.sun.com/sanjeevb/date/200506 >>> >>> It gives an error: >>> >>> god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 >>> dtrace: failed to compile script ./memleak.d: line 3: probe >>> description pid10312:libc.so.1:malloc:entry does not match any probes >>> >>> I am on SunOS 5.10 Generic_127112-07 i86pc i386 i86pc >>> >>> Are there some better scripts for isolating memory leaks? >>> >>> thanks >>> Fletch. >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> dtrace-discuss mailing list >>> dtrace-discuss at opensolaris.org >>> >>> > > > > >-- Solaris Revenue Products Engineering, India Engineering Center, Sun Microsystems India Pvt Ltd. Tel: x27521 +91 80 669 27521
Fletcher, Mark Durney hit similar problem and while I was working with him and talking to my colleague he pointed out that we could use "function:offset" notation when we are using pid-provider. So, I have modified the script to enable the first instruction of malloc. Attached is the script. Please try it out and let me know if it works. If it does I shall update my blog to reflect it. NOTE : If there more functions which fail (for :entry) please replace "entry" with 0. Thanks and regards, Sanjeev. Sanjeev Bagewadi wrote:> Fletcher, > > You could attach mdb to the running process and disassemble the routine > in question : > -- snip -- > # mdb -p <pid> > > malloc::dis > libc.so.1`malloc: pushl %ebp > libc.so.1`malloc+1: movl %esp,%ebp > libc.so.1`malloc+3: pushl %ebx > libc.so.1`malloc+4: pushl %esi > libc.so.1`malloc+5: pushl %edi > libc.so.1`malloc+6: call +0x5 <libc.so.1`malloc+0xb> > libc.so.1`malloc+0xb: popl %ebx > libc.so.1`malloc+0xc: addl $0x88fe1,%ebx > -- snip -- > > So, in my case notice that the first instruction is "pushl". > > Thanks and regards, > Sanjeev. > Fletcher Cocquyt wrote: > >> Can you please provide a reference for disassembling malloc on Solaris 10? >> I am also pursuing the previous suggestion of a Python provider - this one >> seems to be against Python 2.5: >> http://blogs.sun.com/binujp/resource/pydtrace/diffs >> >> Thanks, >> Fletcher >> >> On 7/1/08 9:48 PM, "Sanjeev Bagewadi" <Sanjeev.Bagewadi at Sun.COM> wrote: >> >> >> >>> Hello Fletcher, >>> >>> From the error looks like dtrace is not able recognize it as probe. >>> DTrace needs a signature for the function to be detected as probe. >>> Probably this is missing in case of malloc. >>> >>> Just to double check this you could disassemble malloc and check if we >>> have a "push'' instruction at the beginning. >>> >>> Thanks and regards, >>> Sanjeev. >>> >>> Fletcher Cocquyt wrote: >>> >>> >>>> Hola, I am trying to isolate the memory leak I suspect in a mailman >>>> installation ? I found: >>>> http://blogs.sun.com/sanjeevb/date/200506 >>>> >>>> It gives an error: >>>> >>>> god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 >>>> dtrace: failed to compile script ./memleak.d: line 3: probe >>>> description pid10312:libc.so.1:malloc:entry does not match any probes >>>> >>>> I am on SunOS 5.10 Generic_127112-07 i86pc i386 i86pc >>>> >>>> Are there some better scripts for isolating memory leaks? >>>> >>>> thanks >>>> Fletch. >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> dtrace-discuss mailing list >>>> dtrace-discuss at opensolaris.org >>>> >>>> >>>> >> >> >> >> > > >-- Solaris Revenue Products Engineering, India Engineering Center, Sun Microsystems India Pvt Ltd. Tel: x27521 +91 80 669 27521 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: memleak.d URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080702/50968fb2/attachment.ksh>
Fletcher, First confirm that malloc is in your binary. arwen:nm a.out | grep malloc [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc Then key on any malloc. Something like: pid$target::malloc:return, pid$target::memalign:return, pid$target::realloc:return, pid$target::valloc:return rick -- Rickey C. Weisner Software Development and Performance Specialist Sun Microsystems, INC cell phone: 615-308-1147 email: rick.weisner at sun.com
Looks OK: god at irt-smtp-02:~ 7:22am 60 # !nm nm /bin/python | egrep malloc [3597] | 134599012| 0|FUNC |GLOB |0 |UNDEF |malloc [690] | 0| 0|FILE |LOCL |0 |ABS |obmalloc.c On 7/2/08 6:11 AM, "rickey c weisner" <rick.weisner at sun.com> wrote:> Fletcher, > First confirm that malloc is in your binary. > > arwen:nm a.out | grep malloc > [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc > > Then key on any malloc. > Something like: > pid$target::malloc:return, > pid$target::memalign:return, > pid$target::realloc:return, > pid$target::valloc:return > > rick-- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt at stanford.edu Phone: (650) 724-7485
Sanjeev, I get this with your new version: god at irt-smtp-02:~ 7:26am 64 # ./memleak2.d 7560 dtrace: failed to compile script ./memleak2.d: line 3: probe description pid7560:libc.so.1:malloc:0 does not match any probes god at irt-smtp-02:~ 7:27am 65 # ps -ef | grep 7560 mailman 7560 718 0 06:24:11 ? 0:05 /bin/python /opt/mailman-2.1.9/bin/qrunner --runner=BounceRunner:0:1 -s thanks On 7/2/08 4:29 AM, "Sanjeev Bagewadi" <Sanjeev.Bagewadi at Sun.COM> wrote:> Fletcher, > > Mark Durney hit similar problem and while I was working with him and > talking to > my colleague he pointed out that we could use "function:offset" notation > when we > are using pid-provider. > > So, I have modified the script to enable the first instruction of malloc. > > Attached is the script. Please try it out and let me know if it works. > If it does I shall update my blog to reflect it. > > NOTE : If there more functions which fail (for :entry) please replace > "entry" with 0. > > Thanks and regards, > Sanjeev. > > Sanjeev Bagewadi wrote: >> Fletcher, >> >> You could attach mdb to the running process and disassemble the routine >> in question : >> -- snip -- >> # mdb -p <pid> >>> malloc::dis >> libc.so.1`malloc: pushl %ebp >> libc.so.1`malloc+1: movl %esp,%ebp >> libc.so.1`malloc+3: pushl %ebx >> libc.so.1`malloc+4: pushl %esi >> libc.so.1`malloc+5: pushl %edi >> libc.so.1`malloc+6: call +0x5 <libc.so.1`malloc+0xb> >> libc.so.1`malloc+0xb: popl %ebx >> libc.so.1`malloc+0xc: addl $0x88fe1,%ebx >> -- snip -- >> >> So, in my case notice that the first instruction is "pushl". >> >> Thanks and regards, >> Sanjeev. >> Fletcher Cocquyt wrote: >> >>> Can you please provide a reference for disassembling malloc on Solaris 10? >>> I am also pursuing the previous suggestion of a Python provider - this one >>> seems to be against Python 2.5: >>> http://blogs.sun.com/binujp/resource/pydtrace/diffs >>> >>> Thanks, >>> Fletcher >>> >>> On 7/1/08 9:48 PM, "Sanjeev Bagewadi" <Sanjeev.Bagewadi at Sun.COM> wrote: >>> >>> >>> >>>> Hello Fletcher, >>>> >>>> From the error looks like dtrace is not able recognize it as probe. >>>> DTrace needs a signature for the function to be detected as probe. >>>> Probably this is missing in case of malloc. >>>> >>>> Just to double check this you could disassemble malloc and check if we >>>> have a "push'' instruction at the beginning. >>>> >>>> Thanks and regards, >>>> Sanjeev. >>>> >>>> Fletcher Cocquyt wrote: >>>> >>>> >>>>> Hola, I am trying to isolate the memory leak I suspect in a mailman >>>>> installation ? I found: >>>>> http://blogs.sun.com/sanjeevb/date/200506 >>>>> >>>>> It gives an error: >>>>> >>>>> god at irt-smtp-02:~ 9:21am 65 # ./memleak.d 10312 >>>>> dtrace: failed to compile script ./memleak.d: line 3: probe >>>>> description pid10312:libc.so.1:malloc:entry does not match any probes >>>>> >>>>> I am on SunOS 5.10 Generic_127112-07 i86pc i386 i86pc >>>>> >>>>> Are there some better scripts for isolating memory leaks? >>>>> >>>>> thanks >>>>> Fletch. >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> dtrace-discuss mailing list >>>>> dtrace-discuss at opensolaris.org >>>>> >>>>> >>>>> >>> >>> >>> >>> >> >> >> >-- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt at stanford.edu Phone: (650) 724-7485
Fletcher, This looks suspicious. Perhaps your malloc is not in libc ?> [690] | 0| 0|FILE |LOCL |0 |ABS |obmalloc.cremove the libc.so.1 from your probe description. rick On Wed, Jul 02, 2008 at 07:23:49AM -0700, Fletcher Cocquyt wrote:> Date: Wed, 02 Jul 2008 07:23:49 -0700 > From: Fletcher Cocquyt <fcocquyt at stanford.edu> > Subject: Re: [dtrace-discuss] Memory leak scripts > In-reply-to: <20080702081148.A1624 at arwen.sfbay.sun.com> > To: rickey c weisner <Rick.Weisner at Sun.COM> > Cc: dtrace-discuss at opensolaris.org > Thread-topic: [dtrace-discuss] Memory leak scripts > Thread-index: AcjcTz6M0DPfSSwvAEuYeh4RTBFiig=> X-PMX-Version: 5.4.1.325704 > X-Brightmail-Tracker: AAAAAA=> X-Antispam: No, score=0.0/5.0, scanned in 0.102sec at (localhost [127.0.0.1]) > by smf-spamd v1.3.1 - http://smfs.sf.net/ > User-Agent: Microsoft-Entourage/12.11.0.080522 > Original-recipient: rfc822;rick.weisner at sun.com > > Looks OK: > > god at irt-smtp-02:~ 7:22am 60 # !nm > nm /bin/python | egrep malloc > [3597] | 134599012| 0|FUNC |GLOB |0 |UNDEF |malloc > > > > On 7/2/08 6:11 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: > > > Fletcher, > > First confirm that malloc is in your binary. > > > > arwen:nm a.out | grep malloc > > [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc > > > > Then key on any malloc. > > Something like: > > pid$target::malloc:return, > > pid$target::memalign:return, > > pid$target::realloc:return, > > pid$target::valloc:return > > > > rick > > -- > Fletcher Cocquyt > Senior Systems Administrator > Information Resources and Technology (IRT) > Stanford University School of Medicine > > Email: fcocquyt at stanford.edu > Phone: (650) 724-7485 > >-- Rickey C. Weisner Software Development and Performance Specialist Sun Microsystems, INC cell phone: 615-308-1147 email: rick.weisner at sun.com
The below command should list all the available probemod and probefunc combination. Check if malloc is listed and if so against which probemod is it listed. $ dtrace -lv pid$target:::enter -p <pid> -Shiv On Wed, Jul 2, 2008 at 8:16 PM, rickey c weisner <rick.weisner at sun.com> wrote:> Fletcher, > This looks suspicious. Perhaps your malloc is not in libc ? >> [690] | 0| 0|FILE |LOCL |0 |ABS |obmalloc.c > remove the libc.so.1 from your probe description. > rick > > On Wed, Jul 02, 2008 at 07:23:49AM -0700, Fletcher Cocquyt wrote: >> Date: Wed, 02 Jul 2008 07:23:49 -0700 >> From: Fletcher Cocquyt <fcocquyt at stanford.edu> >> Subject: Re: [dtrace-discuss] Memory leak scripts >> In-reply-to: <20080702081148.A1624 at arwen.sfbay.sun.com> >> To: rickey c weisner <Rick.Weisner at Sun.COM> >> Cc: dtrace-discuss at opensolaris.org >> Thread-topic: [dtrace-discuss] Memory leak scripts >> Thread-index: AcjcTz6M0DPfSSwvAEuYeh4RTBFiig=>> X-PMX-Version: 5.4.1.325704 >> X-Brightmail-Tracker: AAAAAA=>> X-Antispam: No, score=0.0/5.0, scanned in 0.102sec at (localhost [127.0.0.1]) >> by smf-spamd v1.3.1 - http://smfs.sf.net/ >> User-Agent: Microsoft-Entourage/12.11.0.080522 >> Original-recipient: rfc822;rick.weisner at sun.com >> >> Looks OK: >> >> god at irt-smtp-02:~ 7:22am 60 # !nm >> nm /bin/python | egrep malloc >> [3597] | 134599012| 0|FUNC |GLOB |0 |UNDEF |malloc >> >> >> >> On 7/2/08 6:11 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: >> >> > Fletcher, >> > First confirm that malloc is in your binary. >> > >> > arwen:nm a.out | grep malloc >> > [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc >> > >> > Then key on any malloc. >> > Something like: >> > pid$target::malloc:return, >> > pid$target::memalign:return, >> > pid$target::realloc:return, >> > pid$target::valloc:return >> > >> > rick >> >> -- >> Fletcher Cocquyt >> Senior Systems Administrator >> Information Resources and Technology (IRT) >> Stanford University School of Medicine >> >> Email: fcocquyt at stanford.edu >> Phone: (650) 724-7485 >> >> > > -- > > Rickey C. Weisner > Software Development and Performance Specialist > Sun Microsystems, INC > cell phone: 615-308-1147 > email: rick.weisner at sun.com > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >
Yes, removing libc.so.1 Output ! - now to analyze the output I will bring in my parallel mailman-users'' thread folks :) - having the stack trace is awesome by itself... thanks 0 42246 realloc:return Ptr=0x824c268 Oldptr=0x0 Size=16 libc.so.1`realloc+0x33a python`addcleanup+0x45 python`convertsimple+0x145d python`vgetargs1+0x259 python`_PyArg_ParseTuple_SizeT+0x1d python`posix_listdir+0x55 python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x824c268 0 42244 lmalloc:return Ptr=0xcf890300 Size=16 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42244 lmalloc:return Ptr=0xcf894000 Size=8192 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x86d78f0 ^C 0 42246 realloc:return Ptr=0x824c268 Oldptr=0x0 Size=16 libc.so.1`realloc+0x33a python`addcleanup+0x45 python`convertsimple+0x145d python`vgetargs1+0x259 python`_PyArg_ParseTuple_SizeT+0x1d python`posix_listdir+0x55 python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x824c268 0 42244 lmalloc:return Ptr=0xcf890300 Size=16 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42244 lmalloc:return Ptr=0xcf894000 Size=8192 libc.so.1`lmalloc+0x143 libc.so.1`opendir+0x3e python`posix_listdir+0x6d python`PyEval_EvalFrameEx+0x59ff python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalFrameEx+0x49ff python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalFrameEx+0x6133 python`PyEval_EvalCodeEx+0x57f python`PyEval_EvalCode+0x22 python`PyRun_FileExFlags+0xaf python`PyRun_SimpleFileExFlags+0x156 python`Py_Main+0xa6b python`main+0x17 python`_start+0x80 0 42249 free:entry Ptr=0x86d78f0 On 7/2/08 7:46 AM, "rickey c weisner" <rick.weisner at sun.com> wrote:> Fletcher, > This looks suspicious. Perhaps your malloc is not in libc ? >> [690] | 0| 0|FILE |LOCL |0 |ABS |obmalloc.c > remove the libc.so.1 from your probe description. > rick > > On Wed, Jul 02, 2008 at 07:23:49AM -0700, Fletcher Cocquyt wrote: >> Date: Wed, 02 Jul 2008 07:23:49 -0700 >> From: Fletcher Cocquyt <fcocquyt at stanford.edu> >> Subject: Re: [dtrace-discuss] Memory leak scripts >> In-reply-to: <20080702081148.A1624 at arwen.sfbay.sun.com> >> To: rickey c weisner <Rick.Weisner at Sun.COM> >> Cc: dtrace-discuss at opensolaris.org >> Thread-topic: [dtrace-discuss] Memory leak scripts >> Thread-index: AcjcTz6M0DPfSSwvAEuYeh4RTBFiig=>> X-PMX-Version: 5.4.1.325704 >> X-Brightmail-Tracker: AAAAAA=>> X-Antispam: No, score=0.0/5.0, scanned in 0.102sec at (localhost [127.0.0.1]) >> by smf-spamd v1.3.1 - http://smfs.sf.net/ >> User-Agent: Microsoft-Entourage/12.11.0.080522 >> Original-recipient: rfc822;rick.weisner at sun.com >> >> Looks OK: >> >> god at irt-smtp-02:~ 7:22am 60 # !nm >> nm /bin/python | egrep malloc >> [3597] | 134599012| 0|FUNC |GLOB |0 |UNDEF |malloc >> >> >> >> On 7/2/08 6:11 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: >> >>> Fletcher, >>> First confirm that malloc is in your binary. >>> >>> arwen:nm a.out | grep malloc >>> [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc >>> >>> Then key on any malloc. >>> Something like: >>> pid$target::malloc:return, >>> pid$target::memalign:return, >>> pid$target::realloc:return, >>> pid$target::valloc:return >>> >>> rick >> >> -- >> Fletcher Cocquyt >> Senior Systems Administrator >> Information Resources and Technology (IRT) >> Stanford University School of Medicine >> >> Email: fcocquyt at stanford.edu >> Phone: (650) 724-7485 >> >>-- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt at stanford.edu Phone: (650) 724-7485
Ok, maybe this is significant in the context of explaining why my python (mailman) processes seem to grow abnormally? If the libc malloc is not being called, why and is that an important issue? god at irt-smtp-02:~ 2:17pm 54 # ldd /bin/python libresolv.so.2 => /lib/libresolv.so.2 libsocket.so.1 => /lib/libsocket.so.1 libnsl.so.1 => /lib/libnsl.so.1 librt.so.1 => /lib/librt.so.1 libdl.so.1 => /lib/libdl.so.1 libm.so.2 => /lib/libm.so.2 libc.so.1 => /lib/libc.so.1 libmp.so.2 => /lib/libmp.so.2 libmd.so.1 => /lib/libmd.so.1 libscf.so.1 => /lib/libscf.so.1 libaio.so.1 => /lib/libaio.so.1 libdoor.so.1 => /lib/libdoor.so.1 libuutil.so.1 => /lib/libuutil.so.1 libgen.so.1 => /lib/libgen.so.1 This is Python 2.5.2, built with no configure options besides --prefix on Solaris 10. Config.log excerpt: configure:14433: checking for --with-pymalloc configure:14453: result: yes thanks On 7/2/08 7:46 AM, "rickey c weisner" <rick.weisner at sun.com> wrote:> Fletcher, > This looks suspicious. Perhaps your malloc is not in libc ? >> [690] | 0| 0|FILE |LOCL |0 |ABS |obmalloc.c > remove the libc.so.1 from your probe description. > rick > > On Wed, Jul 02, 2008 at 07:23:49AM -0700, Fletcher Cocquyt wrote: >> Date: Wed, 02 Jul 2008 07:23:49 -0700 >> From: Fletcher Cocquyt <fcocquyt at stanford.edu> >> Subject: Re: [dtrace-discuss] Memory leak scripts >> In-reply-to: <20080702081148.A1624 at arwen.sfbay.sun.com> >> To: rickey c weisner <Rick.Weisner at Sun.COM> >> Cc: dtrace-discuss at opensolaris.org >> Thread-topic: [dtrace-discuss] Memory leak scripts >> Thread-index: AcjcTz6M0DPfSSwvAEuYeh4RTBFiig=>> X-PMX-Version: 5.4.1.325704 >> X-Brightmail-Tracker: AAAAAA=>> X-Antispam: No, score=0.0/5.0, scanned in 0.102sec at (localhost [127.0.0.1]) >> by smf-spamd v1.3.1 - http://smfs.sf.net/ >> User-Agent: Microsoft-Entourage/12.11.0.080522 >> Original-recipient: rfc822;rick.weisner at sun.com >> >> Looks OK: >> >> god at irt-smtp-02:~ 7:22am 60 # !nm >> nm /bin/python | egrep malloc >> [3597] | 134599012| 0|FUNC |GLOB |0 |UNDEF |malloc >> >> >> >> On 7/2/08 6:11 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: >> >>> Fletcher, >>> First confirm that malloc is in your binary. >>> >>> arwen:nm a.out | grep malloc >>> [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc >>> >>> Then key on any malloc. >>> Something like: >>> pid$target::malloc:return, >>> pid$target::memalign:return, >>> pid$target::realloc:return, >>> pid$target::valloc:return >>> >>> rick >> >> -- >> Fletcher Cocquyt >> Senior Systems Administrator >> Information Resources and Technology (IRT) >> Stanford University School of Medicine >> >> Email: fcocquyt at stanford.edu >> Phone: (650) 724-7485 >> >>-- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt at stanford.edu Phone: (650) 724-7485
Fletcher, libc malloc being called or not only had to do with the naming of your probe. Looking at your configure options : --with-pymalloc This imples to me that python has his own malloc. I do not recall, but why do you think you have a memory leak ? Just because the process grows over time and does not diminish in size does not necessarily mean a memory leak. How do you measure the size of your process and are you examining the virtual size or the RSS ? The virtual size only grows upward, but I would expect it to eventually stabilize. RSS will go up and down over time. I would be more concerned with RSS than virtual size except for the possibility of exceeding a 4 GB address space for 32 bit applications. pmap -xs would be interesting. rick On Wed, Jul 02, 2008 at 02:27:31PM -0700, Fletcher Cocquyt wrote:> Date: Wed, 02 Jul 2008 14:27:31 -0700 > From: Fletcher Cocquyt <fcocquyt at stanford.edu> > Subject: Re: [dtrace-discuss] Memory leak scripts - analysis > In-reply-to: <20080702094622.B1624 at arwen.sfbay.sun.com> > To: rickey c weisner <Rick.Weisner at Sun.COM> > Cc: dtrace-discuss at opensolaris.org > Thread-topic: [dtrace-discuss] Memory leak scripts - analysis > Thread-index: Acjcim8+knqA/bONQ0GtOzL8hEW0mg=> X-PMX-Version: 5.4.1.325704 > X-Brightmail-Tracker: AAAAAA=> X-Antispam: No, score=0.0/5.0, scanned in 0.085sec at (localhost [127.0.0.1]) > by smf-spamd v1.3.1 - http://smfs.sf.net/ > User-Agent: Microsoft-Entourage/12.11.0.080522 > Original-recipient: rfc822;rick.weisner at sun.com > > Ok, maybe this is significant in the context of explaining why my python > (mailman) processes seem to grow abnormally? > If the libc malloc is not being called, why and is that an important issue? > > > god at irt-smtp-02:~ 2:17pm 54 # ldd /bin/python > libresolv.so.2 => /lib/libresolv.so.2 > libsocket.so.1 => /lib/libsocket.so.1 > libnsl.so.1 => /lib/libnsl.so.1 > librt.so.1 => /lib/librt.so.1 > libdl.so.1 => /lib/libdl.so.1 > libm.so.2 => /lib/libm.so.2 > libc.so.1 => /lib/libc.so.1 > libmp.so.2 => /lib/libmp.so.2 > libmd.so.1 => /lib/libmd.so.1 > libscf.so.1 => /lib/libscf.so.1 > libaio.so.1 => /lib/libaio.so.1 > libdoor.so.1 => /lib/libdoor.so.1 > libuutil.so.1 => /lib/libuutil.so.1 > libgen.so.1 => /lib/libgen.so.1 > > This is Python 2.5.2, built with no configure options besides --prefix on > Solaris 10. > > Config.log excerpt: > configure:14433: checking for --with-pymalloc > configure:14453: result: yes > > thanks > > On 7/2/08 7:46 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: > > > Fletcher, > > This looks suspicious. Perhaps your malloc is not in libc ? > >> [690] | 0| 0|FILE |LOCL |0 |ABS |obmalloc.c > > remove the libc.so.1 from your probe description. > > rick > > > > On Wed, Jul 02, 2008 at 07:23:49AM -0700, Fletcher Cocquyt wrote: > >> Date: Wed, 02 Jul 2008 07:23:49 -0700 > >> From: Fletcher Cocquyt <fcocquyt at stanford.edu> > >> Subject: Re: [dtrace-discuss] Memory leak scripts > >> In-reply-to: <20080702081148.A1624 at arwen.sfbay.sun.com> > >> To: rickey c weisner <Rick.Weisner at Sun.COM> > >> Cc: dtrace-discuss at opensolaris.org > >> Thread-topic: [dtrace-discuss] Memory leak scripts > >> Thread-index: AcjcTz6M0DPfSSwvAEuYeh4RTBFiig=> >> X-PMX-Version: 5.4.1.325704 > >> X-Brightmail-Tracker: AAAAAA=> >> X-Antispam: No, score=0.0/5.0, scanned in 0.102sec at (localhost [127.0.0.1]) > >> by smf-spamd v1.3.1 - http://smfs.sf.net/ > >> User-Agent: Microsoft-Entourage/12.11.0.080522 > >> Original-recipient: rfc822;rick.weisner at sun.com > >> > >> Looks OK: > >> > >> god at irt-smtp-02:~ 7:22am 60 # !nm > >> nm /bin/python | egrep malloc > >> [3597] | 134599012| 0|FUNC |GLOB |0 |UNDEF |malloc > >> > >> > >> > >> On 7/2/08 6:11 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: > >> > >>> Fletcher, > >>> First confirm that malloc is in your binary. > >>> > >>> arwen:nm a.out | grep malloc > >>> [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc > >>> > >>> Then key on any malloc. > >>> Something like: > >>> pid$target::malloc:return, > >>> pid$target::memalign:return, > >>> pid$target::realloc:return, > >>> pid$target::valloc:return > >>> > >>> rick > >> > >> -- > >> Fletcher Cocquyt > >> Senior Systems Administrator > >> Information Resources and Technology (IRT) > >> Stanford University School of Medicine > >> > >> Email: fcocquyt at stanford.edu > >> Phone: (650) 724-7485 > >> > >> > > -- > Fletcher Cocquyt > Senior Systems Administrator > Information Resources and Technology (IRT) > Stanford University School of Medicine > > Email: fcocquyt at stanford.edu > Phone: (650) 724-7485 > >-- Rickey C. Weisner Software Development and Performance Specialist Sun Microsystems, INC cell phone: 615-308-1147 email: rick.weisner at sun.com
My original posts were to the dtrace list and the mailman list "Python process size grows 30x in 8 hours (memory leak?) " and I did not assume it would be a memory leak, but suspected it could be. I did a pmap early on - posted to the mailman group - it shows the heap is the portion growing - but how to get visibility into the heap? As it turns out, getting a python stack trace out of memleak.d was very useful and increased my knowledge of Dtrace a whole lot in the process. If this is not a memory leak, its at least pointed out a deficiency in my mailman (I am not ruling out some uniqueness about my installation) (unbounded growth of runner memory size) I expect the code maintainers may adopt an optional process reaper (autorestart) model apache does with its httpd processes, where they are recycled automatically before they have a chance to grow too much. That is my current manual workaround (restarting twice a day before memory is exhausted and it swaps to death) thanks god at irt-smtp-02:~ 2:56pm 72 # pmap -xs 28001 28001: /bin/python /opt/mailman-2.1.9/bin/qrunner --runner=BounceRunner:0:1 - Address Kbytes RSS Anon Locked Pgsz Mode Mapped File 08038000 64 64 64 - 4K rwx-- [ stack ] 08050000 128 128 - - 4K r-x-- python 08070000 8 8 - - - r-x-- python 08072000 188 188 - - 4K r-x-- python 080A1000 4 - - - - r-x-- python 080A2000 4 4 - - 4K r-x-- python 080A3000 12 8 - - - r-x-- python 080A6000 40 40 - - 4K r-x-- python 080B0000 4 4 - - - r-x-- python 080B1000 32 32 - - 4K r-x-- python 080B9000 24 24 - - - r-x-- python 080BF000 292 292 - - 4K r-x-- python 08108000 4 4 - - - r-x-- python 08109000 16 16 - - 4K r-x-- python 0810D000 4 - - - - r-x-- python 0810E000 28 28 - - 4K r-x-- python 08115000 4 - - - - r-x-- python 08116000 8 8 - - 4K r-x-- python 08118000 16 4 - - - r-x-- python 0811C000 64 64 - - 4K r-x-- python 0812C000 4 4 - - - r-x-- python 0812D000 56 56 - - 4K r-x-- python 0814A000 36 36 36 - 4K rwx-- python 08153000 4 - - - - rwx-- python 08154000 12 12 12 - 4K rwx-- python 08157000 4 - - - - rwx-- python 08158000 4 4 4 - 4K rwx-- python 08159000 20 8 - - - rwx-- python 0815E000 4 4 - - 4K rwx-- python 0815F000 8 4 - - - rwx-- python 08161000 28 28 24 - 4K rwx-- python 08168000 8 - - - - rwx-- python 0816A000 44 44 40 - 4K rwx-- python 08175000 22732 22732 22732 - 4K rwx-- [ heap ] CF1D0000 4 4 - - 4K r-x-- libcrypt_d.so.1 CF1D1000 8 8 - - - r-x-- libcrypt_d.so.1 CF1D3000 4 4 - - 4K r-x-- libcrypt_d.so.1 CF1D4000 12 - - - - r-x-- libcrypt_d.so.1 CF1E7000 4 4 4 - 4K rw--- libcrypt_d.so.1 CF1F0000 4 4 - - 4K r-x-- crypt.so CF200000 4 4 4 - 4K rwx-- crypt.so CF210000 4 4 4 - 4K rwx-- [ anon ] CF220000 16 16 - - 4K r-x-- datetime.so CF224000 4 - - - - r-x-- datetime.so CF225000 8 8 - - 4K r-x-- datetime.so CF227000 12 12 - - - r-x-- datetime.so CF22A000 16 16 - - 4K r-x-- datetime.so CF23D000 12 12 12 - 4K rwx-- datetime.so CF250000 4 4 - - 4K r-x-- _weakref.so CF260000 4 4 4 - 4K rwx-- _weakref.so CF270000 4 4 - - 4K r-x-- _sha512.so CF271000 64 12 - - - r-x-- _sha512.so CF281000 4 4 - - 4K r-x-- _sha512.so CF291000 8 8 8 - 4K rwx-- _sha512.so CF2A0000 4 4 4 - 4K rwx-- [ anon ] CF2B0000 4 4 - - 4K r-x-- _sha256.so CF2B1000 8 8 - - - r-x-- _sha256.so CF2B3000 4 4 - - 4K r-x-- _sha256.so CF2C3000 8 8 8 - 4K rwx-- _sha256.so CF2D0000 8 8 - - 4K r-x-- _hashlib.so CF2E1000 8 8 8 - 4K rwx-- _hashlib.so CF2F0000 12 12 - - 4K r-x-- fcntl.so CF302000 8 8 8 - 4K rwx-- fcntl.so CF310000 4 4 4 - 4K rwx-- [ anon ] CF320000 8 8 - - 4K r-x-- _codecs_kr.so CF322000 96 8 - - - r-x-- _codecs_kr.so CF33A000 4 4 - - 4K r-x-- _codecs_kr.so CF34A000 12 12 12 - 4K rwx-- _codecs_kr.so CF350000 16 16 - - 4K r-x-- _codecs_iso2022.so CF363000 4 4 4 - 4K rwx-- _codecs_iso2022.so CF370000 12 12 - - 4K r-x-- _multibytecodec.so CF373000 4 4 - - - r-x-- _multibytecodec.so CF374000 4 4 - - 4K r-x-- _multibytecodec.so CF384000 8 8 8 - 4K rwx-- _multibytecodec.so CF390000 4 4 4 - 4K rwx-- [ anon ] CF3A0000 8 8 - - 4K r-x-- _codecs_jp.so CF3A2000 4 4 - - - r-x-- _codecs_jp.so CF3A3000 8 8 - - 4K r-x-- _codecs_jp.so CF3A5000 168 4 - - - r-x-- _codecs_jp.so CF3CF000 4 4 - - 4K r-x-- _codecs_jp.so CF3DF000 32 32 32 - 4K rwx-- _codecs_jp.so CF3F0000 4 4 - - 4K r-x-- strop.so CF3F1000 8 8 - - - r-x-- strop.so CF3F3000 4 4 - - 4K r-x-- strop.so CF403000 8 8 8 - 4K rwx-- strop.so CF410000 8 8 - - 4K r-x-- _random.so CF421000 8 8 8 - 4K rwx-- _random.so CF430000 4 4 4 - 4K rwx-- [ anon ] CF440000 12 12 - - 4K r-x-- math.so CF452000 4 4 4 - 4K rwx-- math.so CF460000 4 4 - - 4K r-x-- libmp.so.2 CF461000 4 4 - - - r-x-- libmp.so.2 CF462000 4 4 - - 4K r-x-- libmp.so.2 CF473000 4 4 4 - 4K rw--- libmp.so.2 CF480000 4 4 4 - 4K rwx-- [ anon ] CF490000 20 20 - - 4K r-x-- libgen.so.1 CF495000 4 4 - - - r-x-- libgen.so.1 CF4A6000 4 4 4 - 4K rw--- libgen.so.1 CF4B0000 24 24 - - 4K r-x-- libuutil.so.1 CF4C6000 4 4 4 - 4K rw--- libuutil.so.1 CF4D0000 4 4 - - 4K r-x-- libdoor.so.1 CF4E1000 4 4 4 - 4K rw--- libdoor.so.1 CF4F0000 32 32 - - 4K r-x-- libscf.so.1 CF4F8000 40 40 - - - r-x-- libscf.so.1 CF502000 16 16 - - 4K r-x-- libscf.so.1 CF516000 4 4 4 - 4K rw--- libscf.so.1 CF520000 4 4 4 - 4K rwx-- [ anon ] CF530000 32 32 - - 4K r-x-- libcrypto_extra.so.0.9.7 CF538000 8 4 - - - r-x-- libcrypto_extra.so.0.9.7 CF54A000 4 4 4 - 4K rw--- libcrypto_extra.so.0.9.7 CF550000 16 16 - - 4K r-x-- libssl_extra.so.0.9.7 CF554000 12 12 - - - r-x-- libssl_extra.so.0.9.7 CF557000 4 4 - - 4K r-x-- libssl_extra.so.0.9.7 CF558000 4 4 - - - r-x-- libssl_extra.so.0.9.7 CF569000 4 4 4 - 4K rw--- libssl_extra.so.0.9.7 CF570000 204 204 - - 4K r-x-- libcrypto.so.0.9.7 CF5A3000 72 72 - - - r-x-- libcrypto.so.0.9.7 CF5B5000 8 8 - - 4K r-x-- libcrypto.so.0.9.7 CF5B7000 8 8 - - - r-x-- libcrypto.so.0.9.7 CF5B9000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF5BA000 56 56 - - - r-x-- libcrypto.so.0.9.7 CF5C8000 12 12 - - 4K r-x-- libcrypto.so.0.9.7 CF5CB000 4 4 - - - r-x-- libcrypto.so.0.9.7 CF5CC000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF5CD000 44 44 - - - r-x-- libcrypto.so.0.9.7 CF5D8000 20 20 - - 4K r-x-- libcrypto.so.0.9.7 CF5DD000 4 4 - - - r-x-- libcrypto.so.0.9.7 CF5DE000 20 20 - - 4K r-x-- libcrypto.so.0.9.7 CF5E3000 8 8 - - - r-x-- libcrypto.so.0.9.7 CF5E5000 8 8 - - 4K r-x-- libcrypto.so.0.9.7 CF5E7000 24 24 - - - r-x-- libcrypto.so.0.9.7 CF5ED000 24 24 - - 4K r-x-- libcrypto.so.0.9.7 CF5F3000 24 24 - - - r-x-- libcrypto.so.0.9.7 CF5F9000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF5FA000 8 8 - - - r-x-- libcrypto.so.0.9.7 CF5FC000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF5FD000 4 4 - - - r-x-- libcrypto.so.0.9.7 CF5FE000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF5FF000 16 16 - - - r-x-- libcrypto.so.0.9.7 CF603000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF604000 20 20 - - - r-x-- libcrypto.so.0.9.7 CF609000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF60A000 8 8 - - - r-x-- libcrypto.so.0.9.7 CF60C000 20 20 - - 4K r-x-- libcrypto.so.0.9.7 CF611000 16 12 - - - r-x-- libcrypto.so.0.9.7 CF615000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF616000 24 24 - - - r-x-- libcrypto.so.0.9.7 CF61C000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF61D000 24 24 - - - r-x-- libcrypto.so.0.9.7 CF623000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF624000 28 24 - - - r-x-- libcrypto.so.0.9.7 CF62B000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF62C000 56 56 - - - r-x-- libcrypto.so.0.9.7 CF63A000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF63B000 24 24 - - - r-x-- libcrypto.so.0.9.7 CF641000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF642000 8 8 - - - r-x-- libcrypto.so.0.9.7 CF644000 4 4 - - 4K r-x-- libcrypto.so.0.9.7 CF645000 4 4 - - - r-x-- libcrypto.so.0.9.7 CF646000 8 8 - - 4K r-x-- libcrypto.so.0.9.7 CF648000 64 64 - - - r-x-- libcrypto.so.0.9.7 CF668000 68 68 68 - 4K rw--- libcrypto.so.0.9.7 CF679000 12 12 - - - rw--- libcrypto.so.0.9.7 CF67C000 4 4 4 - 4K rw--- libcrypto.so.0.9.7 CF67D000 8 8 8 - 4K rw--- libcrypto.so.0.9.7 CF690000 36 36 - - 4K r-x-- libssl.so.0.9.7 CF699000 80 68 - - - r-x-- libssl.so.0.9.7 CF6AD000 4 4 - - 4K r-x-- libssl.so.0.9.7 CF6AE000 36 24 - - - r-x-- libssl.so.0.9.7 CF6B7000 4 4 - - 4K r-x-- libssl.so.0.9.7 CF6B8000 40 20 - - - r-x-- libssl.so.0.9.7 CF6D2000 12 12 12 - 4K rw--- libssl.so.0.9.7 CF6E0000 12 12 - - 4K r-x-- _ssl.so CF6F2000 8 8 8 - 4K rwx-- _ssl.so CF700000 4 4 4 - 4K rwx-- [ anon ] CF710000 32 32 - - 4K r-x-- _socket.so CF727000 16 16 12 - 4K rwx-- _socket.so CF730000 12 12 - - 4K r-x-- cStringIO.so CF742000 8 8 8 - 4K rwx-- cStringIO.so CF750000 48 48 - - 4K r-x-- cPickle.so CF75C000 4 - - - - r-x-- cPickle.so CF75D000 16 16 - - 4K r-x-- cPickle.so CF770000 4 4 4 - 4K rwx-- cPickle.so CF780000 4 4 4 - 4K rwx-- [ anon ] CF790000 12 12 - - 4K r-x-- binascii.so CF7A2000 8 8 8 - 4K rwx-- binascii.so CF7B0000 8 8 - - 4K r-x-- _struct.so CF7B2000 8 8 - - - r-x-- _struct.so CF7B4000 4 4 - - 4K r-x-- _struct.so CF7C4000 8 8 8 - 4K rwx-- _struct.so CF7D0000 20 20 - - 4K r-x-- operator.so CF7E4000 8 8 8 - 4K rwx-- operator.so CF800000 4 4 4 - 4K rwx-- [ anon ] CF810000 12 12 - - 4K r-x-- _locale.so CF822000 8 8 8 - 4K rwx-- _locale.so CF830000 8 8 - - 4K r-x-- libgcc_s.so.1 CF832000 8 8 - - - r-x-- libgcc_s.so.1 CF834000 8 8 - - 4K r-x-- libgcc_s.so.1 CF836000 4 4 - - - r-x-- libgcc_s.so.1 CF846000 4 4 4 - 4K rwx-- libgcc_s.so.1 CF850000 4 4 4 - 4K rwx-- [ anon ] CF860000 12 12 - - 4K r-x-- time.so CF872000 8 8 8 - 4K rwx-- time.so CF880000 4 4 - - 4K r-x-- libintl.so.1 CF890000 12 12 12 - 4K rwx-- [ anon ] CF893000 4 - - - - rwx-- [ anon ] CF894000 8 8 8 - 4K rwx-- [ anon ] CF8A0000 12 12 - - 4K r-x-- libmd.so.1 CF8A3000 40 - - - - r-x-- libmd.so.1 CF8AD000 4 4 - - 4K r-x-- libmd.so.1 CF8BE000 4 4 4 - 4K rw--- libmd.so.1 CF8D0000 4 4 4 - 4K rwx-- [ anon ] CF8E0000 8 8 - - 4K r-x-- libaio.so.1 CF8E2000 16 8 - - - r-x-- libaio.so.1 CF8E6000 4 4 - - 4K r-x-- libaio.so.1 CF8F7000 4 4 4 - 4K rw--- libaio.so.1 CF8F8000 4 - - - - rw--- libaio.so.1 CF900000 132 132 - - 4K r-x-- libc.so.1 CF921000 4 4 - - - r-x-- libc.so.1 CF922000 36 36 - - 4K r-x-- libc.so.1 CF92B000 12 12 - - - r-x-- libc.so.1 CF92E000 44 44 - - 4K r-x-- libc.so.1 CF939000 28 28 - - - r-x-- libc.so.1 CF940000 64 64 - - 4K r-x-- libc.so.1 CF950000 4 4 - - - r-x-- libc.so.1 CF951000 40 40 - - 4K r-x-- libc.so.1 CF95B000 24 24 - - - r-x-- libc.so.1 CF961000 4 4 - - 4K r-x-- libc.so.1 CF962000 40 40 - - - r-x-- libc.so.1 CF96C000 20 20 - - 4K r-x-- libc.so.1 CF971000 20 20 - - - r-x-- libc.so.1 CF976000 20 20 - - 4K r-x-- libc.so.1 CF97B000 8 8 - - - r-x-- libc.so.1 CF97D000 20 20 - - 4K r-x-- libc.so.1 CF982000 24 20 - - - r-x-- libc.so.1 CF988000 32 32 - - 4K r-x-- libc.so.1 CF990000 4 4 - - - r-x-- libc.so.1 CF991000 100 100 - - 4K r-x-- libc.so.1 CF9AA000 4 4 - - - r-x-- libc.so.1 CF9AB000 4 4 - - 4K r-x-- libc.so.1 CF9AC000 4 4 - - - r-x-- libc.so.1 CF9AD000 44 44 - - 4K r-x-- libc.so.1 CF9B8000 8 8 - - - r-x-- libc.so.1 CF9BA000 12 12 - - 4K r-x-- libc.so.1 CF9BD000 8 8 - - - r-x-- libc.so.1 CF9CF000 28 28 28 - 4K rw--- libc.so.1 CF9D6000 8 8 8 - 4K rw--- libc.so.1 CF9E0000 28 28 - - 4K r-x-- libm.so.2 CF9E7000 32 32 - - - r-x-- libm.so.2 CF9EF000 12 12 - - 4K r-x-- libm.so.2 CF9F2000 40 24 - - - r-x-- libm.so.2 CF9FC000 8 8 - - 4K r-x-- libm.so.2 CF9FE000 92 92 - - - r-x-- libm.so.2 CFA15000 4 4 - - 4K r-x-- libm.so.2 CFA16000 52 24 - - - r-x-- libm.so.2 CFA32000 4 4 4 - 4K rwx-- libm.so.2 CFA33000 8 4 - - - rwx-- libm.so.2 CFA35000 4 4 4 - 4K rwx-- libm.so.2 CFA40000 4 4 - - 4K r-x-- libdl.so.1 CFA51000 4 4 4 - 4K rw--- libdl.so.1 CFA60000 4 4 4 - 4K rwx-- [ anon ] CFA70000 16 16 - - 4K r-x-- librt.so.1 CFA74000 4 - - - - r-x-- librt.so.1 CFA75000 4 4 - - 4K r-x-- librt.so.1 CFA86000 4 4 4 - 4K rw--- librt.so.1 CFA90000 72 72 - - 4K r-x-- libnsl.so.1 CFAA2000 160 160 - - - r-x-- libnsl.so.1 CFACA000 32 32 - - 4K r-x-- libnsl.so.1 CFAD2000 196 136 - - - r-x-- libnsl.so.1 CFB03000 8 8 - - 4K r-x-- libnsl.so.1 CFB05000 48 32 - - - r-x-- libnsl.so.1 CFB21000 20 20 20 - 4K rw--- libnsl.so.1 CFB26000 12 - - - - rw--- libnsl.so.1 CFB29000 12 12 12 - 4K rw--- libnsl.so.1 CFB2C000 4 - - - - rw--- libnsl.so.1 CFB2D000 4 4 4 - 4K rw--- libnsl.so.1 CFB30000 24 24 - - 4K r-x-- libsocket.so.1 CFB36000 16 16 - - - r-x-- libsocket.so.1 CFB3A000 4 4 - - 4K r-x-- libsocket.so.1 CFB4B000 4 4 4 - 4K rw--- libsocket.so.1 CFB50000 28 28 - - 4K r-x-- libresolv.so.2 CFB57000 160 132 - - - r-x-- libresolv.so.2 CFB7F000 12 12 - - 4K r-x-- libresolv.so.2 CFB82000 16 8 - - - r-x-- libresolv.so.2 CFB96000 8 8 8 - 4K rw--- libresolv.so.2 CFBA0000 4 4 4 - 4K rwx-- [ anon ] CFBB0000 4 4 4 - 4K rwx-- [ anon ] CFBC0000 4 4 - - 4K r--s- dev:61,0 ino:47763 CFBC4000 156 156 - - 4K r-x-- ld.so.1 CFBFB000 4 4 4 - 4K rwx-- ld.so.1 CFBFC000 8 8 8 - 4K rwx-- ld.so.1 -------- ------- ------- ------- ------- total Kb 28724 28044 23440 - On 7/2/08 2:58 PM, "rickey c weisner" <rick.weisner at sun.com> wrote:> Fletcher, > libc malloc being called or not only had to do with the > naming of your probe. > Looking at your configure options : > --with-pymalloc > This imples to me that python has his own malloc. > > I do not recall, but why do you think you have a memory leak ? > > Just because the process grows over time and does not diminish > in size does not necessarily mean a memory leak. How do you measure > the size of your process and are you examining the virtual size > or the RSS ? The virtual size only grows upward, but I would expect > it to eventually stabilize. RSS will go up and > down over time. I would be more concerned with RSS than virtual size > except for the possibility of exceeding a 4 GB address space for 32 bit > applications. pmap -xs would be interesting. > > rick > > On Wed, Jul 02, 2008 at 02:27:31PM -0700, Fletcher Cocquyt wrote: >> Date: Wed, 02 Jul 2008 14:27:31 -0700 >> From: Fletcher Cocquyt <fcocquyt at stanford.edu> >> Subject: Re: [dtrace-discuss] Memory leak scripts - analysis >> In-reply-to: <20080702094622.B1624 at arwen.sfbay.sun.com> >> To: rickey c weisner <Rick.Weisner at Sun.COM> >> Cc: dtrace-discuss at opensolaris.org >> Thread-topic: [dtrace-discuss] Memory leak scripts - analysis >> Thread-index: Acjcim8+knqA/bONQ0GtOzL8hEW0mg=>> X-PMX-Version: 5.4.1.325704 >> X-Brightmail-Tracker: AAAAAA=>> X-Antispam: No, score=0.0/5.0, scanned in 0.085sec at (localhost [127.0.0.1]) >> by smf-spamd v1.3.1 - http://smfs.sf.net/ >> User-Agent: Microsoft-Entourage/12.11.0.080522 >> Original-recipient: rfc822;rick.weisner at sun.com >> >> Ok, maybe this is significant in the context of explaining why my python >> (mailman) processes seem to grow abnormally? >> If the libc malloc is not being called, why and is that an important issue? >> >> >> god at irt-smtp-02:~ 2:17pm 54 # ldd /bin/python >> libresolv.so.2 => /lib/libresolv.so.2 >> libsocket.so.1 => /lib/libsocket.so.1 >> libnsl.so.1 => /lib/libnsl.so.1 >> librt.so.1 => /lib/librt.so.1 >> libdl.so.1 => /lib/libdl.so.1 >> libm.so.2 => /lib/libm.so.2 >> libc.so.1 => /lib/libc.so.1 >> libmp.so.2 => /lib/libmp.so.2 >> libmd.so.1 => /lib/libmd.so.1 >> libscf.so.1 => /lib/libscf.so.1 >> libaio.so.1 => /lib/libaio.so.1 >> libdoor.so.1 => /lib/libdoor.so.1 >> libuutil.so.1 => /lib/libuutil.so.1 >> libgen.so.1 => /lib/libgen.so.1 >> >> This is Python 2.5.2, built with no configure options besides --prefix on >> Solaris 10. >> >> Config.log excerpt: >> configure:14433: checking for --with-pymalloc >> configure:14453: result: yes >> >> thanks >> >> On 7/2/08 7:46 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: >> >>> Fletcher, >>> This looks suspicious. Perhaps your malloc is not in libc ? >>>> [690] | 0| 0|FILE |LOCL |0 |ABS |obmalloc.c >>> remove the libc.so.1 from your probe description. >>> rick >>> >>> On Wed, Jul 02, 2008 at 07:23:49AM -0700, Fletcher Cocquyt wrote: >>>> Date: Wed, 02 Jul 2008 07:23:49 -0700 >>>> From: Fletcher Cocquyt <fcocquyt at stanford.edu> >>>> Subject: Re: [dtrace-discuss] Memory leak scripts >>>> In-reply-to: <20080702081148.A1624 at arwen.sfbay.sun.com> >>>> To: rickey c weisner <Rick.Weisner at Sun.COM> >>>> Cc: dtrace-discuss at opensolaris.org >>>> Thread-topic: [dtrace-discuss] Memory leak scripts >>>> Thread-index: AcjcTz6M0DPfSSwvAEuYeh4RTBFiig=>>>> X-PMX-Version: 5.4.1.325704 >>>> X-Brightmail-Tracker: AAAAAA=>>>> X-Antispam: No, score=0.0/5.0, scanned in 0.102sec at (localhost >>>> [127.0.0.1]) >>>> by smf-spamd v1.3.1 - http://smfs.sf.net/ >>>> User-Agent: Microsoft-Entourage/12.11.0.080522 >>>> Original-recipient: rfc822;rick.weisner at sun.com >>>> >>>> Looks OK: >>>> >>>> god at irt-smtp-02:~ 7:22am 60 # !nm >>>> nm /bin/python | egrep malloc >>>> [3597] | 134599012| 0|FUNC |GLOB |0 |UNDEF |malloc >>>> >>>> >>>> >>>> On 7/2/08 6:11 AM, "rickey c weisner" <rick.weisner at sun.com> wrote: >>>> >>>>> Fletcher, >>>>> First confirm that malloc is in your binary. >>>>> >>>>> arwen:nm a.out | grep malloc >>>>> [70] | 134547228| 0|FUNC |GLOB |0 |UNDEF |malloc >>>>> >>>>> Then key on any malloc. >>>>> Something like: >>>>> pid$target::malloc:return, >>>>> pid$target::memalign:return, >>>>> pid$target::realloc:return, >>>>> pid$target::valloc:return >>>>> >>>>> rick >>>> >>>> -- >>>> Fletcher Cocquyt >>>> Senior Systems Administrator >>>> Information Resources and Technology (IRT) >>>> Stanford University School of Medicine >>>> >>>> Email: fcocquyt at stanford.edu >>>> Phone: (650) 724-7485 >>>> >>>> >> >> -- >> Fletcher Cocquyt >> Senior Systems Administrator >> Information Resources and Technology (IRT) >> Stanford University School of Medicine >> >> Email: fcocquyt at stanford.edu >> Phone: (650) 724-7485 >> >>-- Fletcher Cocquyt Senior Systems Administrator Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt at stanford.edu Phone: (650) 724-7485
Fletcher, If the page scanner is running, and or you have anonymous paging (vmstat -p) then it is RSS that is part of your problem. If you are interested in further leak analysis see http://developers.sun.com/solaris/articles/libumem_library.html if you want your leaks to magically disappear see http://developers.sun.com/solaris/articles/libgc.html rick -- Rickey C. Weisner Software Development and Performance Specialist Sun Microsystems, INC cell phone: 615-308-1147 email: rick.weisner at sun.com