Hi all, Recently I have upgraded a Python application from Xapian 1.0.7 to 1.2.2 in order to use the PostingSource class. It is a long-running process, and I am seeing the number of open file descriptors to the Xapian database steadily increase. I suspect what I am seeing is some kind of resource leak. I have no idea if it is a problem in our code or in the Xapian Python bindings. How do I debug this problem? Regards, Joost -- Joost Cassee http://joost.cassee.net
On Mon, Aug 09, 2010 at 02:41:03PM +0200, Joost Cassee wrote:> Recently I have upgraded a Python application from Xapian 1.0.7 to > 1.2.2 in order to use the PostingSource class. It is a long-running > process, and I am seeing the number of open file descriptors to the > Xapian database steadily increase. I suspect what I am seeing is some > kind of resource leak.We have machinery to report leaked file descriptors in the xapian-core testsuite on Linux and some other platforms, so if there is a leak at the C++ level, it is in a case that isn't covered by the testsuite, or which is specific to certain non-Linux platforms. Or the fd leak checking machinery doesn't work fully!> I have no idea if it is a problem in our code or in the Xapian Python > bindings. How do I debug this problem?If you're on Linux (or another platform which supports it) try: ls -l /proc/PID/fd where PID is the process id of your Xapian-using Python process. This will show all the fds it has open, and which files they are, which should provide a clue. Cheers, Olly
Hi Michel, On Mon, Aug 9, 2010 at 18:09, Michel Pelletier <pelletier.michel at gmail.com> wrote:> Is your code creating database objects in a loop that you are holding > references to? If so each object will hold an open fd to various > database files. If you don't close or release all references to a > database then it will not be garbage collected and it's fds will > remain open.Your comment have set me on the right track. It is a problem with a circular reference. I am trying to extract a minimal example, but in the mean time, how can this happen:>>> import xapian, weakref, gc >>> test = Test() >>> ref = weakref.ref(test) >>> test = Test() >>> ref()<Test object at 0xa16890c>>>> len(gc.get_referrers(ref()))1>>> len(gc.get_referrers(gc.get_referrers(ref())[0]))1>>> len(gc.get_referrers(gc.get_referrers(gc.get_referrers(ref())[0])[0]))1>>> len(gc.get_referrers(gc.get_referrers(gc.get_referrers(gc.get_referrers(ref())[0])[0])[0]))1>>> gc.get_referrers(gc.get_referrers(gc.get_referrers(gc.get_referrers(ref())[0])[0])[0])[0] is ref()True I realize that this may no longer be a Xapian issue, but it only seems to happen with instances holding Xapian objects. Regards, Joost PS. Sorry Michel, if you got this message twice; I sent it to your address by accident earlier. -- Joost Cassee http://joost.cassee.net