Andrej Gessel
2017-Mar-30 21:32 UTC
[Samba] possible memory leak in ldb module while dbcheck on RODC
An embedded and charset-unspecified text was scrubbed... Name: memory_profiler.txt URL: <http://lists.samba.org/pipermail/samba/attachments/20170330/f5d10ac9/memory_profiler.txt>
Andrej Gessel
2017-Mar-31 06:55 UTC
[Samba] possible memory leak in ldb module while dbcheck on RODC
Hello,
my message was not send with my attachment, because content-type was not
allowed, sorry for that.
i am trying to use Samba 4.5.7 as an RODC and maybe found a memory leak in ldb
module.
There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178 and a
known fix in dbcheck.
If i run "samba-tool dbcheck --cross-ncs --fix", this running process
starts with ~300MB and use at the end about 1,6GB of memory on test domain with
15000 users and 15000 groups(ntds.dit is only ~150MB). I tried to find out why
it is increasing continuously.
1) I used memory_profiler and modified dbchecker.py:
root at buildhost samba # git diff dbchecker.py
diff --git a/dbchecker.py b/dbchecker.py
index 032c0e7..6fe0d4d 100644
--- a/dbchecker.py
+++ b/dbchecker.py
@@ -17,6 +17,7 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
+from memory_profiler import profile
import ldb
import samba
import time
@@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
return False
return True
+ @profile
def do_modify(self, m, controls, msg, validate=True):
'''perform a modify with optional verbose
output'''
if self.verbose:
@@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn), str(to_base)))
"Failed to rename object %s into %s" %
(obj.dn, new_dn)):
self.report("Renamed %s into %s" % (obj.dn, new_dn))
+ @profile
def err_wrong_instancetype(self, obj, calculated_instancetype):
'''handle a wrong instanceType'''
self.report("ERROR: wrong instanceType %s on %s, should be
%d" % (obj["instanceType"], obj.dn, calculated_instancetype))
@@ -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn), str(to_base)))
raise KeyError
+ @profile(precision=4)
def check_object(self, dn, attrs=['*']):
'''check one object'''
if self.verbose:
Output of memory_profiler shows me additional ~2MB of memory usage after every
run of check_object function (show memory_profiler.txt for details partial,
because of mailing list limits).
2) After i comment out the line in dbchecker.py, memory is not increasing
anymore.
@@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
return False
return True
+ @profile
def do_modify(self, m, controls, msg, validate=True):
'''perform a modify with optional verbose
output'''
if self.verbose:
self.report(self.samdb.write_ldif(m, ldb.CHANGETYPE_MODIFY))
try:
controls = controls + ["local_oid:%s:0" %
dsdb.DSDB_CONTROL_DBCHECK]
- self.samdb.modify(m, controls=controls, validate=validate)
+# self.samdb.modify(m, controls=controls, validate=validate)
except Exception, err:
if self.in_transaction:
raise CommandError("%s : %s" % (msg, err))
3) My first idea was that it is python problem with starting c modules, but then
i found pyldb.c and py_ldb_modify function in it. Commenting out the line with
ldb_request memory stays on the same value and does not increase anymore.
root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c
diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c
index bea837f..ed5309a 100644
--- a/lib/ldb/pyldb.c
+++ b/lib/ldb/pyldb.c
@@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject *self, PyObject
*args, PyObject *kwar
return NULL;
}
- ret = ldb_request(ldb_ctx, req);
+ //ret = ldb_request(ldb_ctx, req);
if (ret == LDB_SUCCESS) {
ret = ldb_wait(req->handle, LDB_WAIT_ALL);
}
So now to my question:
Is it really a bug or does ldb library do something, that i did not
see/understand till now?
regards,
Andrej
Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von Andrej
Gessel via samba
Gesendet: Donnerstag, 30. März 2017 23:33
An: samba at lists.samba.org
Betreff: [Samba] possible memory leak in ldb module while dbcheck on RODC
L.P.H. van Belle
2017-Mar-31 07:02 UTC
[Samba] possible memory leak in ldb module while dbcheck on RODC
What is the ldb version your using? You need : 1.1.29 with samba 4.5.7. Greetz, Louis> -----Oorspronkelijk bericht----- > Van: samba [mailto:samba-bounces at lists.samba.org] Namens Andrej Gessel via > samba > Verzonden: vrijdag 31 maart 2017 8:55 > Aan: samba at lists.samba.org > Onderwerp: Re: [Samba] possible memory leak in ldb module while dbcheck on > RODC > > Hello, > > my message was not send with my attachment, because content-type was not > allowed, sorry for that. > > i am trying to use Samba 4.5.7 as an RODC and maybe found a memory leak in > ldb module. > > There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178 and > a known fix in dbcheck. > > If i run "samba-tool dbcheck --cross-ncs --fix", this running process > starts with ~300MB and use at the end about 1,6GB of memory on test domain > with 15000 users and 15000 groups(ntds.dit is only ~150MB). I tried to > find out why it is increasing continuously. > > > 1) I used memory_profiler and modified dbchecker.py: > > root at buildhost samba # git diff dbchecker.py > diff --git a/dbchecker.py b/dbchecker.py > index 032c0e7..6fe0d4d 100644 > --- a/dbchecker.py > +++ b/dbchecker.py > @@ -17,6 +17,7 @@ > # along with this program. If not, see <http://www.gnu.org/licenses/>. > # > > +from memory_profiler import profile > import ldb > import samba > import time > @@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" % (dn, guid_suffix), > return False > return True > > + @profile > def do_modify(self, m, controls, msg, validate=True): > '''perform a modify with optional verbose output''' > if self.verbose: > @@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), > str(to_base))) > "Failed to rename object %s into %s" % (obj.dn, > new_dn)): > self.report("Renamed %s into %s" % (obj.dn, new_dn)) > > + @profile > def err_wrong_instancetype(self, obj, calculated_instancetype): > '''handle a wrong instanceType''' > self.report("ERROR: wrong instanceType %s on %s, should be %d" % > (obj["instanceType"], obj.dn, calculated_instancetype)) > @@ -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), > str(to_base))) > > raise KeyError > > + @profile(precision=4) > def check_object(self, dn, attrs=['*']): > '''check one object''' > if self.verbose: > > Output of memory_profiler shows me additional ~2MB of memory usage after > every run of check_object function (show memory_profiler.txt for details > partial, because of mailing list limits). > > 2) After i comment out the line in dbchecker.py, memory is not increasing > anymore. > > @@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" % (dn, guid_suffix), > return False > return True > > + @profile > def do_modify(self, m, controls, msg, validate=True): > '''perform a modify with optional verbose output''' > if self.verbose: > self.report(self.samdb.write_ldif(m, ldb.CHANGETYPE_MODIFY)) > try: > controls = controls + ["local_oid:%s:0" % > dsdb.DSDB_CONTROL_DBCHECK] > - self.samdb.modify(m, controls=controls, validate=validate) > +# self.samdb.modify(m, controls=controls, validate=validate) > except Exception, err: > if self.in_transaction: > raise CommandError("%s : %s" % (msg, err)) > > > > 3) My first idea was that it is python problem with starting c modules, > but then i found pyldb.c and py_ldb_modify function in it. Commenting out > the line with ldb_request memory stays on the same value and does not > increase anymore. > > root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c > diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c > index bea837f..ed5309a 100644 > --- a/lib/ldb/pyldb.c > +++ b/lib/ldb/pyldb.c > @@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject *self, > PyObject *args, PyObject *kwar > return NULL; > } > > - ret = ldb_request(ldb_ctx, req); > + //ret = ldb_request(ldb_ctx, req); > if (ret == LDB_SUCCESS) { > ret = ldb_wait(req->handle, LDB_WAIT_ALL); > } > > > > So now to my question: > Is it really a bug or does ldb library do something, that i did not > see/understand till now? > > > regards, > > Andrej > > Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von Andrej > Gessel via samba > Gesendet: Donnerstag, 30. März 2017 23:33 > An: samba at lists.samba.org > Betreff: [Samba] possible memory leak in ldb module while dbcheck on RODC > > > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba
Andrej Gessel
2017-Mar-31 07:10 UTC
[Samba] possible memory leak in ldb module while dbcheck on RODC
Hello, i just checkout git repository and run # ./configure.develorer # make # make install As i can read from lib/ldb/wscript it is version 1.1.27? How can i change to the version 1.1.29? Regards, Andrej -----Ursprüngliche Nachricht----- Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von L.P.H. van Belle via samba Gesendet: Freitag, 31. März 2017 09:03 An: samba at lists.samba.org Betreff: Re: [Samba] possible memory leak in ldb module while dbcheck on RODC What is the ldb version your using? You need : 1.1.29 with samba 4.5.7. Greetz, Louis> -----Oorspronkelijk bericht----- > Van: samba [mailto:samba-bounces at lists.samba.org] Namens Andrej Gessel > via samba > Verzonden: vrijdag 31 maart 2017 8:55 > Aan: samba at lists.samba.org > Onderwerp: Re: [Samba] possible memory leak in ldb module while > dbcheck on RODC > > Hello, > > my message was not send with my attachment, because content-type was > not allowed, sorry for that. > > i am trying to use Samba 4.5.7 as an RODC and maybe found a memory > leak in ldb module. > > There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178 > and a known fix in dbcheck. > > If i run "samba-tool dbcheck --cross-ncs --fix", this running process > starts with ~300MB and use at the end about 1,6GB of memory on test > domain with 15000 users and 15000 groups(ntds.dit is only ~150MB). I > tried to find out why it is increasing continuously. > > > 1) I used memory_profiler and modified dbchecker.py: > > root at buildhost samba # git diff dbchecker.py diff --git a/dbchecker.py > b/dbchecker.py index 032c0e7..6fe0d4d 100644 > --- a/dbchecker.py > +++ b/dbchecker.py > @@ -17,6 +17,7 @@ > # along with this program. If not, see <http://www.gnu.org/licenses/>. > # > > +from memory_profiler import profile > import ldb > import samba > import time > @@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" % (dn, guid_suffix), > return False > return True > > + @profile > def do_modify(self, m, controls, msg, validate=True): > '''perform a modify with optional verbose output''' > if self.verbose: > @@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), > str(to_base))) > "Failed to rename object %s into %s" % > (obj.dn, > new_dn)): > self.report("Renamed %s into %s" % (obj.dn, new_dn)) > > + @profile > def err_wrong_instancetype(self, obj, calculated_instancetype): > '''handle a wrong instanceType''' > self.report("ERROR: wrong instanceType %s on %s, should be > %d" % (obj["instanceType"], obj.dn, calculated_instancetype)) @@ > -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), > str(to_base))) > > raise KeyError > > + @profile(precision=4) > def check_object(self, dn, attrs=['*']): > '''check one object''' > if self.verbose: > > Output of memory_profiler shows me additional ~2MB of memory usage > after every run of check_object function (show memory_profiler.txt for > details partial, because of mailing list limits). > > 2) After i comment out the line in dbchecker.py, memory is not > increasing anymore. > > @@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" % (dn, guid_suffix), > return False > return True > > + @profile > def do_modify(self, m, controls, msg, validate=True): > '''perform a modify with optional verbose output''' > if self.verbose: > self.report(self.samdb.write_ldif(m, ldb.CHANGETYPE_MODIFY)) > try: > controls = controls + ["local_oid:%s:0" % > dsdb.DSDB_CONTROL_DBCHECK] > - self.samdb.modify(m, controls=controls, validate=validate) > +# self.samdb.modify(m, controls=controls, validate=validate) > except Exception, err: > if self.in_transaction: > raise CommandError("%s : %s" % (msg, err)) > > > > 3) My first idea was that it is python problem with starting c > modules, but then i found pyldb.c and py_ldb_modify function in it. > Commenting out the line with ldb_request memory stays on the same > value and does not increase anymore. > > root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c > diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c index bea837f..ed5309a > 100644 > --- a/lib/ldb/pyldb.c > +++ b/lib/ldb/pyldb.c > @@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject > *self, PyObject *args, PyObject *kwar > return NULL; > } > > - ret = ldb_request(ldb_ctx, req); > + //ret = ldb_request(ldb_ctx, req); > if (ret == LDB_SUCCESS) { > ret = ldb_wait(req->handle, LDB_WAIT_ALL); > } > > > > So now to my question: > Is it really a bug or does ldb library do something, that i did not > see/understand till now? > > > regards, > > Andrej > > Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von > Andrej Gessel via samba > Gesendet: Donnerstag, 30. März 2017 23:33 > An: samba at lists.samba.org > Betreff: [Samba] possible memory leak in ldb module while dbcheck on > RODC > > > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba-- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
L.P.H. van Belle
2017-Mar-31 07:28 UTC
[Samba] possible memory leak in ldb module while dbcheck on RODC
Hai Andrej, Oeps, Sorry, my mistake 4.6.x needs at least 1.1.29. But you can upgrade to 4.6.x ;-) Check again and if that problem then still exists, then i think you can better ask this one in the samba-technical list. Or if Rowland knows something more about this. One tip, wait untill tomorrow or the weekend, there wil be a new release due to the regression bug. There planning is to be release today. Louis> -----Oorspronkelijk bericht----- > Van: Andrej Gessel [mailto:Andrej.Gessel at janztec.com] > Verzonden: vrijdag 31 maart 2017 9:11 > Aan: L.P.H. van Belle; samba at lists.samba.org > Onderwerp: AW: [Samba] possible memory leak in ldb module while dbcheck on > RODC > > Hello, > > i just checkout git repository and run > # ./configure.develorer > # make > # make install > > As i can read from lib/ldb/wscript it is version 1.1.27? > > How can i change to the version 1.1.29? > > Regards, > > Andrej > > > -----Ursprüngliche Nachricht----- > Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von L.P.H. > van Belle via samba > Gesendet: Freitag, 31. März 2017 09:03 > An: samba at lists.samba.org > Betreff: Re: [Samba] possible memory leak in ldb module while dbcheck on > RODC > > What is the ldb version your using? > You need : 1.1.29 with samba 4.5.7. > > Greetz, > > Louis > > > > -----Oorspronkelijk bericht----- > > Van: samba [mailto:samba-bounces at lists.samba.org] Namens Andrej Gessel > > via samba > > Verzonden: vrijdag 31 maart 2017 8:55 > > Aan: samba at lists.samba.org > > Onderwerp: Re: [Samba] possible memory leak in ldb module while > > dbcheck on RODC > > > > Hello, > > > > my message was not send with my attachment, because content-type was > > not allowed, sorry for that. > > > > i am trying to use Samba 4.5.7 as an RODC and maybe found a memory > > leak in ldb module. > > > > There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178 > > and a known fix in dbcheck. > > > > If i run "samba-tool dbcheck --cross-ncs --fix", this running process > > starts with ~300MB and use at the end about 1,6GB of memory on test > > domain with 15000 users and 15000 groups(ntds.dit is only ~150MB). I > > tried to find out why it is increasing continuously. > > > > > > 1) I used memory_profiler and modified dbchecker.py: > > > > root at buildhost samba # git diff dbchecker.py diff --git a/dbchecker.py > > b/dbchecker.py index 032c0e7..6fe0d4d 100644 > > --- a/dbchecker.py > > +++ b/dbchecker.py > > @@ -17,6 +17,7 @@ > > # along with this program. If not, see <http://www.gnu.org/licenses/>. > > # > > > > +from memory_profiler import profile > > import ldb > > import samba > > import time > > @@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" % (dn, guid_suffix), > > return False > > return True > > > > + @profile > > def do_modify(self, m, controls, msg, validate=True): > > '''perform a modify with optional verbose output''' > > if self.verbose: > > @@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), > > str(to_base))) > > "Failed to rename object %s into %s" % > > (obj.dn, > > new_dn)): > > self.report("Renamed %s into %s" % (obj.dn, new_dn)) > > > > + @profile > > def err_wrong_instancetype(self, obj, calculated_instancetype): > > '''handle a wrong instanceType''' > > self.report("ERROR: wrong instanceType %s on %s, should be > > %d" % (obj["instanceType"], obj.dn, calculated_instancetype)) @@ > > -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), > > str(to_base))) > > > > raise KeyError > > > > + @profile(precision=4) > > def check_object(self, dn, attrs=['*']): > > '''check one object''' > > if self.verbose: > > > > Output of memory_profiler shows me additional ~2MB of memory usage > > after every run of check_object function (show memory_profiler.txt for > > details partial, because of mailing list limits). > > > > 2) After i comment out the line in dbchecker.py, memory is not > > increasing anymore. > > > > @@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" % (dn, guid_suffix), > > return False > > return True > > > > + @profile > > def do_modify(self, m, controls, msg, validate=True): > > '''perform a modify with optional verbose output''' > > if self.verbose: > > self.report(self.samdb.write_ldif(m, > ldb.CHANGETYPE_MODIFY)) > > try: > > controls = controls + ["local_oid:%s:0" % > > dsdb.DSDB_CONTROL_DBCHECK] > > - self.samdb.modify(m, controls=controls, validate=validate) > > +# self.samdb.modify(m, controls=controls, validate=validate) > > except Exception, err: > > if self.in_transaction: > > raise CommandError("%s : %s" % (msg, err)) > > > > > > > > 3) My first idea was that it is python problem with starting c > > modules, but then i found pyldb.c and py_ldb_modify function in it. > > Commenting out the line with ldb_request memory stays on the same > > value and does not increase anymore. > > > > root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c > > diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c index bea837f..ed5309a > > 100644 > > --- a/lib/ldb/pyldb.c > > +++ b/lib/ldb/pyldb.c > > @@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject > > *self, PyObject *args, PyObject *kwar > > return NULL; > > } > > > > - ret = ldb_request(ldb_ctx, req); > > + //ret = ldb_request(ldb_ctx, req); > > if (ret == LDB_SUCCESS) { > > ret = ldb_wait(req->handle, LDB_WAIT_ALL); > > } > > > > > > > > So now to my question: > > Is it really a bug or does ldb library do something, that i did not > > see/understand till now? > > > > > > regards, > > > > Andrej > > > > Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von > > Andrej Gessel via samba > > Gesendet: Donnerstag, 30. März 2017 23:33 > > An: samba at lists.samba.org > > Betreff: [Samba] possible memory leak in ldb module while dbcheck on > > RODC > > > > > > -- > > To unsubscribe from this list go to the following URL and read the > > instructions: https://lists.samba.org/mailman/options/samba > > > > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba
Apparently Analagous Threads
- possible memory leak in ldb module while dbcheck on RODC
- possible memory leak in ldb module while dbcheck on RODC
- Error running "samba-tool dbcheck" after going from 4.8.6 to 4.9.2
- Error running "samba-tool dbcheck" after going from 4.8.6 to 4.9.2
- Can not build Samba 4.7.10