thr3ads.net - samba - [Samba] possible memory leak in ldb module while dbcheck on RODC [Mar 2017]

If this information is useful, please help other people find it:
Share via:

Andrej Gessel

2017-Mar-30 21:32 UTC

[Samba] possible memory leak in ldb module while dbcheck on RODC

An embedded and charset-unspecified text was scrubbed...
Name: memory_profiler.txt
URL:
<http://lists.samba.org/pipermail/samba/attachments/20170330/f5d10ac9/memory_profiler.txt>

Andrej Gessel

2017-Mar-31 06:55 UTC

head link

[Samba] possible memory leak in ldb module while dbcheck on RODC

Hello,

my message was not send with my attachment, because content-type was not
allowed,  sorry for that.

i am trying to use Samba 4.5.7 as an RODC and maybe found a memory leak in ldb
module.

There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178 and a
known fix in dbcheck.

If i run "samba-tool dbcheck --cross-ncs --fix", this running process
starts with ~300MB and use at the end about 1,6GB of memory on test domain with
15000 users and 15000 groups(ntds.dit is only ~150MB). I tried to find out why
it is increasing continuously.


1) I used memory_profiler and modified dbchecker.py:

root at buildhost samba # git diff dbchecker.py
diff --git a/dbchecker.py b/dbchecker.py
index 032c0e7..6fe0d4d 100644
--- a/dbchecker.py
+++ b/dbchecker.py
@@ -17,6 +17,7 @@
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
#

+from memory_profiler import profile
import ldb
import samba
import time
@@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
             return False
         return True

+    @profile
     def do_modify(self, m, controls, msg, validate=True):
         '''perform a modify with optional verbose
output'''
         if self.verbose:
@@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn), str(to_base)))
                           "Failed to rename object %s into %s" %
(obj.dn, new_dn)):
             self.report("Renamed %s into %s" % (obj.dn, new_dn))

+    @profile
     def err_wrong_instancetype(self, obj, calculated_instancetype):
         '''handle a wrong instanceType'''
         self.report("ERROR: wrong instanceType %s on %s, should be
%d" % (obj["instanceType"], obj.dn, calculated_instancetype))
@@ -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn), str(to_base)))

         raise KeyError

+    @profile(precision=4)
     def check_object(self, dn, attrs=['*']):
         '''check one object'''
         if self.verbose:

Output of memory_profiler shows me additional ~2MB of memory usage after every
run of check_object function (show memory_profiler.txt for details partial,
because of mailing list limits).

2) After i comment out the line in dbchecker.py, memory is not increasing
anymore.

@@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
             return False
         return True

+    @profile
     def do_modify(self, m, controls, msg, validate=True):
         '''perform a modify with optional verbose
output'''
         if self.verbose:
             self.report(self.samdb.write_ldif(m, ldb.CHANGETYPE_MODIFY))
         try:
             controls = controls + ["local_oid:%s:0" %
dsdb.DSDB_CONTROL_DBCHECK]
-            self.samdb.modify(m, controls=controls, validate=validate)
+#            self.samdb.modify(m, controls=controls, validate=validate)
         except Exception, err:
            if self.in_transaction:
                 raise CommandError("%s : %s" % (msg, err))



3) My first idea was that it is python problem with starting c modules, but then
i found pyldb.c and py_ldb_modify function in it. Commenting out the line with
ldb_request memory stays on the same value and does not increase anymore.

root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c
diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c
index bea837f..ed5309a 100644
--- a/lib/ldb/pyldb.c
+++ b/lib/ldb/pyldb.c
@@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject *self, PyObject
*args, PyObject *kwar
                return NULL;
        }

-       ret = ldb_request(ldb_ctx, req);
+       //ret = ldb_request(ldb_ctx, req);
        if (ret == LDB_SUCCESS) {
                ret = ldb_wait(req->handle, LDB_WAIT_ALL);
        }



So now to my question:
Is it really a bug or does ldb library do something, that i did not
see/understand till now?


regards,

Andrej

Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von Andrej
Gessel via samba
Gesendet: Donnerstag, 30. März 2017 23:33
An: samba at lists.samba.org
Betreff: [Samba] possible memory leak in ldb module while dbcheck on RODC

L.P.H. van Belle

2017-Mar-31 07:02 UTC

head link

[Samba] possible memory leak in ldb module while dbcheck on RODC

What is the ldb version your using? 
You need : 1.1.29 with samba 4.5.7. 

Greetz, 

Louis

> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Andrej Gessel
via
> samba
> Verzonden: vrijdag 31 maart 2017 8:55
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] possible memory leak in ldb module while dbcheck on
> RODC
> 
> Hello,
> 
> my message was not send with my attachment, because content-type was not
> allowed,  sorry for that.
> 
> i am trying to use Samba 4.5.7 as an RODC and maybe found a memory leak in
> ldb module.
> 
> There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178 and
> a known fix in dbcheck.
> 
> If i run "samba-tool dbcheck --cross-ncs --fix", this running
process
> starts with ~300MB and use at the end about 1,6GB of memory on test domain
> with 15000 users and 15000 groups(ntds.dit is only ~150MB). I tried to
> find out why it is increasing continuously.
> 
> 
> 1) I used memory_profiler and modified dbchecker.py:
> 
> root at buildhost samba # git diff dbchecker.py
> diff --git a/dbchecker.py b/dbchecker.py
> index 032c0e7..6fe0d4d 100644
> --- a/dbchecker.py
> +++ b/dbchecker.py
> @@ -17,6 +17,7 @@
> # along with this program.  If not, see
<http://www.gnu.org/licenses/>.
> #
> 
> +from memory_profiler import profile
> import ldb
> import samba
> import time
> @@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
>              return False
>          return True
> 
> +    @profile
>      def do_modify(self, m, controls, msg, validate=True):
>          '''perform a modify with optional verbose
output'''
>          if self.verbose:
> @@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn),
> str(to_base)))
>                            "Failed to rename object %s into %s" %
(obj.dn,
> new_dn)):
>              self.report("Renamed %s into %s" % (obj.dn, new_dn))
> 
> +    @profile
>      def err_wrong_instancetype(self, obj, calculated_instancetype):
>          '''handle a wrong instanceType'''
>          self.report("ERROR: wrong instanceType %s on %s, should be
%d" %
> (obj["instanceType"], obj.dn, calculated_instancetype))
> @@ -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn),
> str(to_base)))
> 
>          raise KeyError
> 
> +    @profile(precision=4)
>      def check_object(self, dn, attrs=['*']):
>          '''check one object'''
>          if self.verbose:
> 
> Output of memory_profiler shows me additional ~2MB of memory usage after
> every run of check_object function (show memory_profiler.txt for details
> partial, because of mailing list limits).
> 
> 2) After i comment out the line in dbchecker.py, memory is not increasing
> anymore.
> 
> @@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
>              return False
>          return True
> 
> +    @profile
>      def do_modify(self, m, controls, msg, validate=True):
>          '''perform a modify with optional verbose
output'''
>          if self.verbose:
>              self.report(self.samdb.write_ldif(m, ldb.CHANGETYPE_MODIFY))
>          try:
>              controls = controls + ["local_oid:%s:0" %
> dsdb.DSDB_CONTROL_DBCHECK]
> -            self.samdb.modify(m, controls=controls, validate=validate)
> +#            self.samdb.modify(m, controls=controls, validate=validate)
>          except Exception, err:
>             if self.in_transaction:
>                  raise CommandError("%s : %s" % (msg, err))
> 
> 
> 
> 3) My first idea was that it is python problem with starting c modules,
> but then i found pyldb.c and py_ldb_modify function in it. Commenting out
> the line with ldb_request memory stays on the same value and does not
> increase anymore.
> 
> root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c
> diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c
> index bea837f..ed5309a 100644
> --- a/lib/ldb/pyldb.c
> +++ b/lib/ldb/pyldb.c
> @@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject *self,
> PyObject *args, PyObject *kwar
>                 return NULL;
>         }
> 
> -       ret = ldb_request(ldb_ctx, req);
> +       //ret = ldb_request(ldb_ctx, req);
>         if (ret == LDB_SUCCESS) {
>                 ret = ldb_wait(req->handle, LDB_WAIT_ALL);
>         }
> 
> 
> 
> So now to my question:
> Is it really a bug or does ldb library do something, that i did not
> see/understand till now?
> 
> 
> regards,
> 
> Andrej
> 
> Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von Andrej
> Gessel via samba
> Gesendet: Donnerstag, 30. März 2017 23:33
> An: samba at lists.samba.org
> Betreff: [Samba] possible memory leak in ldb module while dbcheck on RODC
> 
> 
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

Andrej Gessel

2017-Mar-31 07:10 UTC

head link

[Samba] possible memory leak in ldb module while dbcheck on RODC

Hello, 

i just checkout git repository and run 
# ./configure.develorer
# make
# make install

As i can read from lib/ldb/wscript it is version 1.1.27?

How can i change to the version 1.1.29?

Regards,

Andrej


-----Ursprüngliche Nachricht-----
Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von L.P.H. van
Belle via samba
Gesendet: Freitag, 31. März 2017 09:03
An: samba at lists.samba.org
Betreff: Re: [Samba] possible memory leak in ldb module while dbcheck on RODC

What is the ldb version your using? 
You need : 1.1.29 with samba 4.5.7. 

Greetz, 

Louis

> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Andrej Gessel 
> via samba
> Verzonden: vrijdag 31 maart 2017 8:55
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] possible memory leak in ldb module while 
> dbcheck on RODC
> 
> Hello,
> 
> my message was not send with my attachment, because content-type was 
> not allowed,  sorry for that.
> 
> i am trying to use Samba 4.5.7 as an RODC and maybe found a memory 
> leak in ldb module.
> 
> There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178 
> and a known fix in dbcheck.
> 
> If i run "samba-tool dbcheck --cross-ncs --fix", this running
process
> starts with ~300MB and use at the end about 1,6GB of memory on test 
> domain with 15000 users and 15000 groups(ntds.dit is only ~150MB). I 
> tried to find out why it is increasing continuously.
> 
> 
> 1) I used memory_profiler and modified dbchecker.py:
> 
> root at buildhost samba # git diff dbchecker.py diff --git a/dbchecker.py 
> b/dbchecker.py index 032c0e7..6fe0d4d 100644
> --- a/dbchecker.py
> +++ b/dbchecker.py
> @@ -17,6 +17,7 @@
> # along with this program.  If not, see
<http://www.gnu.org/licenses/>.
> #
> 
> +from memory_profiler import profile
> import ldb
> import samba
> import time
> @@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
>              return False
>          return True
> 
> +    @profile
>      def do_modify(self, m, controls, msg, validate=True):
>          '''perform a modify with optional verbose
output'''
>          if self.verbose:
> @@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn),
> str(to_base)))
>                            "Failed to rename object %s into %s" %
> (obj.dn,
> new_dn)):
>              self.report("Renamed %s into %s" % (obj.dn, new_dn))
> 
> +    @profile
>      def err_wrong_instancetype(self, obj, calculated_instancetype):
>          '''handle a wrong instanceType'''
>          self.report("ERROR: wrong instanceType %s on %s, should be 
> %d" % (obj["instanceType"], obj.dn,
calculated_instancetype)) @@
> -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn),
> str(to_base)))
> 
>          raise KeyError
> 
> +    @profile(precision=4)
>      def check_object(self, dn, attrs=['*']):
>          '''check one object'''
>          if self.verbose:
> 
> Output of memory_profiler shows me additional ~2MB of memory usage 
> after every run of check_object function (show memory_profiler.txt for 
> details partial, because of mailing list limits).
> 
> 2) After i comment out the line in dbchecker.py, memory is not 
> increasing anymore.
> 
> @@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" % (dn,
guid_suffix),
>              return False
>          return True
> 
> +    @profile
>      def do_modify(self, m, controls, msg, validate=True):
>          '''perform a modify with optional verbose
output'''
>          if self.verbose:
>              self.report(self.samdb.write_ldif(m, ldb.CHANGETYPE_MODIFY))
>          try:
>              controls = controls + ["local_oid:%s:0" % 
> dsdb.DSDB_CONTROL_DBCHECK]
> -            self.samdb.modify(m, controls=controls, validate=validate)
> +#            self.samdb.modify(m, controls=controls, validate=validate)
>          except Exception, err:
>             if self.in_transaction:
>                  raise CommandError("%s : %s" % (msg, err))
> 
> 
> 
> 3) My first idea was that it is python problem with starting c 
> modules, but then i found pyldb.c and py_ldb_modify function in it. 
> Commenting out the line with ldb_request memory stays on the same 
> value and does not increase anymore.
> 
> root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c 
> diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c index bea837f..ed5309a 
> 100644
> --- a/lib/ldb/pyldb.c
> +++ b/lib/ldb/pyldb.c
> @@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject 
> *self, PyObject *args, PyObject *kwar
>                 return NULL;
>         }
> 
> -       ret = ldb_request(ldb_ctx, req);
> +       //ret = ldb_request(ldb_ctx, req);
>         if (ret == LDB_SUCCESS) {
>                 ret = ldb_wait(req->handle, LDB_WAIT_ALL);
>         }
> 
> 
> 
> So now to my question:
> Is it really a bug or does ldb library do something, that i did not 
> see/understand till now?
> 
> 
> regards,
> 
> Andrej
> 
> Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von 
> Andrej Gessel via samba
> Gesendet: Donnerstag, 30. März 2017 23:33
> An: samba at lists.samba.org
> Betreff: [Samba] possible memory leak in ldb module while dbcheck on 
> RODC
> 
> 
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

L.P.H. van Belle

2017-Mar-31 07:28 UTC

head link

[Samba] possible memory leak in ldb module while dbcheck on RODC

Hai Andrej, 

Oeps, Sorry, my mistake 4.6.x needs at least 1.1.29. 

But you can upgrade to 4.6.x ;-) 

Check again and if that problem then still exists, 
then i think you can better ask this one in the samba-technical list. 
Or if Rowland knows something more about this. 

One tip, wait untill tomorrow or the weekend, there wil be a new release due to
the regression bug.
There planning is to be release today. 


Louis

> -----Oorspronkelijk bericht-----
> Van: Andrej Gessel [mailto:Andrej.Gessel at janztec.com]
> Verzonden: vrijdag 31 maart 2017 9:11
> Aan: L.P.H. van Belle; samba at lists.samba.org
> Onderwerp: AW: [Samba] possible memory leak in ldb module while dbcheck on
> RODC
> 
> Hello,
> 
> i just checkout git repository and run
> # ./configure.develorer
> # make
> # make install
> 
> As i can read from lib/ldb/wscript it is version 1.1.27?
> 
> How can i change to the version 1.1.29?
> 
> Regards,
> 
> Andrej
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von L.P.H.
> van Belle via samba
> Gesendet: Freitag, 31. März 2017 09:03
> An: samba at lists.samba.org
> Betreff: Re: [Samba] possible memory leak in ldb module while dbcheck on
> RODC
> 
> What is the ldb version your using?
> You need : 1.1.29 with samba 4.5.7.
> 
> Greetz,
> 
> Louis
> 
> 
> > -----Oorspronkelijk bericht-----
> > Van: samba [mailto:samba-bounces at lists.samba.org] Namens Andrej
Gessel
> > via samba
> > Verzonden: vrijdag 31 maart 2017 8:55
> > Aan: samba at lists.samba.org
> > Onderwerp: Re: [Samba] possible memory leak in ldb module while
> > dbcheck on RODC
> >
> > Hello,
> >
> > my message was not send with my attachment, because content-type was
> > not allowed,  sorry for that.
> >
> > i am trying to use Samba 4.5.7 as an RODC and maybe found a memory
> > leak in ldb module.
> >
> > There is an known BUG https://bugzilla.samba.org/show_bug.cgi?id=11178
> > and a known fix in dbcheck.
> >
> > If i run "samba-tool dbcheck --cross-ncs --fix", this
running process
> > starts with ~300MB and use at the end about 1,6GB of memory on test
> > domain with 15000 users and 15000 groups(ntds.dit is only ~150MB). I
> > tried to find out why it is increasing continuously.
> >
> >
> > 1) I used memory_profiler and modified dbchecker.py:
> >
> > root at buildhost samba # git diff dbchecker.py diff --git
a/dbchecker.py
> > b/dbchecker.py index 032c0e7..6fe0d4d 100644
> > --- a/dbchecker.py
> > +++ b/dbchecker.py
> > @@ -17,6 +17,7 @@
> > # along with this program.  If not, see
<http://www.gnu.org/licenses/>.
> > #
> >
> > +from memory_profiler import profile
> > import ldb
> > import samba
> > import time
> > @@ -356,6 +357,7 @@ systemFlags: -1946157056%s""" %
(dn, guid_suffix),
> >              return False
> >          return True
> >
> > +    @profile
> >      def do_modify(self, m, controls, msg, validate=True):
> >          '''perform a modify with optional verbose
output'''
> >          if self.verbose:
> > @@ -751,6 +753,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn),
> > str(to_base)))
> >                            "Failed to rename object %s into
%s" %
> > (obj.dn,
> > new_dn)):
> >              self.report("Renamed %s into %s" % (obj.dn,
new_dn))
> >
> > +    @profile
> >      def err_wrong_instancetype(self, obj, calculated_instancetype):
> >          '''handle a wrong instanceType'''
> >          self.report("ERROR: wrong instanceType %s on %s, should
be
> > %d" % (obj["instanceType"], obj.dn,
calculated_instancetype)) @@
> > -1541,6 +1544,7 @@ newSuperior: %s""" % (str(from_dn),
str(to_rdn),
> > str(to_base)))
> >
> >          raise KeyError
> >
> > +    @profile(precision=4)
> >      def check_object(self, dn, attrs=['*']):
> >          '''check one object'''
> >          if self.verbose:
> >
> > Output of memory_profiler shows me additional ~2MB of memory usage
> > after every run of check_object function (show memory_profiler.txt for
> > details partial, because of mailing list limits).
> >
> > 2) After i comment out the line in dbchecker.py, memory is not
> > increasing anymore.
> >
> > @@ -356,13 +357,14 @@ systemFlags: -1946157056%s""" %
(dn, guid_suffix),
> >              return False
> >          return True
> >
> > +    @profile
> >      def do_modify(self, m, controls, msg, validate=True):
> >          '''perform a modify with optional verbose
output'''
> >          if self.verbose:
> >              self.report(self.samdb.write_ldif(m,
> ldb.CHANGETYPE_MODIFY))
> >          try:
> >              controls = controls + ["local_oid:%s:0" %
> > dsdb.DSDB_CONTROL_DBCHECK]
> > -            self.samdb.modify(m, controls=controls,
validate=validate)
> > +#            self.samdb.modify(m, controls=controls,
validate=validate)
> >          except Exception, err:
> >             if self.in_transaction:
> >                  raise CommandError("%s : %s" % (msg, err))
> >
> >
> >
> > 3) My first idea was that it is python problem with starting c
> > modules, but then i found pyldb.c and py_ldb_modify function in it.
> > Commenting out the line with ldb_request memory stays on the same
> > value and does not increase anymore.
> >
> > root at buildhost samba (git)-[samba-4.5.7] # git diff lib/ldb/pyldb.c
> > diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c index bea837f..ed5309a
> > 100644
> > --- a/lib/ldb/pyldb.c
> > +++ b/lib/ldb/pyldb.c
> > @@ -1255,7 +1255,7 @@ static PyObject *py_ldb_modify(PyLdbObject
> > *self, PyObject *args, PyObject *kwar
> >                 return NULL;
> >         }
> >
> > -       ret = ldb_request(ldb_ctx, req);
> > +       //ret = ldb_request(ldb_ctx, req);
> >         if (ret == LDB_SUCCESS) {
> >                 ret = ldb_wait(req->handle, LDB_WAIT_ALL);
> >         }
> >
> >
> >
> > So now to my question:
> > Is it really a bug or does ldb library do something, that i did not
> > see/understand till now?
> >
> >
> > regards,
> >
> > Andrej
> >
> > Von: samba [mailto:samba-bounces at lists.samba.org] Im Auftrag von
> > Andrej Gessel via samba
> > Gesendet: Donnerstag, 30. März 2017 23:33
> > An: samba at lists.samba.org
> > Betreff: [Samba] possible memory leak in ldb module while dbcheck on
> > RODC
> >
> >
> > --
> > To unsubscribe from this list go to the following URL and read the
> > instructions:  https://lists.samba.org/mailman/options/samba
> 
> 
> 
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

Maybe Matching Threads

Search for more maybe matching threads

samba - Mar 2017 - possible memory leak in ldb module while dbcheck on RODC

[Samba] possible memory leak in ldb module while dbcheck on RODC

[Samba] possible memory leak in ldb module while dbcheck on RODC

[Samba] possible memory leak in ldb module while dbcheck on RODC

[Samba] possible memory leak in ldb module while dbcheck on RODC

[Samba] possible memory leak in ldb module while dbcheck on RODC

Maybe Matching Threads