IT Admin
2016-Mar-03 15:32 UTC
[Samba] Samba AD/DC crashed again, third time in as many months
Apt and to think those packages aren't installed: sudo apt-get remove libtdb-dev libtalloc-dev python-talloc-dev Reading package lists... Done Building dependency tree Reading state information... Done Package 'libtalloc-dev' is not installed, so not removed Package 'libtdb-dev' is not installed, so not removed Package 'python-talloc-dev' is not installed, so not removed The following packages were automatically installed and are no longer required: linux-image-4.2.0-27-generic linux-image-extra-4.2.0-27-generic Use 'apt-get autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Should I just rename the files? If the database is corrupt my only recourse is to follow the restore guidelines in the Samba wiki and roll back to a previous version of the db, correct? JS On Mar 3, 2016 7:40 AM, "Sketch" <smblist at rednsx.org> wrote:> I'd remove the distro packages providing those libs in /usr/lib, as they > could possibly cause problems. One more I forgot, which might possibly be > responsible for corrupting your ldb database if the wrong one is loaded by > samba at runtime, is libldb. > > From the log, I'm guessing your database is corrupt if it can't read the > schema, but someone else might have more insight. > > On Wed, 2 Mar 2016, IT Admin wrote: > > pytalloc: >> >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2 >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.2 >> /usr/local/samba/include/pytalloc.h >> /usr/local/samba/lib/private/libpytalloc-util.so.2 >> /usr/local/samba/lib/private/libpytalloc-util.so.2.1.3 >> >> libtalloc: >> >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2 >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.2 >> /usr/local/samba/lib/private/libtalloc-report-samba4.so >> /usr/local/samba/lib/private/libtalloc.so.2 >> /usr/local/samba/lib/private/libtalloc.so.2.1.3 >> >> >> libtdb: >> >> /usr/lib/x86_64-linux-gnu/libtdb.so.1 >> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5 >> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0 >> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0 >> /usr/local/samba/lib/private/libtdb-wrap-samba4.so >> /usr/local/samba/lib/private/libtdb.so.1 >> /usr/local/samba/lib/private/libtdb.so.1.3.7 >> >> >> I also noticed that I've got a log.smbd in /usr/local/samba/var: >> http://www.anonpaste.net/?p=bdfa3 >> >> JS >> >> >> >> On Wed, Mar 2, 2016 at 9:00 PM, Sketch <smblist at rednsx.org> wrote: >> >> Also check for pytalloc|libtalloc and libtdb. >>> >>> >>> On Wed, 2 Mar 2016, IT Admin wrote: >>> >>> I poked around the system using locate, I don't think there are any >>> >>>> packages installed on this system other than those associated with the >>>> 4.3.3 build I compiled. >>>> >>>> smbd >>>> >>>> /usr/lib/python2.7/dist-packages/samba/samba3/smbd.so >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0 >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_conn.so.0 >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0 >>>> /usr/local/samba/lib/private/libsmbd-base-samba4.so >>>> /usr/local/samba/lib/private/libsmbd-conn-samba4.so >>>> /usr/local/samba/lib/private/libsmbd-shim-samba4.so >>>> /usr/local/samba/lib/python2.7/site-packages/samba/samba3/smbd.so >>>> /usr/local/samba/private/smbd.tmp >>>> /usr/local/samba/sbin/smbd >>>> /usr/local/samba/share/man/man8/smbd.8 >>>> /usr/local/samba/var/log.smbd >>>> /usr/local/samba/var/log.smbd.old >>>> /usr/local/samba/var/run/smbd.pid >>>> /var/log/upstart/smbd.log.1.gz >>>> >>>> nmbd >>>> >>>> /usr/local/samba/sbin/nmbd >>>> /usr/local/samba/share/man/man8/nmbd.8 >>>> /var/log/upstart/nmbd.log.1.gz >>>> >>>> samba >>>> >>>> /usr/local/samba/sbin/nmbd >>>> /usr/local/samba/sbin/samba >>>> /usr/local/samba/sbin/samba_dnsupdate >>>> /usr/local/samba/sbin/samba_kcc >>>> /usr/local/samba/sbin/samba_spnupdate >>>> /usr/local/samba/sbin/samba_upgradedns >>>> /usr/local/samba/sbin/smbd >>>> /usr/local/samba/sbin/winbind >>>> >>>> >>>> If I'm overlooking something obvious please let me know. >>>> >>>> JS >>>> >>>> On Wed, Mar 2, 2016 at 5:42 PM, Marc Muehlfeld <mmuehlfeld at samba.org> >>>> wrote: >>>> >>>> Am 02.03.2016 um 18:52 schrieb IT Admin: >>>> >>>>> >>>>> Samba is compiled from source. >>>>>> Samba DB is stored on local RAID array. >>>>>> Changes to AD are done using ADUC from a Windows 7 box. >>>>>> AD is used for authentication, user shares (folder redirection), and >>>>>> >>>>>> shared >>>>> >>>>> folders. >>>>>> >>>>>> >>>>> Can you add >>>>> log level = 10 >>>>> to your smb.conf, empty your log directory and start Samba. It should >>>>> generate a new log, that captures all output. Then put it to >>>>> cpaste.org >>>>> or some other paste service and share the link with us. Maybe we see >>>>> something interesting. >>>>> >>>>> >>>>> Does something changed when the problem occured the first time? Some >>>>> package updates, crashes, etc.? >>>>> >>>>> >>>>> Can you make sure that no kind of Samba package (daemon, libs, etc.) is >>>>> installed on the system? Maybe your selfcompiled version overwrites >>>>> some >>>>> stuff and your OS installed an update, that mixes now with the self >>>>> compiled version. Just a guess. >>>>> >>>>> Regards, >>>>> Marc >>>>> >>>>> -- >>>>> >>>> To unsubscribe from this list go to the following URL and read the >>>> instructions: https://lists.samba.org/mailman/options/samba >>>> >>>> >>>> >>>> -- >> To unsubscribe from this list go to the following URL and read the >> instructions: https://lists.samba.org/mailman/options/samba >> >> >>
mathias dufresne
2016-Mar-03 16:30 UTC
[Samba] Samba AD/DC crashed again, third time in as many months
2016-03-03 16:32 GMT+01:00 IT Admin <it at cliffbells.com>:> Apt and to think those packages aren't installed: > > sudo apt-get remove libtdb-dev libtalloc-dev python-talloc-dev > Reading package lists... Done > Building dependency tree > Reading state information... Done > Package 'libtalloc-dev' is not installed, so not removed > Package 'libtdb-dev' is not installed, so not removed > Package 'python-talloc-dev' is not installed, so not removed > The following packages were automatically installed and are no longer > required: > linux-image-4.2.0-27-generic linux-image-extra-4.2.0-27-generic > Use 'apt-get autoremove' to remove them. > 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. > > Should I just rename the files? > > If the database is corrupt my only recourse is to follow the restore > guidelines in the Samba wiki and roll back to a previous version of the db, > correct? > > JS >I'd say that depend on how complex is that corruption and how much you have around you to understand that corruption. If it is too complex (main reason to be too complex is a boss in a hurry) restore the whole thing. During restoration what you need is the same kind of system, you don't really need to restore on the very same system. What I mean is you can take advantage of that issue to deploy a new system (using same version of that system, ie if it was centos6, use centos6). As you seem to have a big system doing lot of things, part of these things is running a Samba and even that software is configured to do several things (AD + file server) which is not advised, I would use that down time to switch Samba from that big system to some virtual machine with minimalistic system doing one and only one thing: Samba as AD. Then in a second VM I would install file server. In fact before installing file server I would create another VM to host a second DC. AD with one DC is not advised, anywhere. If you were having several DC perhaps you would not have the whole AD broken but only one DC broken and the other(s) one(s) working well. This was already explained to you today, I was just insisting a little bit : )> On Mar 3, 2016 7:40 AM, "Sketch" <smblist at rednsx.org> wrote: > > > I'd remove the distro packages providing those libs in /usr/lib, as they > > could possibly cause problems. One more I forgot, which might possibly > be > > responsible for corrupting your ldb database if the wrong one is loaded > by > > samba at runtime, is libldb. > > > > From the log, I'm guessing your database is corrupt if it can't read the > > schema, but someone else might have more insight. > > > > On Wed, 2 Mar 2016, IT Admin wrote: > > > > pytalloc: > >> > >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2 > >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.2 > >> /usr/local/samba/include/pytalloc.h > >> /usr/local/samba/lib/private/libpytalloc-util.so.2 > >> /usr/local/samba/lib/private/libpytalloc-util.so.2.1.3 > >> > >> libtalloc: > >> > >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2 > >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.2 > >> /usr/local/samba/lib/private/libtalloc-report-samba4.so > >> /usr/local/samba/lib/private/libtalloc.so.2 > >> /usr/local/samba/lib/private/libtalloc.so.2.1.3 > >> > >> > >> libtdb: > >> > >> /usr/lib/x86_64-linux-gnu/libtdb.so.1 > >> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5 > >> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0 > >> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0 > >> /usr/local/samba/lib/private/libtdb-wrap-samba4.so > >> /usr/local/samba/lib/private/libtdb.so.1 > >> /usr/local/samba/lib/private/libtdb.so.1.3.7 > >> > >> > >> I also noticed that I've got a log.smbd in /usr/local/samba/var: > >> http://www.anonpaste.net/?p=bdfa3 > >> > >> JS > >> > >> > >> > >> On Wed, Mar 2, 2016 at 9:00 PM, Sketch <smblist at rednsx.org> wrote: > >> > >> Also check for pytalloc|libtalloc and libtdb. > >>> > >>> > >>> On Wed, 2 Mar 2016, IT Admin wrote: > >>> > >>> I poked around the system using locate, I don't think there are any > >>> > >>>> packages installed on this system other than those associated with the > >>>> 4.3.3 build I compiled. > >>>> > >>>> smbd > >>>> > >>>> /usr/lib/python2.7/dist-packages/samba/samba3/smbd.so > >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0 > >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_conn.so.0 > >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0 > >>>> /usr/local/samba/lib/private/libsmbd-base-samba4.so > >>>> /usr/local/samba/lib/private/libsmbd-conn-samba4.so > >>>> /usr/local/samba/lib/private/libsmbd-shim-samba4.so > >>>> /usr/local/samba/lib/python2.7/site-packages/samba/samba3/smbd.so > >>>> /usr/local/samba/private/smbd.tmp > >>>> /usr/local/samba/sbin/smbd > >>>> /usr/local/samba/share/man/man8/smbd.8 > >>>> /usr/local/samba/var/log.smbd > >>>> /usr/local/samba/var/log.smbd.old > >>>> /usr/local/samba/var/run/smbd.pid > >>>> /var/log/upstart/smbd.log.1.gz > >>>> > >>>> nmbd > >>>> > >>>> /usr/local/samba/sbin/nmbd > >>>> /usr/local/samba/share/man/man8/nmbd.8 > >>>> /var/log/upstart/nmbd.log.1.gz > >>>> > >>>> samba > >>>> > >>>> /usr/local/samba/sbin/nmbd > >>>> /usr/local/samba/sbin/samba > >>>> /usr/local/samba/sbin/samba_dnsupdate > >>>> /usr/local/samba/sbin/samba_kcc > >>>> /usr/local/samba/sbin/samba_spnupdate > >>>> /usr/local/samba/sbin/samba_upgradedns > >>>> /usr/local/samba/sbin/smbd > >>>> /usr/local/samba/sbin/winbind > >>>> > >>>> > >>>> If I'm overlooking something obvious please let me know. > >>>> > >>>> JS > >>>> > >>>> On Wed, Mar 2, 2016 at 5:42 PM, Marc Muehlfeld <mmuehlfeld at samba.org> > >>>> wrote: > >>>> > >>>> Am 02.03.2016 um 18:52 schrieb IT Admin: > >>>> > >>>>> > >>>>> Samba is compiled from source. > >>>>>> Samba DB is stored on local RAID array. > >>>>>> Changes to AD are done using ADUC from a Windows 7 box. > >>>>>> AD is used for authentication, user shares (folder redirection), and > >>>>>> > >>>>>> shared > >>>>> > >>>>> folders. > >>>>>> > >>>>>> > >>>>> Can you add > >>>>> log level = 10 > >>>>> to your smb.conf, empty your log directory and start Samba. It should > >>>>> generate a new log, that captures all output. Then put it to > >>>>> cpaste.org > >>>>> or some other paste service and share the link with us. Maybe we see > >>>>> something interesting. > >>>>> > >>>>> > >>>>> Does something changed when the problem occured the first time? Some > >>>>> package updates, crashes, etc.? > >>>>> > >>>>> > >>>>> Can you make sure that no kind of Samba package (daemon, libs, etc.) > is > >>>>> installed on the system? Maybe your selfcompiled version overwrites > >>>>> some > >>>>> stuff and your OS installed an update, that mixes now with the self > >>>>> compiled version. Just a guess. > >>>>> > >>>>> Regards, > >>>>> Marc > >>>>> >
IT Admin
2016-Mar-03 17:34 UTC
[Samba] Samba AD/DC crashed again, third time in as many months
Well, this puts me in a catch-22 situation... I can see the benefit of spinning up VMs as ADCs, unfortunately this machine is already leveraged to the limit and there aren't any resources available to support a single additional VM, let alone two of them. And I'm scratching my head a bit here as I have another Samba ADC deployed on another network, similar host OS, similar RAID setup, and it has been rock solid... I think the best course of action for this specific situation short-term is to restore one of the samba backups to get their domain up and running ASAP. I'll then have to figure out how to shuffle resources to get a second ADC running in a VM and with any luck redundancy will mitigate future corruption issues. That being said I'm very keen to understand what the source of this corruption is, as I mentioned earlier I haven't had any other issues with data corruption on this host and suspect that mismatched libraries are a big part of the problem. So, I need to verify, what is the proper way to remove the unwanted packages in /usr/lib? Am I trying to remove the correct packages with apt? Can I simply rename the offending files and reboot? Once I've gotten rid of those files I'll follow the restore procedure and attempt to get samba running again with last week's backup, that will allow the client to move forward with end of year accounting work and give me a chance to figure out how to shuffle resources so I can implement the dual VM architecture. Please advise, I really need to get this domain functional again by end of day, I've got about 5 hours to do so. Thanks again everyone for all of your help. JS On Mar 3, 2016 11:30 AM, "mathias dufresne" <infractory at gmail.com> wrote:> > > 2016-03-03 16:32 GMT+01:00 IT Admin <it at cliffbells.com>: > >> Apt and to think those packages aren't installed: >> >> sudo apt-get remove libtdb-dev libtalloc-dev python-talloc-dev >> Reading package lists... Done >> Building dependency tree >> Reading state information... Done >> Package 'libtalloc-dev' is not installed, so not removed >> Package 'libtdb-dev' is not installed, so not removed >> Package 'python-talloc-dev' is not installed, so not removed >> The following packages were automatically installed and are no longer >> required: >> linux-image-4.2.0-27-generic linux-image-extra-4.2.0-27-generic >> Use 'apt-get autoremove' to remove them. >> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. >> >> Should I just rename the files? >> >> If the database is corrupt my only recourse is to follow the restore >> guidelines in the Samba wiki and roll back to a previous version of the >> db, >> correct? >> >> JS >> > > I'd say that depend on how complex is that corruption and how much you > have around you to understand that corruption. > > If it is too complex (main reason to be too complex is a boss in a hurry) > restore the whole thing. > > During restoration what you need is the same kind of system, you don't > really need to restore on the very same system. What I mean is you can take > advantage of that issue to deploy a new system (using same version of that > system, ie if it was centos6, use centos6). > As you seem to have a big system doing lot of things, part of these things > is running a Samba and even that software is configured to do several > things (AD + file server) which is not advised, I would use that down time > to switch Samba from that big system to some virtual machine with > minimalistic system doing one and only one thing: Samba as AD. Then in a > second VM I would install file server. In fact before installing file > server I would create another VM to host a second DC. > > AD with one DC is not advised, anywhere. If you were having several DC > perhaps you would not have the whole AD broken but only one DC broken and > the other(s) one(s) working well. This was already explained to you today, > I was just insisting a little bit : ) > > > > >> On Mar 3, 2016 7:40 AM, "Sketch" <smblist at rednsx.org> wrote: >> >> > I'd remove the distro packages providing those libs in /usr/lib, as they >> > could possibly cause problems. One more I forgot, which might possibly >> be >> > responsible for corrupting your ldb database if the wrong one is loaded >> by >> > samba at runtime, is libldb. >> > >> > From the log, I'm guessing your database is corrupt if it can't read the >> > schema, but someone else might have more insight. >> > >> > On Wed, 2 Mar 2016, IT Admin wrote: >> > >> > pytalloc: >> >> >> >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2 >> >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.2 >> >> /usr/local/samba/include/pytalloc.h >> >> /usr/local/samba/lib/private/libpytalloc-util.so.2 >> >> /usr/local/samba/lib/private/libpytalloc-util.so.2.1.3 >> >> >> >> libtalloc: >> >> >> >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2 >> >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.2 >> >> /usr/local/samba/lib/private/libtalloc-report-samba4.so >> >> /usr/local/samba/lib/private/libtalloc.so.2 >> >> /usr/local/samba/lib/private/libtalloc.so.2.1.3 >> >> >> >> >> >> libtdb: >> >> >> >> /usr/lib/x86_64-linux-gnu/libtdb.so.1 >> >> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5 >> >> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0 >> >> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0 >> >> /usr/local/samba/lib/private/libtdb-wrap-samba4.so >> >> /usr/local/samba/lib/private/libtdb.so.1 >> >> /usr/local/samba/lib/private/libtdb.so.1.3.7 >> >> >> >> >> >> I also noticed that I've got a log.smbd in /usr/local/samba/var: >> >> http://www.anonpaste.net/?p=bdfa3 >> >> >> >> JS >> >> >> >> >> >> >> >> On Wed, Mar 2, 2016 at 9:00 PM, Sketch <smblist at rednsx.org> wrote: >> >> >> >> Also check for pytalloc|libtalloc and libtdb. >> >>> >> >>> >> >>> On Wed, 2 Mar 2016, IT Admin wrote: >> >>> >> >>> I poked around the system using locate, I don't think there are any >> >>> >> >>>> packages installed on this system other than those associated with >> the >> >>>> 4.3.3 build I compiled. >> >>>> >> >>>> smbd >> >>>> >> >>>> /usr/lib/python2.7/dist-packages/samba/samba3/smbd.so >> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0 >> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_conn.so.0 >> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0 >> >>>> /usr/local/samba/lib/private/libsmbd-base-samba4.so >> >>>> /usr/local/samba/lib/private/libsmbd-conn-samba4.so >> >>>> /usr/local/samba/lib/private/libsmbd-shim-samba4.so >> >>>> /usr/local/samba/lib/python2.7/site-packages/samba/samba3/smbd.so >> >>>> /usr/local/samba/private/smbd.tmp >> >>>> /usr/local/samba/sbin/smbd >> >>>> /usr/local/samba/share/man/man8/smbd.8 >> >>>> /usr/local/samba/var/log.smbd >> >>>> /usr/local/samba/var/log.smbd.old >> >>>> /usr/local/samba/var/run/smbd.pid >> >>>> /var/log/upstart/smbd.log.1.gz >> >>>> >> >>>> nmbd >> >>>> >> >>>> /usr/local/samba/sbin/nmbd >> >>>> /usr/local/samba/share/man/man8/nmbd.8 >> >>>> /var/log/upstart/nmbd.log.1.gz >> >>>> >> >>>> samba >> >>>> >> >>>> /usr/local/samba/sbin/nmbd >> >>>> /usr/local/samba/sbin/samba >> >>>> /usr/local/samba/sbin/samba_dnsupdate >> >>>> /usr/local/samba/sbin/samba_kcc >> >>>> /usr/local/samba/sbin/samba_spnupdate >> >>>> /usr/local/samba/sbin/samba_upgradedns >> >>>> /usr/local/samba/sbin/smbd >> >>>> /usr/local/samba/sbin/winbind >> >>>> >> >>>> >> >>>> If I'm overlooking something obvious please let me know. >> >>>> >> >>>> JS >> >>>> >> >>>> On Wed, Mar 2, 2016 at 5:42 PM, Marc Muehlfeld <mmuehlfeld at samba.org >> > >> >>>> wrote: >> >>>> >> >>>> Am 02.03.2016 um 18:52 schrieb IT Admin: >> >>>> >> >>>>> >> >>>>> Samba is compiled from source. >> >>>>>> Samba DB is stored on local RAID array. >> >>>>>> Changes to AD are done using ADUC from a Windows 7 box. >> >>>>>> AD is used for authentication, user shares (folder redirection), >> and >> >>>>>> >> >>>>>> shared >> >>>>> >> >>>>> folders. >> >>>>>> >> >>>>>> >> >>>>> Can you add >> >>>>> log level = 10 >> >>>>> to your smb.conf, empty your log directory and start Samba. It >> should >> >>>>> generate a new log, that captures all output. Then put it to >> >>>>> cpaste.org >> >>>>> or some other paste service and share the link with us. Maybe we see >> >>>>> something interesting. >> >>>>> >> >>>>> >> >>>>> Does something changed when the problem occured the first time? Some >> >>>>> package updates, crashes, etc.? >> >>>>> >> >>>>> >> >>>>> Can you make sure that no kind of Samba package (daemon, libs, >> etc.) is >> >>>>> installed on the system? Maybe your selfcompiled version overwrites >> >>>>> some >> >>>>> stuff and your OS installed an update, that mixes now with the self >> >>>>> compiled version. Just a guess. >> >>>>> >> >>>>> Regards, >> >>>>> Marc >> >>>>> >> >
Possibly Parallel Threads
- Samba AD/DC crashed again, third time in as many months
- Samba AD/DC crashed again, third time in as many months
- Samba AD/DC crashed again, third time in as many months
- Samba AD/DC crashed again, third time in as many months
- Samba AD/DC crashed again, third time in as many months