samba - Sep 2016 - drs showrepl - Failed to bind to UUID - Undetermined error

Hi Guys,

I have now updated to 4.5.0 - thank you to all the team for your efforts on
this :)

I was excited to read in the release notes that there were many replication
improvements, and I have run 'samba-tool dbcheck --cross-ncs --fix' on
all
my DCs; there were many, many replPropertyMetaData and other errors which
have now been found and fixed - thanks!

However, I think something still isn't right in my domain; this is probably
not the fault of 4.5.0 but rather an inconsistency caused when one of my
DCs died and was rebuilt - however I'm now not sure where to look
(presumably with ADSIEdit / ldbsearch) to check which object I need to
remove / update.

The symptom I can see is that running 'samba-tool drs showrepl' fails on
one of my DCs, but works on the other two. On the failing DC I get the
message:

user at dc2:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
Failed to bind to uuid aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee for
ncacn_ip_tcp:1.2.3.4[1024,seal,target_hostname=dc2.mydomain.org.uk
,abstract_syntax=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee/0x00000004,

FUL
ERROR(<class 'samba.drs_utils.drsException'>): DRS connection to
dc2.mydomain.org.uk failed - drsException: DRS connection to
dc2.mydomain.org.uk failed: (-1073741823, 'Undetermined error')
  File
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/drs.py",
line 41, in drsuapi_connect
    (ctx.drsuapi, ctx.drsuapi_handle, ctx.bind_supported_extensions)
drs_utils.drsuapi_connect(ctx.server, ctx.lp, ctx.creds)
  File
"/usr/local/samba/lib/python2.7/site-packages/samba/drs_utils.py",
line 54, in drsuapi_connect
    raise drsException("DRS connection to %s failed: %s" % (server,
e))


Replication of objects between DCs does seem to work fine (at least,
changing the description on a test user object on any DC did propagate
between all 3 DCs) so I don't think the basic mechanism is broken.. but I
suspect whatever objects 'samba-tool drs showrepl' looks at aren't
quite
right.

I don't understand why this only fails on one DC, though - all three are
built pretty much identically, so I would have expected this to work or not
equally across all three.

Where should I be looking in AD? The inter-site links seem to be defined OK
from what I can tell, but I don't know much about the internals of these
beyond looking in AD Sites & Services and things "look OK" there.

Any pointers would be much appreciated, I'll do some digging from there.

Thanks!

Jonathan



On an unrelated note, on DC3, 'samba-tool drs showrepl' does work, but
shows the following warnings/errors, before then working fine and showing
the usual output that I would expect to see. Should I file a bug for this -
can anyone else reproduce it?

user at dc3:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
Failed to connect host 127.0.1.1 on port 135 - NT_STATUS_CONNECTION_REFUSED
Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 135 -
NT_STATUS_CONNECTION_REFUSED.
Failed to connect host 127.0.1.1 on port 1024 - NT_STATUS_CONNECTION_REFUSED
Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 1024 -
NT_STATUS_CONNECTION_REFUSED.
mysite\DC3
[...]


I think I have tracked this one down to the following smb.conf items that
were present on this machine (and which I have now removed):

       bind interfaces only = yes
       interfaces = eth0 lo
The 'lo' interface has the IP 127.0.0.1, but for some reason
'samba-tool
drs showrepl' is trying to connect to 127.0.1.1 - which then fails.

-- 
"If we knew what it was we were doing, it would not be called research,
would it?"
      - Albert Einstein

Quick, befor i get my beer here..  
>> Failed to connect host 127.0.1.1 
Check you hosts file for this one, you got 127.0.1.1 there from an install with
dhcp.


Greetz, 

Louis


> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Jonathan Hunter
> via samba
> Verzonden: vrijdag 9 september 2016 16:25
> Aan: samba
> Onderwerp: [Samba] drs showrepl - Failed to bind to UUID - Undetermined
> error
> 
> Hi Guys,
> 
> I have now updated to 4.5.0 - thank you to all the team for your efforts
> on
> this :)
> 
> I was excited to read in the release notes that there were many
> replication
> improvements, and I have run 'samba-tool dbcheck --cross-ncs --fix'
on all
> my DCs; there were many, many replPropertyMetaData and other errors which
> have now been found and fixed - thanks!
> 
> However, I think something still isn't right in my domain; this is
> probably
> not the fault of 4.5.0 but rather an inconsistency caused when one of my
> DCs died and was rebuilt - however I'm now not sure where to look
> (presumably with ADSIEdit / ldbsearch) to check which object I need to
> remove / update.
> 
> The symptom I can see is that running 'samba-tool drs showrepl'
fails on
> one of my DCs, but works on the other two. On the failing DC I get the
> message:
> 
> user at dc2:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
> Failed to bind to uuid aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee for
> ncacn_ip_tcp:1.2.3.4[1024,seal,target_hostname=dc2.mydomain.org.uk
> ,abstract_syntax=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee/0x00000004,
> 
> FUL
> ERROR(<class 'samba.drs_utils.drsException'>): DRS connection
to
> dc2.mydomain.org.uk failed - drsException: DRS connection to
> dc2.mydomain.org.uk failed: (-1073741823, 'Undetermined error')
>   File
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/drs.py",
> line 41, in drsuapi_connect
>     (ctx.drsuapi, ctx.drsuapi_handle, ctx.bind_supported_extensions) >
drs_utils.drsuapi_connect(ctx.server, ctx.lp, ctx.creds)
>   File
"/usr/local/samba/lib/python2.7/site-packages/samba/drs_utils.py",
> line 54, in drsuapi_connect
>     raise drsException("DRS connection to %s failed: %s" %
(server, e))
> 
> 
> Replication of objects between DCs does seem to work fine (at least,
> changing the description on a test user object on any DC did propagate
> between all 3 DCs) so I don't think the basic mechanism is broken.. but
I
> suspect whatever objects 'samba-tool drs showrepl' looks at
aren't quite
> right.
> 
> I don't understand why this only fails on one DC, though - all three
are
> built pretty much identically, so I would have expected this to work or
> not
> equally across all three.
> 
> Where should I be looking in AD? The inter-site links seem to be defined
> OK
> from what I can tell, but I don't know much about the internals of
these
> beyond looking in AD Sites & Services and things "look OK"
there.
> 
> Any pointers would be much appreciated, I'll do some digging from
there.
> 
> Thanks!
> 
> Jonathan
> 
> 
> 
> On an unrelated note, on DC3, 'samba-tool drs showrepl' does work,
but
> shows the following warnings/errors, before then working fine and showing
> the usual output that I would expect to see. Should I file a bug for this
> -
> can anyone else reproduce it?
> 
> user at dc3:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
> Failed to connect host 127.0.1.1 on port 135 -
> NT_STATUS_CONNECTION_REFUSED
> Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 135 -
> NT_STATUS_CONNECTION_REFUSED.
> Failed to connect host 127.0.1.1 on port 1024 -
> NT_STATUS_CONNECTION_REFUSED
> Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 1024 -
> NT_STATUS_CONNECTION_REFUSED.
> mysite\DC3
> [...]
> 
> 
> I think I have tracked this one down to the following smb.conf items that
> were present on this machine (and which I have now removed):
> 
>        bind interfaces only = yes
>        interfaces = eth0 lo
> The 'lo' interface has the IP 127.0.0.1, but for some reason
'samba-tool
> drs showrepl' is trying to connect to 127.0.1.1 - which then fails.
> 
> --
> "If we knew what it was we were doing, it would not be called
research,
> would it?"
>       - Albert Einstein
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

On Fri, 9 Sep 2016 15:24:40 +0100
Jonathan Hunter via samba <samba at lists.samba.org> wrote:
> Hi Guys,
> 
> I have now updated to 4.5.0 - thank you to all the team for your
> efforts on this :)
> 
> I was excited to read in the release notes that there were many
> replication improvements, and I have run 'samba-tool dbcheck
> --cross-ncs --fix' on all my DCs; there were many, many
> replPropertyMetaData and other errors which have now been found and
> fixed - thanks!
> 
> However, I think something still isn't right in my domain; this is
> probably not the fault of 4.5.0 but rather an inconsistency caused
> when one of my DCs died and was rebuilt - however I'm now not sure
> where to look (presumably with ADSIEdit / ldbsearch) to check which
> object I need to remove / update.
> 
> The symptom I can see is that running 'samba-tool drs showrepl'
fails
> on one of my DCs, but works on the other two. On the failing DC I get
> the message:
> 
> user at dc2:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
> Failed to bind to uuid aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee for
> ncacn_ip_tcp:1.2.3.4[1024,seal,target_hostname=dc2.mydomain.org.uk
> ,abstract_syntax=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee/0x00000004,
> 
> FUL
> ERROR(<class 'samba.drs_utils.drsException'>): DRS connection
to
> dc2.mydomain.org.uk failed - drsException: DRS connection to
> dc2.mydomain.org.uk failed: (-1073741823, 'Undetermined error')
>   File
>
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/drs.py",
> line 41, in drsuapi_connect (ctx.drsuapi, ctx.drsuapi_handle,
> ctx.bind_supported_extensions) > drs_utils.drsuapi_connect(ctx.server,
ctx.lp, ctx.creds) File
>
"/usr/local/samba/lib/python2.7/site-packages/samba/drs_utils.py",
> line 54, in drsuapi_connect raise drsException("DRS connection to %s
> failed: %s" % (server, e))
> 
> 
> Replication of objects between DCs does seem to work fine (at least,
> changing the description on a test user object on any DC did propagate
> between all 3 DCs) so I don't think the basic mechanism is broken..
> but I suspect whatever objects 'samba-tool drs showrepl' looks at
> aren't quite right.
> 
> I don't understand why this only fails on one DC, though - all three
> are built pretty much identically, so I would have expected this to
> work or not equally across all three.
> 
> Where should I be looking in AD? The inter-site links seem to be
> defined OK from what I can tell, but I don't know much about the
> internals of these beyond looking in AD Sites & Services and things
> "look OK" there.
> 
> Any pointers would be much appreciated, I'll do some digging from
> there.
> 
> Thanks!
> 
> Jonathan
> 
> 
> 
> On an unrelated note, on DC3, 'samba-tool drs showrepl' does work,
but
> shows the following warnings/errors, before then working fine and
> showing the usual output that I would expect to see. Should I file a
> bug for this - can anyone else reproduce it?
> 
> user at dc3:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
> Failed to connect host 127.0.1.1 on port 135 -
> NT_STATUS_CONNECTION_REFUSED Failed to connect host 127.0.1.1
> (dc3.mydomain.org.uk) on port 135 - NT_STATUS_CONNECTION_REFUSED.
> Failed to connect host 127.0.1.1 on port 1024 -
> NT_STATUS_CONNECTION_REFUSED Failed to connect host 127.0.1.1
> (dc3.mydomain.org.uk) on port 1024 - NT_STATUS_CONNECTION_REFUSED.
> mysite\DC3
> [...]
> 
> 
> I think I have tracked this one down to the following smb.conf items
> that were present on this machine (and which I have now removed):
> 
>        bind interfaces only = yes
>        interfaces = eth0 lo
> The 'lo' interface has the IP 127.0.0.1, but for some reason
> 'samba-tool drs showrepl' is trying to connect to 127.0.1.1 - which
> then fails.
> 
Is this in ubuntu ?
check /etc/hosts and remove any line starting with 127.0.1.1 (or
comment it out)
If networkmanager is running, stop it using dnsmasq.
This is what usually cause the ptoblem you are having.

Rowland

Thank you Louis and Rowland!

I feel a little silly, having not checked the "obvious" place
/etc/hosts -
but in my defence I have not used debian anything like as much as I have
used Slackware, RedHat and CentOS.. I'd never come across this behaviour at
all.

These DCs are running on Raspberry Pis which are Debian based. Clearly they
have static IPs now but apparently yes, some cruft from the originally
as-shipped DHCP config or whatever was still present.

I can confirm after I update /etc/hosts to 127.0.0.1 (from 127.0.1.1) that
that part works perfectly again on dc3. (I have still removed the lines
from smb.conf, as I don't need them - but it does now work with them in)

Now, I just need to find out why "samba-tool drs showrepl" doesn't
work on
one of my other DCs, that's the real mystery :) Editing /etc/hosts on dc3
doesn't help dc2 to work, I still get the "Failed to bind to uuid"
error
when running "samba-tool drs showrepl" on dc2.

On 9 September 2016 at 16:01, L.P.H. van Belle via samba <
samba at lists.samba.org> wrote:
> Quick, befor i get my beer here..
>
> >> Failed to connect host 127.0.1.1
> Check you hosts file for this one, you got 127.0.1.1 there from an install
> with dhcp.
>
>
> Greetz,
>
> Louis
>
>
>
> > -----Oorspronkelijk bericht-----
> > Van: samba [mailto:samba-bounces at lists.samba.org] Namens Jonathan
Hunter
> > via samba
> > Verzonden: vrijdag 9 september 2016 16:25
> > Aan: samba
> > Onderwerp: [Samba] drs showrepl - Failed to bind to UUID -
Undetermined
> > error
> >
> > Hi Guys,
> >
> > I have now updated to 4.5.0 - thank you to all the team for your
efforts
> > on
> > this :)
> >
> > I was excited to read in the release notes that there were many
> > replication
> > improvements, and I have run 'samba-tool dbcheck --cross-ncs
--fix' on
> all
> > my DCs; there were many, many replPropertyMetaData and other errors
which
> > have now been found and fixed - thanks!
> >
> > However, I think something still isn't right in my domain; this is
> > probably
> > not the fault of 4.5.0 but rather an inconsistency caused when one of
my
> > DCs died and was rebuilt - however I'm now not sure where to look
> > (presumably with ADSIEdit / ldbsearch) to check which object I need to
> > remove / update.
> >
> > The symptom I can see is that running 'samba-tool drs
showrepl' fails on
> > one of my DCs, but works on the other two. On the failing DC I get the
> > message:
> >
> > user at dc2:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
> > Failed to bind to uuid aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee for
> > ncacn_ip_tcp:1.2.3.4[1024,seal,target_hostname=dc2.mydomain.org.uk
> > ,abstract_syntax=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee/0x00000004,
> >
> > FUL
> > ERROR(<class 'samba.drs_utils.drsException'>): DRS
connection to
> > dc2.mydomain.org.uk failed - drsException: DRS connection to
> > dc2.mydomain.org.uk failed: (-1073741823, 'Undetermined
error')
> >   File
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/
> drs.py",
> > line 41, in drsuapi_connect
> >     (ctx.drsuapi, ctx.drsuapi_handle, ctx.bind_supported_extensions)
> > drs_utils.drsuapi_connect(ctx.server, ctx.lp, ctx.creds)
> >   File "/usr/local/samba/lib/python2.7/site-packages/samba/drs_
> utils.py",
> > line 54, in drsuapi_connect
> >     raise drsException("DRS connection to %s failed: %s" %
(server, e))
> >
> >
> > Replication of objects between DCs does seem to work fine (at least,
> > changing the description on a test user object on any DC did propagate
> > between all 3 DCs) so I don't think the basic mechanism is
broken.. but I
> > suspect whatever objects 'samba-tool drs showrepl' looks at
aren't quite
> > right.
> >
> > I don't understand why this only fails on one DC, though - all
three are
> > built pretty much identically, so I would have expected this to work
or
> > not
> > equally across all three.
> >
> > Where should I be looking in AD? The inter-site links seem to be
defined
> > OK
> > from what I can tell, but I don't know much about the internals of
these
> > beyond looking in AD Sites & Services and things "look
OK" there.
> >
> > Any pointers would be much appreciated, I'll do some digging from
there.
> >
> > Thanks!
> >
> > Jonathan
> >
> >
> >
> > On an unrelated note, on DC3, 'samba-tool drs showrepl' does
work, but
> > shows the following warnings/errors, before then working fine and
showing
> > the usual output that I would expect to see. Should I file a bug for
this
> > -
> > can anyone else reproduce it?
> >
> > user at dc3:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
> > Failed to connect host 127.0.1.1 on port 135 -
> > NT_STATUS_CONNECTION_REFUSED
> > Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 135 -
> > NT_STATUS_CONNECTION_REFUSED.
> > Failed to connect host 127.0.1.1 on port 1024 -
> > NT_STATUS_CONNECTION_REFUSED
> > Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 1024 -
> > NT_STATUS_CONNECTION_REFUSED.
> > mysite\DC3
> > [...]
> >
> >
> > I think I have tracked this one down to the following smb.conf items
that
> > were present on this machine (and which I have now removed):
> >
> >        bind interfaces only = yes
> >        interfaces = eth0 lo
> > The 'lo' interface has the IP 127.0.0.1, but for some reason
'samba-tool
> > drs showrepl' is trying to connect to 127.0.1.1 - which then
fails.
> >
> > --
> > "If we knew what it was we were doing, it would not be called
research,
> > would it?"
> >       - Albert Einstein
> > --
> > To unsubscribe from this list go to the following URL and read the
> > instructions:  https://lists.samba.org/mailman/options/samba
>
>
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>


-- 
"If we knew what it was we were doing, it would not be called research,
would it?"
      - Albert Einstein

On Fri, 2016-09-09 at 15:24 +0100, Jonathan Hunter via samba
wrote:
> Hi Guys,
> 
> I have now updated to 4.5.0 - thank you to all the team for your
> efforts on
> this :)
> 
> I was excited to read in the release notes that there were many
> replication
> improvements, and I have run 'samba-tool dbcheck --cross-ncs --fix'
> on all
> my DCs; there were many, many replPropertyMetaData and other errors
> which
> have now been found and fixed - thanks!
> 
> However, I think something still isn't right in my domain; this is
> probably
> not the fault of 4.5.0 but rather an inconsistency caused when one of
> my
> DCs died and was rebuilt - however I'm now not sure where to look
> (presumably with ADSIEdit / ldbsearch) to check which object I need
> to
> remove / update.
It looks like others have solved your issue, but just checking on the
broader issue of removing servers.  Is the UUID for the removed server,
and if so how did you remove the DC that died?

We now have 'samba-tool domain demote --remove-other-dead-server' that
will do a more comprehensive job cleaning out the old DC.

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Thanks Andrew.

No - it was my fault for including an easily-solved side query in the same
email as the main query.. :) I haven't solved the original issue, which is
that 'samba-tool drs showrepl' runs on two of my DCs but not on the
third.

I don't know if anything else also doesn't work, e.g. some aspect of
replication I haven't observed yet - but the only problem I can actually
see is that 'samba-tool drs showrepl' doesn't run on this one DC.

You ask a good question in terms of removing the DC that died. I think I
probably did not do this step correctly. I had two DCs die within a short
time of each other (disk issues) and I built new machines and simply joined
them to the domain 'over the top', using the same name and IP address as
previously. I now realise that this might not have been the best idea, as
they would now have new UUIDs and I have done nothing much to remove the
old UUIDs, apart from removing them from DNS/LDAP where I found them.
Perhaps I should have explicitly removed the DCs, before re-adding them? I
may well not have removed them fully myself.

Is there an easy place in AD where these UUIDs are stored - I'm happy to go
through and remove stale entries myself using ADSIEdit or similar? Or would
you recommend I temporarily remove each DC in turn using the demote tool,
then re-add? (Would the demote tool remove *all* UUIDs from the DCs, or
only the first one?)

Is there some form of AD-checker tool, perhaps (either MS or Samba) that
would check all the various LDAP entries, DNS entries (_msdcs, _sites,
_tcp, _kerberos etc.) and point out what I have wrong? :-)

At the moment I guess there might be multiple UUIDs somewhere in the
directory for this one DC, which might be why 'samba-tool drs showrepl'
chokes. There may well be multiple UUIDs for my other server that died,
too, but perhaps the first one that is returned from LDAP for that other
server is the current one, which is why 'samba-tool drs showrepl' works
on
that?

Many thanks,

Jonathan

On 9 September 2016 at 21:01, Andrew Bartlett <abartlet at samba.org>
wrote:
> On Fri, 2016-09-09 at 15:24 +0100, Jonathan Hunter via samba wrote:
> > Hi Guys,
> >
> > I have now updated to 4.5.0 - thank you to all the team for your
> > efforts on
> > this :)
> >
> > I was excited to read in the release notes that there were many
> > replication
> > improvements, and I have run 'samba-tool dbcheck --cross-ncs
--fix'
> > on all
> > my DCs; there were many, many replPropertyMetaData and other errors
> > which
> > have now been found and fixed - thanks!
> >
> > However, I think something still isn't right in my domain; this is
> > probably
> > not the fault of 4.5.0 but rather an inconsistency caused when one of
> > my
> > DCs died and was rebuilt - however I'm now not sure where to look
> > (presumably with ADSIEdit / ldbsearch) to check which object I need
> > to
> > remove / update.
>
> It looks like others have solved your issue, but just checking on the
> broader issue of removing servers.  Is the UUID for the removed server,
> and if so how did you remove the DC that died?
>
> We now have 'samba-tool domain demote --remove-other-dead-server'
that
> will do a more comprehensive job cleaning out the old DC.
>
> Thanks,
>
> Andrew Bartlett
> --
> Andrew Bartlett                       http://samba.org/~abartlet/
> Authentication Developer, Samba Team  http://samba.org
> Samba Developer, Catalyst IT          http://catalyst.net.nz/
> services/samba
>
>

-- 
"If we knew what it was we were doing, it would not be called research,
would it?"
      - Albert Einstein