samba - Mar 2024 - How to diagnose a busy LDAP server process in the Samba AD DC

Thanks for getting back to me.
It seems to me the LDAP process being busy would be the root cause
here.  Working out what is going on here shouldn't is a detective task
- I always start with a wireshark trace.  The client making all the
noise/traffic will be the one causing the trouble.
If it isn't clear from that, then look into the DB audit logging for
perhaps busy writes
https://wiki.samba.org/index.php/Setting_up_Audit_Logging#Enabling_AD_DC_Database_Audit_Logging
Finally, set 'log level = 5' and look for logs like: LDAP Query:
Duration was 
This will tell you about how long each query is taking, potentially
showing a particularly slow query that needs to be stopped.
Andrew Bartlett
On Sun, 2024-03-10 at 19:46 -0300, Elias Pereira wrote:
> > Is the drepl local processes very busy doing inbound replication?
> 
> How can I check this?
> 
> > My instinct is either the server is very busy (and this should show
> > up in CPU use) or a transaction is being held open excessively.
>  
> I use VMs on Proxmox. In DC1, I installed the Proxmox agent, and CPU
> usage via the dashboard is very low. However, when I checked using
> 'top,' the LDAP process is consuming around 94/96% of the CPU. Very
> strange.
It is probably 94% of a single CPU, but you might have 8 CPUs in the
VM, so overall use is low. 
> The VM has 4 CPUs and 6GB of memory.
> 
> 
> 
> On Sun, Mar 10, 2024 at 5:55?PM Andrew Bartlett <abartlet at
samba.org>
> wrote:
> > Either the local server is busy, or possibly (but it would not
> > explain the samba_kcc) Samba's drepl process is stuck talking to a
> > remote server.
> > 
> > Is the drepl local processes very busy doing inbound replication?  
> > 
> > My instinct is either the server is very busy (and this should show
> > up in CPU use) or a transaction is being held open excessively. 
> > 
> > Andrew Bartlett
> > 
> > On Sat, 2024-03-09 at 19:11 -0300, Elias Pereira via samba wrote:
> > > I've been grappling with a recurring set of errors for quite
some
> > > time now:- UpdateRefs failed with NT_STATUS_IO_TIMEOUT- Failed
> > > samba_kcc - NT_STATUS_IO_TIMEOUT- IRPC callback failed for
> > > DsReplicaSync - NT_STATUS_IO_TIMEOUT
> > > Despite cranking up the log level to 10, the returned information
> > > remainsfrustratingly cryptic and hard to decipher.
> > > This error, being overly generic, continues to elude
> > > identification evenwiththe heightened log verbosity. The
> > > challenge lies in tracing its origin.
> > > Running samba-tool dbcheck doesn't reveal any problems, yet
> > > executing thecommand while monitoring the Samba log with
"tail
> > > -f" exposes errorsidenticalto those described above.
> > > Interestingly, samba-tool drs showrepl doesn't report any
errors.
> > > So, what additional steps can be taken to unearth the root
> > > causeof these persistent NT_STATUS_IO_TIMEOUT errors?
> > > 
> > > On Fri, Mar 1, 2024 at 10:32?PM Elias Pereira <empbilly at
gmail.com
> > > > wrote:
> > > > There is probably nothing wrong with your log, but Firefox
> > > > doesn't
> > > > > like it, it thinks it contains a virus.
> > > > 
> > > > I just saw now that your response ended up in spam, probably
> > > > because ofthe link with the log. O.o
> > > > I still receive the error in the
> > > > logs:source4/dsdb/kcc/kcc_periodic.c:790: Failed samba_kcc
> > > > -NT_STATUS_IO_TIMEOUT
> > > > The strangest thing is that it occurs when the command is
> > > > executed:samba-tool dbcheck --cross-ncs --fix --yes
> > > > Could it be some object causing this error?
> > > > On Mon, Feb 12, 2024 at 4:40?PM Rowland Penny via samba <
> > > > samba at lists.samba.org> wrote:
> > > > > On Mon, 12 Feb 2024 16:20:27 -0300Elias Pereira via
samba <
> > > > > samba at lists.samba.org> wrote:
> > > > > > hi,
> > > > > > My saga continues...
> > > > > > I've configured the audit log for drs_repl in
smb.conf, and
> > > > > > below isthe log generated.
> > > > > > https://transfer.sh/7fen4qCNIQ/drs_repl.log
> > > > > > 
> > > > > > The log level was
5.drs_repl:5@/var/log/samba/drs_repl.log
> > > > > > Could someone take a look and help me understand
the log?
> > > > > 
> > > > > There is probably nothing wrong with your log, but
Firefox
> > > > > doesn'tlike it, it thinks it contains a virus.
> > > > > Rowland
> > > > > 
> > > > > 
> > > > > --To unsubscribe from this list go to the following URL
and
> > > > > read theinstructions:  
> > > > > https://lists.samba.org/mailman/options/samba
> > > > > 
> > > > 
> > > > --Elias Pereira
> > > 
> > > -- Elias Pereira
> > -- 
> > Andrew Bartlett (he/him)       https://samba.org/~abartlet/
> > Samba Team Member (since 2001) https://samba.org
> > Samba Team Lead                
> > https://catalyst.net.nz/services/samba
> > Catalyst.Net Ltd
> > 
> > 
> > Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group
> > company
> > 
> > Samba Development and Support: 
> > https://catalyst.net.nz/services/samba
> > 
> > Catalyst IT - Expert Open Source Solutions
> > 
> > 
> > 
> 
> -- 
> Elias Pereira
-- 
Andrew Bartlett (he/him)       https://samba.org/~abartlet/
Samba Team Member (since 2001) https://samba.org
Samba Team Lead                https://catalyst.net.nz/services/samba
Catalyst.Net Ltd


Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group
company

Samba Development and Support: https://catalyst.net.nz/services/samba

Catalyst IT - Expert Open Source Solutions

hi Andrew, thanks for the help!!!

It seems to me the LDAP process being busy would be the root cause
here.
> Working out what is going on here shouldn't is a detective task - I
always
> start with a wireshark trace.  The client making all the noise/traffic will
> be the one causing the trouble.

In the wireshark analysis, should I filter only by the ldap protocol or
leave everything? Should I look at something specific in the client logs?

On Sun, Mar 10, 2024 at 9:31?PM Andrew Bartlett <abartlet at samba.org>
wrote:
> Thanks for getting back to me.
>
> It seems to me the LDAP process being busy would be the root cause here.
> Working out what is going on here shouldn't is a detective task - I
always
> start with a wireshark trace.  The client making all the noise/traffic will
> be the one causing the trouble.
>
> If it isn't clear from that, then look into the DB audit logging for
> perhaps busy writes
>
>
>
https://wiki.samba.org/index.php/Setting_up_Audit_Logging#Enabling_AD_DC_Database_Audit_Logging
>
> Finally, set 'log level = 5' and look for logs like: LDAP Query:
Duration
> was
>
> This will tell you about how long each query is taking, potentially
> showing a particularly slow query that needs to be stopped.
>
> Andrew Bartlett
>
> On Sun, 2024-03-10 at 19:46 -0300, Elias Pereira wrote:
>
> Is the drepl local processes very busy doing inbound replication?
>
>
> How can I check this?
>
> My instinct is either the server is very busy (and this should show up in
> CPU use) or a transaction is being held open excessively.
>
>
> I use VMs on Proxmox. In DC1, I installed the Proxmox agent, and CPU usage
> via the dashboard is very low. However, when I checked using 'top,'
the
> LDAP process is consuming around 94/96% of the CPU. Very strange.
>
>
> It is probably 94% of a single CPU, but you might have 8 CPUs in the VM,
> so overall use is low.
>
> The VM has 4 CPUs and 6GB of memory.
>
>
>
> On Sun, Mar 10, 2024 at 5:55?PM Andrew Bartlett <abartlet at
samba.org>
> wrote:
>
> Either the local server is busy, or possibly (but it would not explain the
> samba_kcc) Samba's drepl process is stuck talking to a remote server.
>
> Is the drepl local processes very busy doing inbound replication?
>
> My instinct is either the server is very busy (and this should show up in
> CPU use) or a transaction is being held open excessively.
>
> Andrew Bartlett
>
> On Sat, 2024-03-09 at 19:11 -0300, Elias Pereira via samba wrote:
>
> I've been grappling with a recurring set of errors for quite some time
now:
>
> - UpdateRefs failed with NT_STATUS_IO_TIMEOUT
>
> - Failed samba_kcc - NT_STATUS_IO_TIMEOUT
>
> - IRPC callback failed for DsReplicaSync - NT_STATUS_IO_TIMEOUT
>
>
> Despite cranking up the log level to 10, the returned information remains
>
> frustratingly cryptic and hard to decipher.
>
>
> This error, being overly generic, continues to elude identification even
>
> with
>
> the heightened log verbosity. The challenge lies in tracing its origin.
>
>
> Running samba-tool dbcheck doesn't reveal any problems, yet executing
the
>
> command while monitoring the Samba log with "tail -f" exposes
errors
>
> identical
>
> to those described above.
>
>
> Interestingly, samba-tool drs showrepl doesn't report any errors.
>
>
> So, what additional steps can be taken to unearth the root cause
>
> of these persistent NT_STATUS_IO_TIMEOUT errors?
>
>
>
> On Fri, Mar 1, 2024 at 10:32?PM Elias Pereira <
>
> empbilly at gmail.com
>
> > wrote:
>
>
> There is probably nothing wrong with your log, but Firefox doesn't
>
> like it, it thinks it contains a virus.
>
>
>
> I just saw now that your response ended up in spam, probably because of
>
> the link with the log. O.o
>
>
> I still receive the error in the logs:
>
> source4/dsdb/kcc/kcc_periodic.c:790: Failed samba_kcc -
>
> NT_STATUS_IO_TIMEOUT
>
>
> The strangest thing is that it occurs when the command is executed:
>
> samba-tool dbcheck --cross-ncs --fix --yes
>
>
> Could it be some object causing this error?
>
>
> On Mon, Feb 12, 2024 at 4:40?PM Rowland Penny via samba <
>
> samba at lists.samba.org
>
> > wrote:
>
>
> On Mon, 12 Feb 2024 16:20:27 -0300
>
> Elias Pereira via samba <
>
> samba at lists.samba.org
>
> > wrote:
>
>
> hi,
>
>
> My saga continues...
>
>
> I've configured the audit log for drs_repl in smb.conf, and below is
>
> the log generated.
>
> https://transfer.sh/7fen4qCNIQ/drs_repl.log
>
>
>
> The log level was 5.
>
> drs_repl:5@/var/log/samba/drs_repl.log
>
>
> Could someone take a look and help me understand the log?
>
>
>
> There is probably nothing wrong with your log, but Firefox doesn't
>
> like it, it thinks it contains a virus.
>
>
> Rowland
>
>
>
>
> --
>
> To unsubscribe from this list go to the following URL and read the
>
> instructions:
>
> https://lists.samba.org/mailman/options/samba
>
>
>
>
>
> --
>
> Elias Pereira
>
>
>
>
> --
>
> Elias Pereira
>
> --
>
>
> Andrew Bartlett (he/him)       https://samba.org/~abartlet/
> Samba Team Member (since 2001) https://samba.org
> Samba Team Lead                https://catalyst.net.nz/services/samba
> Catalyst.Net Ltd
>
> Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group company
>
> Samba Development and Support: https://catalyst.net.nz/services/samba
>
> Catalyst IT - Expert Open Source Solutions
>
>
>
>
> --
> Elias Pereira
>
> --
>
> Andrew Bartlett (he/him)       https://samba.org/~abartlet/
> Samba Team Member (since 2001) https://samba.org
> Samba Team Lead                https://catalyst.net.nz/services/samba
> Catalyst.Net Ltd
>
> Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group company
>
> Samba Development and Support: https://catalyst.net.nz/services/samba
>
> Catalyst IT - Expert Open Source Solutions
>
>
>
-- 
Elias Pereira