samba - Mar 2024 - How to diagnose a busy LDAP server process in the Samba AD DC

hi Andrew, thanks for the help!!!

It seems to me the LDAP process being busy would be the root cause
here.
> Working out what is going on here shouldn't is a detective task - I
always
> start with a wireshark trace.  The client making all the noise/traffic will
> be the one causing the trouble.

In the wireshark analysis, should I filter only by the ldap protocol or
leave everything? Should I look at something specific in the client logs?

On Sun, Mar 10, 2024 at 9:31?PM Andrew Bartlett <abartlet at samba.org>
wrote:
> Thanks for getting back to me.
>
> It seems to me the LDAP process being busy would be the root cause here.
> Working out what is going on here shouldn't is a detective task - I
always
> start with a wireshark trace.  The client making all the noise/traffic will
> be the one causing the trouble.
>
> If it isn't clear from that, then look into the DB audit logging for
> perhaps busy writes
>
>
>
https://wiki.samba.org/index.php/Setting_up_Audit_Logging#Enabling_AD_DC_Database_Audit_Logging
>
> Finally, set 'log level = 5' and look for logs like: LDAP Query:
Duration
> was
>
> This will tell you about how long each query is taking, potentially
> showing a particularly slow query that needs to be stopped.
>
> Andrew Bartlett
>
> On Sun, 2024-03-10 at 19:46 -0300, Elias Pereira wrote:
>
> Is the drepl local processes very busy doing inbound replication?
>
>
> How can I check this?
>
> My instinct is either the server is very busy (and this should show up in
> CPU use) or a transaction is being held open excessively.
>
>
> I use VMs on Proxmox. In DC1, I installed the Proxmox agent, and CPU usage
> via the dashboard is very low. However, when I checked using 'top,'
the
> LDAP process is consuming around 94/96% of the CPU. Very strange.
>
>
> It is probably 94% of a single CPU, but you might have 8 CPUs in the VM,
> so overall use is low.
>
> The VM has 4 CPUs and 6GB of memory.
>
>
>
> On Sun, Mar 10, 2024 at 5:55?PM Andrew Bartlett <abartlet at
samba.org>
> wrote:
>
> Either the local server is busy, or possibly (but it would not explain the
> samba_kcc) Samba's drepl process is stuck talking to a remote server.
>
> Is the drepl local processes very busy doing inbound replication?
>
> My instinct is either the server is very busy (and this should show up in
> CPU use) or a transaction is being held open excessively.
>
> Andrew Bartlett
>
> On Sat, 2024-03-09 at 19:11 -0300, Elias Pereira via samba wrote:
>
> I've been grappling with a recurring set of errors for quite some time
now:
>
> - UpdateRefs failed with NT_STATUS_IO_TIMEOUT
>
> - Failed samba_kcc - NT_STATUS_IO_TIMEOUT
>
> - IRPC callback failed for DsReplicaSync - NT_STATUS_IO_TIMEOUT
>
>
> Despite cranking up the log level to 10, the returned information remains
>
> frustratingly cryptic and hard to decipher.
>
>
> This error, being overly generic, continues to elude identification even
>
> with
>
> the heightened log verbosity. The challenge lies in tracing its origin.
>
>
> Running samba-tool dbcheck doesn't reveal any problems, yet executing
the
>
> command while monitoring the Samba log with "tail -f" exposes
errors
>
> identical
>
> to those described above.
>
>
> Interestingly, samba-tool drs showrepl doesn't report any errors.
>
>
> So, what additional steps can be taken to unearth the root cause
>
> of these persistent NT_STATUS_IO_TIMEOUT errors?
>
>
>
> On Fri, Mar 1, 2024 at 10:32?PM Elias Pereira <
>
> empbilly at gmail.com
>
> > wrote:
>
>
> There is probably nothing wrong with your log, but Firefox doesn't
>
> like it, it thinks it contains a virus.
>
>
>
> I just saw now that your response ended up in spam, probably because of
>
> the link with the log. O.o
>
>
> I still receive the error in the logs:
>
> source4/dsdb/kcc/kcc_periodic.c:790: Failed samba_kcc -
>
> NT_STATUS_IO_TIMEOUT
>
>
> The strangest thing is that it occurs when the command is executed:
>
> samba-tool dbcheck --cross-ncs --fix --yes
>
>
> Could it be some object causing this error?
>
>
> On Mon, Feb 12, 2024 at 4:40?PM Rowland Penny via samba <
>
> samba at lists.samba.org
>
> > wrote:
>
>
> On Mon, 12 Feb 2024 16:20:27 -0300
>
> Elias Pereira via samba <
>
> samba at lists.samba.org
>
> > wrote:
>
>
> hi,
>
>
> My saga continues...
>
>
> I've configured the audit log for drs_repl in smb.conf, and below is
>
> the log generated.
>
> https://transfer.sh/7fen4qCNIQ/drs_repl.log
>
>
>
> The log level was 5.
>
> drs_repl:5@/var/log/samba/drs_repl.log
>
>
> Could someone take a look and help me understand the log?
>
>
>
> There is probably nothing wrong with your log, but Firefox doesn't
>
> like it, it thinks it contains a virus.
>
>
> Rowland
>
>
>
>
> --
>
> To unsubscribe from this list go to the following URL and read the
>
> instructions:
>
> https://lists.samba.org/mailman/options/samba
>
>
>
>
>
> --
>
> Elias Pereira
>
>
>
>
> --
>
> Elias Pereira
>
> --
>
>
> Andrew Bartlett (he/him)       https://samba.org/~abartlet/
> Samba Team Member (since 2001) https://samba.org
> Samba Team Lead                https://catalyst.net.nz/services/samba
> Catalyst.Net Ltd
>
> Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group company
>
> Samba Development and Support: https://catalyst.net.nz/services/samba
>
> Catalyst IT - Expert Open Source Solutions
>
>
>
>
> --
> Elias Pereira
>
> --
>
> Andrew Bartlett (he/him)       https://samba.org/~abartlet/
> Samba Team Member (since 2001) https://samba.org
> Samba Team Lead                https://catalyst.net.nz/services/samba
> Catalyst.Net Ltd
>
> Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group company
>
> Samba Development and Support: https://catalyst.net.nz/services/samba
>
> Catalyst IT - Expert Open Source Solutions
>
>
>
-- 
Elias Pereira

Hello Andrew,

What's the explanation for when the log level is set to 5, the error
NT_STATUS_IO_TIMEOUT doesn't appear, but when it's at the default log
level, it does?

On Mon, Mar 18, 2024 at 10:33?AM Elias Pereira <empbilly at gmail.com>
wrote:
> hi Andrew, thanks for the help!!!
>
> It seems to me the LDAP process being busy would be the root cause here.
>> Working out what is going on here shouldn't is a detective task - I
always
>> start with a wireshark trace.  The client making all the noise/traffic
will
>> be the one causing the trouble.
>
>
> In the wireshark analysis, should I filter only by the ldap protocol or
> leave everything? Should I look at something specific in the client logs?
>
> On Sun, Mar 10, 2024 at 9:31?PM Andrew Bartlett <abartlet at
samba.org>
> wrote:
>
>> Thanks for getting back to me.
>>
>> It seems to me the LDAP process being busy would be the root cause
here.
>> Working out what is going on here shouldn't is a detective task - I
always
>> start with a wireshark trace.  The client making all the noise/traffic
will
>> be the one causing the trouble.
>>
>> If it isn't clear from that, then look into the DB audit logging
for
>> perhaps busy writes
>>
>>
>>
https://wiki.samba.org/index.php/Setting_up_Audit_Logging#Enabling_AD_DC_Database_Audit_Logging
>>
>> Finally, set 'log level = 5' and look for logs like: LDAP
Query:
>> Duration was
>>
>> This will tell you about how long each query is taking, potentially
>> showing a particularly slow query that needs to be stopped.
>>
>> Andrew Bartlett
>>
>> On Sun, 2024-03-10 at 19:46 -0300, Elias Pereira wrote:
>>
>> Is the drepl local processes very busy doing inbound replication?
>>
>>
>> How can I check this?
>>
>> My instinct is either the server is very busy (and this should show up
in
>> CPU use) or a transaction is being held open excessively.
>>
>>
>> I use VMs on Proxmox. In DC1, I installed the Proxmox agent, and CPU
>> usage via the dashboard is very low. However, when I checked using
'top,'
>> the LDAP process is consuming around 94/96% of the CPU. Very strange.
>>
>>
>> It is probably 94% of a single CPU, but you might have 8 CPUs in the
VM,
>> so overall use is low.
>>
>> The VM has 4 CPUs and 6GB of memory.
>>
>>
>>
>> On Sun, Mar 10, 2024 at 5:55?PM Andrew Bartlett <abartlet at
samba.org>
>> wrote:
>>
>> Either the local server is busy, or possibly (but it would not explain
>> the samba_kcc) Samba's drepl process is stuck talking to a remote
server.
>>
>> Is the drepl local processes very busy doing inbound replication?
>>
>> My instinct is either the server is very busy (and this should show up
in
>> CPU use) or a transaction is being held open excessively.
>>
>> Andrew Bartlett
>>
>> On Sat, 2024-03-09 at 19:11 -0300, Elias Pereira via samba wrote:
>>
>> I've been grappling with a recurring set of errors for quite some
time now:
>>
>> - UpdateRefs failed with NT_STATUS_IO_TIMEOUT
>>
>> - Failed samba_kcc - NT_STATUS_IO_TIMEOUT
>>
>> - IRPC callback failed for DsReplicaSync - NT_STATUS_IO_TIMEOUT
>>
>>
>> Despite cranking up the log level to 10, the returned information
remains
>>
>> frustratingly cryptic and hard to decipher.
>>
>>
>> This error, being overly generic, continues to elude identification
even
>>
>> with
>>
>> the heightened log verbosity. The challenge lies in tracing its origin.
>>
>>
>> Running samba-tool dbcheck doesn't reveal any problems, yet
executing the
>>
>> command while monitoring the Samba log with "tail -f" exposes
errors
>>
>> identical
>>
>> to those described above.
>>
>>
>> Interestingly, samba-tool drs showrepl doesn't report any errors.
>>
>>
>> So, what additional steps can be taken to unearth the root cause
>>
>> of these persistent NT_STATUS_IO_TIMEOUT errors?
>>
>>
>>
>> On Fri, Mar 1, 2024 at 10:32?PM Elias Pereira <
>>
>> empbilly at gmail.com
>>
>> > wrote:
>>
>>
>> There is probably nothing wrong with your log, but Firefox doesn't
>>
>> like it, it thinks it contains a virus.
>>
>>
>>
>> I just saw now that your response ended up in spam, probably because of
>>
>> the link with the log. O.o
>>
>>
>> I still receive the error in the logs:
>>
>> source4/dsdb/kcc/kcc_periodic.c:790: Failed samba_kcc -
>>
>> NT_STATUS_IO_TIMEOUT
>>
>>
>> The strangest thing is that it occurs when the command is executed:
>>
>> samba-tool dbcheck --cross-ncs --fix --yes
>>
>>
>> Could it be some object causing this error?
>>
>>
>> On Mon, Feb 12, 2024 at 4:40?PM Rowland Penny via samba <
>>
>> samba at lists.samba.org
>>
>> > wrote:
>>
>>
>> On Mon, 12 Feb 2024 16:20:27 -0300
>>
>> Elias Pereira via samba <
>>
>> samba at lists.samba.org
>>
>> > wrote:
>>
>>
>> hi,
>>
>>
>> My saga continues...
>>
>>
>> I've configured the audit log for drs_repl in smb.conf, and below
is
>>
>> the log generated.
>>
>> https://transfer.sh/7fen4qCNIQ/drs_repl.log
>>
>>
>>
>> The log level was 5.
>>
>> drs_repl:5@/var/log/samba/drs_repl.log
>>
>>
>> Could someone take a look and help me understand the log?
>>
>>
>>
>> There is probably nothing wrong with your log, but Firefox doesn't
>>
>> like it, it thinks it contains a virus.
>>
>>
>> Rowland
>>
>>
>>
>>
>> --
>>
>> To unsubscribe from this list go to the following URL and read the
>>
>> instructions:
>>
>> https://lists.samba.org/mailman/options/samba
>>
>>
>>
>>
>>
>> --
>>
>> Elias Pereira
>>
>>
>>
>>
>> --
>>
>> Elias Pereira
>>
>> --
>>
>>
>> Andrew Bartlett (he/him)       https://samba.org/~abartlet/
>> Samba Team Member (since 2001) https://samba.org
>> Samba Team Lead                https://catalyst.net.nz/services/samba
>> Catalyst.Net Ltd
>>
>> Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group
>> company
>>
>> Samba Development and Support: https://catalyst.net.nz/services/samba
>>
>> Catalyst IT - Expert Open Source Solutions
>>
>>
>>
>>
>> --
>> Elias Pereira
>>
>> --
>>
>> Andrew Bartlett (he/him)       https://samba.org/~abartlet/
>> Samba Team Member (since 2001) https://samba.org
>> Samba Team Lead                https://catalyst.net.nz/services/samba
>> Catalyst.Net Ltd
>>
>> Proudly developing Samba for Catalyst.Net Ltd - a Catalyst IT group
>> company
>>
>> Samba Development and Support: https://catalyst.net.nz/services/samba
>>
>> Catalyst IT - Expert Open Source Solutions
>>
>>
>>
>
> --
> Elias Pereira
>

-- 
Elias Pereira