Hi Michal, Seems we are doing similar things at the moment: getting samba to work with azure AD. We also see the high CPU usage on the DC that the Azure AD Connect server connected to. Between 70 - 100 percent in our case. We are not seeing any replication issues after azure AD Connect, and I have a script that automatically checks replication every few minutes. I was the one reporting the highwatermark back in 2017, but we're not getting those now. Between 2017 and now, we stopped testing azure, and I took it up again only last week. On the samba DC side, we observe that samba is very buzy talking to the Azure AD Connect machine, even though we filter sync using one group containing only three members, and none of them are actually changing. (this is just for testing now) How many users are you syncing? Are you using pass-through authentication, or syncing password hashes? And is what you chose working for you, authentication-wise? Regards, MJ On 10/21/20 7:10 PM, Michal Bruncko via samba wrote:> ups, seems pictures (attachments in general) are not accepted here, > screen (graph) is available here: > https://i.postimg.cc/xCk6k038/image-2020-10-21-190940.png > > On 10/21/2020 6:00 PM, Michal Bruncko wrote: >> hello >> >> our AD domain is hosted by two samba AD domain controllers version 4.12.6 >> - replication between controllers is fine, no problems. >> - no schema errors. >> - no database errors, all fine. >> - no CPU utilizations >> - wthout noticeable bandwidth utilization >> >> Recently we have deployed Azure AD connector on dedicated windows >> system (system is domain member server). since this deployment we are >> observing following issues on DCs: >> - CPU utilization issue (one CPU core fully utilized) >> - high BW utilization >> - replication issue messages: >> [2020/10/21 17:41:55.043563,? 0] >> ../../source4/rpc_server/drsuapi/getncchanges.c:2910(dcesrv_drsuapi_DsGetNCChanges) >> >> ? ../../source4/rpc_server/drsuapi/getncchanges.c:2910: DsGetNCChanges >> 2nd replication on DN DC= older highwatermark (last_dn >> CN=userXYZ,OU=Users,DC=) >> >> >> and this is happening only on one DC server in time - the one, to >> which this AD connector is connected for doing AD to AAD sync tasks. >> >> More details: >> >> CPU: mostly only one CPU core from all system-assigned cores is >> utilized at 100%: >> >> >> BW utilization: you can see example here (peak starts once the Azure >> AD connector connects to particular DC server) (notice the "uploaded" >> data - 54GB - value from DC system): >> >> >> >> Replicaton errors: repeating messages (example above) every each 4-5 >> seconds. the "last_dn" is changing during time slowly: it is changed >> to another (user) object each several hours. >> >> no other issues observed. >> >> - If we deactivate this Azure connector, all issues stopped (but of >> course we are out of sync with AAD) >> - if we reboot/stop DC1 services (serving for Azure connector), the >> Azure connector switch to DC2 and same story happen again >> (CPU/bandwidth/replication logs) >> >> I've found similar issue reported back in 2017: >> https://lists.samba.org/archive/samba/2017-October/211756.html >> ([Samba] samba getting stuck, highwatermark replication issue?) >> >> seems this issue is still in place now. no difference. >> >> >> does anyone else have similar issues? does anyone else how to resolve >> them? either on Azure AD connector side (there are various >> confiuration option available) or (possibly) on samba side? >> >> >> thank you >> michal >> >
hello mj we are school - we are syncing approx 500 users and several groups. we use pass-through authentication because AAD authentication was not working (that time we used samba 4.8.12, but since moving to 4.12 I didnt tested this), we have (as recommended) one AAD connector plus two authentication agents deployed in our environment. the thing what I dont understand is the amount of data which is connector getting each day (~50GB). I have to check AAD connector tshoot tool to understand whether there are some issues (like sync will not get properly completed and therefore it is always retrying). but from AAD point of view all looks fine - in AAD portal I see the sync is done every 30 minutes without errors. but this is definitely not correct behaviour regards to CPU and BW utilization... regards michal On 10/21/2020 9:16 PM, mj via samba wrote:> Hi Michal, > > Seems we are doing similar things at the moment: getting samba to work > with azure AD. > > We also see the high CPU usage on the DC that the Azure AD Connect > server connected to. Between 70 - 100 percent in our case. > > We are not seeing any replication issues after azure AD Connect, and I > have a script that automatically checks replication every few minutes. > > I was the one reporting the highwatermark back in 2017, but we're not > getting those now. Between 2017 and now, we stopped testing azure, and > I took it up again only last week. > > On the samba DC side, we observe that samba is very buzy talking to > the Azure AD Connect machine, even though we filter sync using one > group containing only three members, and none of them are actually > changing. (this is just for testing now) How many users are you syncing? > > Are you using pass-through authentication, or syncing password hashes? > And is what you chose working for you, authentication-wise? > > Regards, > MJ > > On 10/21/20 7:10 PM, Michal Bruncko via samba wrote: >> ups, seems pictures (attachments in general) are not accepted here, >> screen (graph) is available here: >> https://i.postimg.cc/xCk6k038/image-2020-10-21-190940.png >> >> On 10/21/2020 6:00 PM, Michal Bruncko wrote: >>> hello >>> >>> our AD domain is hosted by two samba AD domain controllers version >>> 4.12.6 >>> - replication between controllers is fine, no problems. >>> - no schema errors. >>> - no database errors, all fine. >>> - no CPU utilizations >>> - wthout noticeable bandwidth utilization >>> >>> Recently we have deployed Azure AD connector on dedicated windows >>> system (system is domain member server). since this deployment we >>> are observing following issues on DCs: >>> - CPU utilization issue (one CPU core fully utilized) >>> - high BW utilization >>> - replication issue messages: >>> [2020/10/21 17:41:55.043563,? 0] >>> ../../source4/rpc_server/drsuapi/getncchanges.c:2910(dcesrv_drsuapi_DsGetNCChanges) >>> >>> ? ../../source4/rpc_server/drsuapi/getncchanges.c:2910: >>> DsGetNCChanges 2nd replication on DN DC= older highwatermark >>> (last_dn CN=userXYZ,OU=Users,DC=) >>> >>> >>> and this is happening only on one DC server in time - the one, to >>> which this AD connector is connected for doing AD to AAD sync tasks. >>> >>> More details: >>> >>> CPU: mostly only one CPU core from all system-assigned cores is >>> utilized at 100%: >>> >>> >>> BW utilization: you can see example here (peak starts once the Azure >>> AD connector connects to particular DC server) (notice the >>> "uploaded" data - 54GB - value from DC system): >>> >>> >>> >>> Replicaton errors: repeating messages (example above) every each 4-5 >>> seconds. the "last_dn" is changing during time slowly: it is changed >>> to another (user) object each several hours. >>> >>> no other issues observed. >>> >>> - If we deactivate this Azure connector, all issues stopped (but of >>> course we are out of sync with AAD) >>> - if we reboot/stop DC1 services (serving for Azure connector), the >>> Azure connector switch to DC2 and same story happen again >>> (CPU/bandwidth/replication logs) >>> >>> I've found similar issue reported back in 2017: >>> https://lists.samba.org/archive/samba/2017-October/211756.html >>> ([Samba] samba getting stuck, highwatermark replication issue?) >>> >>> seems this issue is still in place now. no difference. >>> >>> >>> does anyone else have similar issues? does anyone else how to >>> resolve them? either on Azure AD connector side (there are various >>> confiuration option available) or (possibly) on samba side? >>> >>> >>> thank you >>> michal >>> >> >
just small update: - idfix tool (Directory Synchronization Error Remediation Tool / https://github.com/microsoft/idfix) shows just small issues like empty/missing displayName attrib in some of objects which I have corrected and no more issues present at all. - no errors from AAD connect event viewer: final log message is "Scheduler::SchedulerThreadMain : Completed configured scheduler operations." (no errors before) - aad connect info: Azure AD Tenant - Settings PasswordHashSync: True ForcePasswordChangeOnLogOn: False UserWriteback: False DeviceWriteback: False UnifiedGroupWriteback: False Azure AD Connect - Global Settings Microsoft.AADFilter.ApplicationList: Microsoft.AADFilter.AttributeExclusionList: Microsoft.Configuration.ImportDate: 2020-10-19 17:08:19Z Microsoft.ConnectDirectories.WizardDirectoryMode: AD Microsoft.DeviceWriteBack.Container: Microsoft.DeviceWriteBack.Forest: Microsoft.DirectoryExtension.SourceTargetAttributesMap: Microsoft.OptionalFeature.AutoUpgradeState: Enabled Microsoft.OptionalFeature.AutoUpgradeSuspensionReason: Microsoft.OptionalFeature.DeviceWriteBack: False Microsoft.OptionalFeature.DeviceWriteUp: True Microsoft.OptionalFeature.DirectoryExtension: True Microsoft.OptionalFeature.DirectoryExtensionAttributes: Microsoft.OptionalFeature.ExchangeMailPublicFolder: False Microsoft.OptionalFeature.ExportDeletionThreshold: True Microsoft.OptionalFeature.ExportDeletionThresholdValue: 500 Microsoft.OptionalFeature.FilterAAD: True Microsoft.OptionalFeature.GroupFiltering: False Microsoft.OptionalFeature.GroupWriteBack: False Microsoft.OptionalFeature.HybridExchange: False Microsoft.OptionalFeature.UserWriteBack: False Microsoft.SynchronizationOption.AnchorAttribute: mS-DS-ConsistencyGuid Microsoft.SynchronizationOption.CustomAttribute: Microsoft.SynchronizationOption.JoinCriteria: AlwaysProvision Microsoft.SynchronizationOption.UPNAttribute: userPrincipalName Microsoft.Synchronize.MaintenanceEnabled: True Microsoft.Synchronize.NextStartTime: Thu, 22 Oct 2020 15:37:03 GMT Microsoft.Synchronize.RunHistoryPurgeInterval: 7.00:00:00 Microsoft.Synchronize.ServerConfigurationVersion: 1.5.45.0 Microsoft.Synchronize.SchedulerSuspended: False Microsoft.Synchronize.StagingMode: False Microsoft.Synchronize.SynchronizationPolicy: Delta Microsoft.Synchronize.SynchronizationSchedule: True Microsoft.Synchronize.TimeInterval: 00:30:00 Microsoft.SystemInformation.MachineRole: RoleMemberServer Microsoft.UserSignIn.DesktopSsoEnabled: True Microsoft.UserSignIn.SignOnMethod: PassThroughAuthentication Microsoft.UserWriteBack.Container: Microsoft.UserWriteBack.Forest: Microsoft.Version.SynchronizationRuleImmutableTag: V1 Azure AD Connect Sync Scheduler - Settings AllowedSyncCycleInterval: 00:30:00 CurrentlyEffectiveSyncCycleInterval: 00:30:00 CustomizedSyncCycleInterval: NextSyncCyclePolicyType: Delta NextSyncCycleStartTimeInUTC: 22.10.2020 15:37:03 PurgeRunHistoryInterval: 7.00:00:00 SyncCycleEnabled: True MaintenanceEnabled: True StagingModeEnabled: False SchedulerSuspended: False SyncCycleInProgress: False On 10/22/2020 1:19 PM, Michal Bruncko via samba wrote:> hello mj > > we are school - we are syncing approx 500 users and several groups. > > we use pass-through authentication because AAD authentication was not > working (that time we used samba 4.8.12, but since moving to 4.12 I > didnt tested this), we have (as recommended) one AAD connector plus > two authentication agents deployed in our environment. > > the thing what I dont understand is the amount of data which is > connector getting each day (~50GB). I have to check AAD connector > tshoot tool to understand whether there are some issues (like sync > will not get properly completed and therefore it is always retrying). > > but from AAD point of view all looks fine - in AAD portal I see the > sync is done every 30 minutes without errors. > > but this is definitely not correct behaviour regards to CPU and BW > utilization... > > regards > michal > > > On 10/21/2020 9:16 PM, mj via samba wrote: >> Hi Michal, >> >> Seems we are doing similar things at the moment: getting samba to >> work with azure AD. >> >> We also see the high CPU usage on the DC that the Azure AD Connect >> server connected to. Between 70 - 100 percent in our case. >> >> We are not seeing any replication issues after azure AD Connect, and >> I have a script that automatically checks replication every few minutes. >> >> I was the one reporting the highwatermark back in 2017, but we're not >> getting those now. Between 2017 and now, we stopped testing azure, >> and I took it up again only last week. >> >> On the samba DC side, we observe that samba is very buzy talking to >> the Azure AD Connect machine, even though we filter sync using one >> group containing only three members, and none of them are actually >> changing. (this is just for testing now) How many users are you syncing? >> >> Are you using pass-through authentication, or syncing password >> hashes? And is what you chose working for you, authentication-wise? >> >> Regards, >> MJ >> >> On 10/21/20 7:10 PM, Michal Bruncko via samba wrote: >>> ups, seems pictures (attachments in general) are not accepted here, >>> screen (graph) is available here: >>> https://i.postimg.cc/xCk6k038/image-2020-10-21-190940.png >>> >>> On 10/21/2020 6:00 PM, Michal Bruncko wrote: >>>> hello >>>> >>>> our AD domain is hosted by two samba AD domain controllers version >>>> 4.12.6 >>>> - replication between controllers is fine, no problems. >>>> - no schema errors. >>>> - no database errors, all fine. >>>> - no CPU utilizations >>>> - wthout noticeable bandwidth utilization >>>> >>>> Recently we have deployed Azure AD connector on dedicated windows >>>> system (system is domain member server). since this deployment we >>>> are observing following issues on DCs: >>>> - CPU utilization issue (one CPU core fully utilized) >>>> - high BW utilization >>>> - replication issue messages: >>>> [2020/10/21 17:41:55.043563,? 0] >>>> ../../source4/rpc_server/drsuapi/getncchanges.c:2910(dcesrv_drsuapi_DsGetNCChanges) >>>> >>>> ? ../../source4/rpc_server/drsuapi/getncchanges.c:2910: >>>> DsGetNCChanges 2nd replication on DN DC= older highwatermark >>>> (last_dn CN=userXYZ,OU=Users,DC=) >>>> >>>> >>>> and this is happening only on one DC server in time - the one, to >>>> which this AD connector is connected for doing AD to AAD sync tasks. >>>> >>>> More details: >>>> >>>> CPU: mostly only one CPU core from all system-assigned cores is >>>> utilized at 100%: >>>> >>>> >>>> BW utilization: you can see example here (peak starts once the >>>> Azure AD connector connects to particular DC server) (notice the >>>> "uploaded" data - 54GB - value from DC system): >>>> >>>> >>>> >>>> Replicaton errors: repeating messages (example above) every each >>>> 4-5 seconds. the "last_dn" is changing during time slowly: it is >>>> changed to another (user) object each several hours. >>>> >>>> no other issues observed. >>>> >>>> - If we deactivate this Azure connector, all issues stopped (but of >>>> course we are out of sync with AAD) >>>> - if we reboot/stop DC1 services (serving for Azure connector), the >>>> Azure connector switch to DC2 and same story happen again >>>> (CPU/bandwidth/replication logs) >>>> >>>> I've found similar issue reported back in 2017: >>>> https://lists.samba.org/archive/samba/2017-October/211756.html >>>> ([Samba] samba getting stuck, highwatermark replication issue?) >>>> >>>> seems this issue is still in place now. no difference. >>>> >>>> >>>> does anyone else have similar issues? does anyone else how to >>>> resolve them? either on Azure AD connector side (there are various >>>> confiuration option available) or (possibly) on samba side? >>>> >>>> >>>> thank you >>>> michal >>>> >>> >> > >