samba - Oct 2020 - Azure AD Connect and the challenge of funding Samba bugs

Hi Michal,

Thanks for updating, I can confirm your findings on our site. 
Pass-through auth works nicely here as well.

However, when reading the ticket, we understand that Ralph B?hme claims 
that password-hashes should also work, after making the sync account 
MSOL_604447e... a member of "domain admins".

(instead of keeping the default rights that are granted by the 
installer, which directory ACLs are not honored by the Samba DRSR RPC 
service)

Buzy today, but in the coming days I will spend some time trying to see 
if I can make the password hash sync work as well. (if it works for 
Ralph B?hme, we should also be able to make it work) :-)

Thanks for updating here!

MJ


On 25/10/2020 17:11, Michal Bruncko via samba wrote:
> hello all
> 
> just to summarize this topic for anybody else.
> after some time spent with tshooting there seems is a sufficient 
> workaround for this issue: disable "sync of password hashes"
option
> within Azure AD connect tool.
> with disabling this option all three issues (CPU/Bandwidth 
> utilization/replication logs) are gone!
> seems that for now the only working option for samba and Azure connect 
> tool is to:
> - use pass-through authentication
> - deploy at least one authentication agent (agent is installed a part of 
> Azure AD connect tool), but two agents are recommended
> - and disable "sync of password hashes" option in Azure AD
connect tool
> 
> seems that syncing password hahes is still problem for samba: 
> https://bugzilla.samba.org/show_bug.cgi?id=10635
> and it looks like Azure connect tool keeps trying (and endlessly 
> retrying) to get them synced which is causing all three issues in my 
> environment (CPU/Bandwidth utilization/replication logs).
> 
> Michal
> 
> 
> On 10/22/2020 5:40 PM, Michal Bruncko via samba wrote:
>> just small update:
>>
>> - idfix tool (Directory Synchronization Error Remediation Tool / 
>> https://github.com/microsoft/idfix) shows just small issues like 
>> empty/missing displayName attrib in some of objects which I have 
>> corrected and no more issues present at all.
>> - no errors from AAD connect event viewer: final log message is 
>> "Scheduler::SchedulerThreadMain : Completed configured scheduler 
>> operations." (no errors before)
>> - aad connect info:
>>
>> Azure AD Tenant - Settings
>>
>> PasswordHashSync: True
>> ForcePasswordChangeOnLogOn: False
>> UserWriteback: False
>> DeviceWriteback: False
>> UnifiedGroupWriteback: False
>>
>>
>> Azure AD Connect - Global Settings
>>
>> Microsoft.AADFilter.ApplicationList:
>> Microsoft.AADFilter.AttributeExclusionList:
>> Microsoft.Configuration.ImportDate: 2020-10-19 17:08:19Z
>> Microsoft.ConnectDirectories.WizardDirectoryMode: AD
>> Microsoft.DeviceWriteBack.Container:
>> Microsoft.DeviceWriteBack.Forest:
>> Microsoft.DirectoryExtension.SourceTargetAttributesMap:
>> Microsoft.OptionalFeature.AutoUpgradeState: Enabled
>> Microsoft.OptionalFeature.AutoUpgradeSuspensionReason:
>> Microsoft.OptionalFeature.DeviceWriteBack: False
>> Microsoft.OptionalFeature.DeviceWriteUp: True
>> Microsoft.OptionalFeature.DirectoryExtension: True
>> Microsoft.OptionalFeature.DirectoryExtensionAttributes:
>> Microsoft.OptionalFeature.ExchangeMailPublicFolder: False
>> Microsoft.OptionalFeature.ExportDeletionThreshold: True
>> Microsoft.OptionalFeature.ExportDeletionThresholdValue: 500
>> Microsoft.OptionalFeature.FilterAAD: True
>> Microsoft.OptionalFeature.GroupFiltering: False
>> Microsoft.OptionalFeature.GroupWriteBack: False
>> Microsoft.OptionalFeature.HybridExchange: False
>> Microsoft.OptionalFeature.UserWriteBack: False
>> Microsoft.SynchronizationOption.AnchorAttribute: mS-DS-ConsistencyGuid
>> Microsoft.SynchronizationOption.CustomAttribute:
>> Microsoft.SynchronizationOption.JoinCriteria: AlwaysProvision
>> Microsoft.SynchronizationOption.UPNAttribute: userPrincipalName
>> Microsoft.Synchronize.MaintenanceEnabled: True
>> Microsoft.Synchronize.NextStartTime: Thu, 22 Oct 2020 15:37:03 GMT
>> Microsoft.Synchronize.RunHistoryPurgeInterval: 7.00:00:00
>> Microsoft.Synchronize.ServerConfigurationVersion: 1.5.45.0
>> Microsoft.Synchronize.SchedulerSuspended: False
>> Microsoft.Synchronize.StagingMode: False
>> Microsoft.Synchronize.SynchronizationPolicy: Delta
>> Microsoft.Synchronize.SynchronizationSchedule: True
>> Microsoft.Synchronize.TimeInterval: 00:30:00
>> Microsoft.SystemInformation.MachineRole: RoleMemberServer
>> Microsoft.UserSignIn.DesktopSsoEnabled: True
>> Microsoft.UserSignIn.SignOnMethod: PassThroughAuthentication
>> Microsoft.UserWriteBack.Container:
>> Microsoft.UserWriteBack.Forest:
>> Microsoft.Version.SynchronizationRuleImmutableTag: V1
>>
>>
>> Azure AD Connect Sync Scheduler - Settings
>>
>>
>> AllowedSyncCycleInterval: 00:30:00
>> CurrentlyEffectiveSyncCycleInterval: 00:30:00
>> CustomizedSyncCycleInterval:
>> NextSyncCyclePolicyType: Delta
>> NextSyncCycleStartTimeInUTC: 22.10.2020 15:37:03
>> PurgeRunHistoryInterval: 7.00:00:00
>> SyncCycleEnabled: True
>> MaintenanceEnabled: True
>> StagingModeEnabled: False
>> SchedulerSuspended: False
>> SyncCycleInProgress: False
>>
>>
>>
>> On 10/22/2020 1:19 PM, Michal Bruncko via samba wrote:
>>> hello mj
>>>
>>> we are school - we are syncing approx 500 users and several groups.
>>>
>>> we use pass-through authentication because AAD authentication was
not
>>> working (that time we used samba 4.8.12, but since moving to 4.12 I
>>> didnt tested this), we have (as recommended) one AAD connector plus
>>> two authentication agents deployed in our environment.
>>>
>>> the thing what I dont understand is the amount of data which is 
>>> connector getting each day (~50GB). I have to check AAD connector 
>>> tshoot tool to understand whether there are some issues (like sync 
>>> will not get properly completed and therefore it is always
retrying).
>>>
>>> but from AAD point of view all looks fine - in AAD portal I see the
>>> sync is done every 30 minutes without errors.
>>>
>>> but this is definitely not correct behaviour regards to CPU and BW 
>>> utilization...
>>>
>>> regards
>>> michal
>>>
>>>
>>> On 10/21/2020 9:16 PM, mj via samba wrote:
>>>> Hi Michal,
>>>>
>>>> Seems we are doing similar things at the moment: getting samba
to
>>>> work with azure AD.
>>>>
>>>> We also see the high CPU usage on the DC that the Azure AD
Connect
>>>> server connected to. Between 70 - 100 percent in our case.
>>>>
>>>> We are not seeing any replication issues after azure AD
Connect, and
>>>> I have a script that automatically checks replication every few
>>>> minutes.
>>>>
>>>> I was the one reporting the highwatermark back in 2017, but
we're
>>>> not getting those now. Between 2017 and now, we stopped testing
>>>> azure, and I took it up again only last week.
>>>>
>>>> On the samba DC side, we observe that samba is very buzy
talking to
>>>> the Azure AD Connect machine, even though we filter sync using
one
>>>> group containing only three members, and none of them are
actually
>>>> changing. (this is just for testing now) How many users are you
>>>> syncing?
>>>>
>>>> Are you using pass-through authentication, or syncing password 
>>>> hashes? And is what you chose working for you,
authentication-wise?
>>>>
>>>> Regards,
>>>> MJ
>>>>
>>>> On 10/21/20 7:10 PM, Michal Bruncko via samba wrote:
>>>>> ups, seems pictures (attachments in general) are not
accepted here,
>>>>> screen (graph) is available here: 
>>>>> https://i.postimg.cc/xCk6k038/image-2020-10-21-190940.png
>>>>>
>>>>> On 10/21/2020 6:00 PM, Michal Bruncko wrote:
>>>>>> hello
>>>>>>
>>>>>> our AD domain is hosted by two samba AD domain
controllers version
>>>>>> 4.12.6
>>>>>> - replication between controllers is fine, no problems.
>>>>>> - no schema errors.
>>>>>> - no database errors, all fine.
>>>>>> - no CPU utilizations
>>>>>> - wthout noticeable bandwidth utilization
>>>>>>
>>>>>> Recently we have deployed Azure AD connector on
dedicated windows
>>>>>> system (system is domain member server). since this
deployment we
>>>>>> are observing following issues on DCs:
>>>>>> - CPU utilization issue (one CPU core fully utilized)
>>>>>> - high BW utilization
>>>>>> - replication issue messages:
>>>>>> [2020/10/21 17:41:55.043563,? 0] 
>>>>>>
../../source4/rpc_server/drsuapi/getncchanges.c:2910(dcesrv_drsuapi_DsGetNCChanges)
>>>>>>
>>>>>> ? ../../source4/rpc_server/drsuapi/getncchanges.c:2910:
>>>>>> DsGetNCChanges 2nd replication on DN DC= older
highwatermark
>>>>>> (last_dn CN=userXYZ,OU=Users,DC=)
>>>>>>
>>>>>>
>>>>>> and this is happening only on one DC server in time -
the one, to
>>>>>> which this AD connector is connected for doing AD to
AAD sync tasks.
>>>>>>
>>>>>> More details:
>>>>>>
>>>>>> CPU: mostly only one CPU core from all system-assigned
cores is
>>>>>> utilized at 100%:
>>>>>>
>>>>>>
>>>>>> BW utilization: you can see example here (peak starts
once the
>>>>>> Azure AD connector connects to particular DC server)
(notice the
>>>>>> "uploaded" data - 54GB - value from DC
system):
>>>>>>
>>>>>>
>>>>>>
>>>>>> Replicaton errors: repeating messages (example above)
every each
>>>>>> 4-5 seconds. the "last_dn" is changing during
time slowly: it is
>>>>>> changed to another (user) object each several hours.
>>>>>>
>>>>>> no other issues observed.
>>>>>>
>>>>>> - If we deactivate this Azure connector, all issues
stopped (but
>>>>>> of course we are out of sync with AAD)
>>>>>> - if we reboot/stop DC1 services (serving for Azure
connector),
>>>>>> the Azure connector switch to DC2 and same story happen
again
>>>>>> (CPU/bandwidth/replication logs)
>>>>>>
>>>>>> I've found similar issue reported back in 2017: 
>>>>>>
https://lists.samba.org/archive/samba/2017-October/211756.html
>>>>>> ([Samba] samba getting stuck, highwatermark replication
issue?)
>>>>>>
>>>>>> seems this issue is still in place now. no difference.
>>>>>>
>>>>>>
>>>>>> does anyone else have similar issues? does anyone else
how to
>>>>>> resolve them? either on Azure AD connector side (there
are various
>>>>>> confiuration option available) or (possibly) on samba
side?
>>>>>>
>>>>>>
>>>>>> thank you
>>>>>> michal
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
> 
>

Thank you both for digging into this.

As you have seen this is a tricky area for Samba, being as it involves
external services, additional (and it seems changing) essentially
third-party software (not just base Windows) and has consistently
fallen into a gap of 'almost working' for a number of years.

The fact that there is a viable workaround (pass-though authentication)
also seems to be making this harder to fix - because it remains an
annoyance, not a deal-breaker.

Furthermore, it seems to have been quite difficult to get good
diagnostic logs of the issue - both on Samba's bugzilla and when
speaking with potential commercial customers, I've never actually seen
a clear set of logs!

But I want to touch on something more broadly:

Samba's AD DC development is to each of or community members a
volunteer effort.  The Samba Team isn't funded for development, we take
donations which help pay our hosting bills (CI in particular) and until
the year of the pandemic covered travel for those who needed it. 

Nobody is on the long-term payroll of the Samba Team, instead Samba is
where is is thanks to the spare time of our committed volunteers and
the work time of those who are supported by their employers to work in
Samba for their $DAYJOB. 

What that does mean is that what gets fixed and does not get fixed is
in large part comes down to what catches the interest of developers who
some flexibility (in their job or their own time) or what paying
customers need via support or development contracts.  

Now, to be clear we all work hard to ensure that what do advances a
common Samba vision and helps more than just an immediate customer, but
my point is that we have a problem:  Bugs like this sometimes have to
really hurt someone before they get attention.

As to a solution?  I don't have a good one.  

Samba isn't funded (by our donations) to a scale where we can afford
even a significant fraction of a senior developer salary.  I'll also
warn that genuinely independent open source projects paying folks has a
difficult history.  

Many say crowd-funding, but that essentially requires that someone
offer and hold to a fixed price quote for a fix.  Given that much of
the work in Samba is to work out the problem, this is a challenge.  (It
might work for feature development.) 

I'm skating on really thin ice here, but I like the idea of support
contracts.  I like them because they are outside Samba.org: a
commercial agreement between a single customer and a Samba support and
development shop.  I also like them because for difficult issues at
least the customer and commercial support provider can work out what
the root of the problem is, which is often half the challenge.  

Sadly I also know the economics of those are difficult for many of our
users, who are attracted to Samba not just for the awesome features but
the low cost.

Samba is a proud free software project and always will be.  The freedom
to use, modify and redistribute is central to what we care about.  The
challenge of keeping Samba development funded away from per-seat
licences and in an environment where the community expectation is that
Samba should not just be free but cost $0 remains however quite
profound. 

Even more difficult is the funding of security support, but I'll leave
that for another time.

Finally, I do realise that compared to fond memories of times long ago
Samba is much harder to contribute to.  Even despite recent work with
our move to GitLab new Contribute page on the wiki, making a meaningful
change to a C based project and following that up with a comprehensive
(and often Python based) testsuite is a big ask.  Yes, it keeps Samba
from breaking but perhaps pushes away the 'Jack of all trades' sysadmin
or student (I was both...) who in times past might have just dived in
and fixed things.

Andrew Bartlett

-- 
Andrew Bartlett                       https://samba.org/~abartlet/
Authentication Developer, Samba Team  https://samba.org
Samba Developer, Catalyst IT          
https://catalyst.net.nz/services/samba

Hi Andrew,

Op 26-10-2020 om 22:24 schreef Andrew Bartlett:
> Furthermore, it seems to have been quite difficult to get good
> diagnostic logs of the issue - both on Samba's bugzilla and when
> speaking with potential commercial customers, I've never actually seen
> a clear set of logs!
For us, the problem was/is that it was difficult to get clean "Azure AD 
connect"-only (not polluted) level 10 logs from a production AD DC.

Things are buzy here, but I have the intention to setup a specific test 
AD on latest samba, log level 10, reproduce the high-CPU usage and get 
you the resulting logs. (and/or attach them to the ticket) Perhaps that 
will tell us that there *are* some quick wins after all.

Thanks for your explanation. Your work (your team, that of sernet, 
Louis, Rowland, and many MANY more that help out here regularly) is 
*very* much valued and appreciated.

MJ

Hi Andrew,

I remember we talked about crowed-funding in SambaXP previous year. The
main question was tax issue for a company and if I am not mistaken, some
other legal issues was on the table from software freedom conservacy
perspective.

I just check the samba web site to be sure if donation button still works
and it looks ok. If that funding system is acceptable for team needs, we
may be open some most wanted futures and everybody can donate and follow
the fundraising process for that future.

This can create more collaboration between team and community. Also
community can have some drive motivation on the samba project.

Do you think that is applicable?

All the best,
Caglar

On 27 Oct 2020 Tue at 00:25 Andrew Bartlett via samba <samba at
lists.samba.org>
wrote:
> Thank you both for digging into this.
>
> As you have seen this is a tricky area for Samba, being as it involves
> external services, additional (and it seems changing) essentially
> third-party software (not just base Windows) and has consistently
> fallen into a gap of 'almost working' for a number of years.
>
> The fact that there is a viable workaround (pass-though authentication)
> also seems to be making this harder to fix - because it remains an
> annoyance, not a deal-breaker.
>
> Furthermore, it seems to have been quite difficult to get good
> diagnostic logs of the issue - both on Samba's bugzilla and when
> speaking with potential commercial customers, I've never actually seen
> a clear set of logs!
>
> But I want to touch on something more broadly:
>
> Samba's AD DC development is to each of or community members a
> volunteer effort.  The Samba Team isn't funded for development, we take
> donations which help pay our hosting bills (CI in particular) and until
> the year of the pandemic covered travel for those who needed it.
>
> Nobody is on the long-term payroll of the Samba Team, instead Samba is
> where is is thanks to the spare time of our committed volunteers and
> the work time of those who are supported by their employers to work in
> Samba for their $DAYJOB.
>
> What that does mean is that what gets fixed and does not get fixed is
> in large part comes down to what catches the interest of developers who
> some flexibility (in their job or their own time) or what paying
> customers need via support or development contracts.
>
> Now, to be clear we all work hard to ensure that what do advances a
> common Samba vision and helps more than just an immediate customer, but
> my point is that we have a problem:  Bugs like this sometimes have to
> really hurt someone before they get attention.
>
> As to a solution?  I don't have a good one.
>
> Samba isn't funded (by our donations) to a scale where we can afford
> even a significant fraction of a senior developer salary.  I'll also
> warn that genuinely independent open source projects paying folks has a
> difficult history.
>
> Many say crowd-funding, but that essentially requires that someone
> offer and hold to a fixed price quote for a fix.  Given that much of
> the work in Samba is to work out the problem, this is a challenge.  (It
> might work for feature development.)
>
> I'm skating on really thin ice here, but I like the idea of support
> contracts.  I like them because they are outside Samba.org: a
> commercial agreement between a single customer and a Samba support and
> development shop.  I also like them because for difficult issues at
> least the customer and commercial support provider can work out what
> the root of the problem is, which is often half the challenge.
>
> Sadly I also know the economics of those are difficult for many of our
> users, who are attracted to Samba not just for the awesome features but
> the low cost.
>
> Samba is a proud free software project and always will be.  The freedom
> to use, modify and redistribute is central to what we care about.  The
> challenge of keeping Samba development funded away from per-seat
> licences and in an environment where the community expectation is that
> Samba should not just be free but cost $0 remains however quite
> profound.
>
> Even more difficult is the funding of security support, but I'll leave
> that for another time.
>
> Finally, I do realise that compared to fond memories of times long ago
> Samba is much harder to contribute to.  Even despite recent work with
> our move to GitLab new Contribute page on the wiki, making a meaningful
> change to a C based project and following that up with a comprehensive
> (and often Python based) testsuite is a big ask.  Yes, it keeps Samba
> from breaking but perhaps pushes away the 'Jack of all trades'
sysadmin
> or student (I was both...) who in times past might have just dived in
> and fixed things.
>
> Andrew Bartlett
>
> --
> Andrew Bartlett                       https://samba.org/~abartlet/
> Authentication Developer, Samba Team  https://samba.org
> Samba Developer, Catalyst IT
> https://catalyst.net.nz/services/samba
>
>
>
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

Hi all,

An update.

On 10/26/20 10:24 PM, Andrew Bartlett wrote:
> The fact that there is a viable workaround (pass-though authentication)
> also seems to be making this harder to fix - because it remains an
> annoyance, not a deal-breaker.
Today I tried again with these ingredients:
- fresh azure tenant
- fresh installed AD (samba 4.12.8 sernet)
- an azure "custom domain name" for our AD realm, status
"verified"
- new Azure AD Connect Cloud Provisioning agent, using a "domain
admins"
AD account
- with password-hash sync

And it works. :-)

No high CPU usage on the samba DC so far. I tried turning off the samba 
DC, and I can still authentiate on office365, meaning the password-hash 
successfully synced as well.

The new tool is different in many ways, but the way we see it, it has 
many advantages over the older Azure AD Connect. AD Connect required a 
mssql server and you could have only one Azure Connect server per AD. 
The new one is very light-weight, processing/configuration done in 
Azure, and you can simply install multiple agents for HA.

But most importantly: it seems to work nicely with samba. (so far, 
anyway...) :-)

Here is a small article about the differences between the two:
https://docs.microsoft.com/nl-nl/azure/active-directory/cloud-provisioning/what-is-cloud-provisioning

Enjoy your evening,
MJ