samba - Jul 2021 - howto optimize samba/kerberos for 20k requests per minute - help needed

Exacly what i mean. ... Kopano, i was already thinking is kopano.. 
But there is one big difference i think. With your setup and mine

I think you run Kopano from within UCS 4.3 
( and i tested also USC 5.0, no kopano there, the move to licenced kopano. ) 
I run Kopano on clean Debian 10 install.

And what version kopano/web app is running because
i dont see that here. IO looks normal. 

Greetz, 

Louis
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Stefan Bauer via samba
> Verzonden: dinsdag 20 juli 2021 7:57
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] howto optimize samba/kerberos for 20k 
> requests per minute - help needed
> 
> Hi,
> 
> it is kopano. They state on their website?:
> 
> -----------------
> 
> Note
> 
> Please note that due to performance problems in Samba 4, 
> Samba 4 is not 
> supported as a user source for setups larger than 50 users.
> 
> -----------------
> 
> And that is indeed what i notice. The high amount of 
> kerberos-requests 
> between samba-DC and kopano server is causing a very high 
> io-load on the 
> samba system - renders the system unstable. Also the time an 
> authentication request takes, is between 200-400ms.
> 
> As Kopano is doing an authentication every time a user clicks 
> a single 
> mail and no caching is possible, it is hitting the samba 
> system too hard.
> 
> 
> Stefan
> 
> 
> ? 
> https://documentation.kopano.io/kopanocore_administrator_manua
> l/user_management.html
> 
> 
> On 19.07.21 11:27, Rowland Penny via samba wrote:
> > On Mon, 2021-07-19 at 11:13 +0200, Stefan Bauer via samba wrote:
> >> Hi and thank you for your time.
> >>
> >> We got now the confirmation that samba 4 is not supported by our
> >> software-vendor.
> > If I might ask, who is your software vendor and what is the 
> software ?
> > In most cases, when a supplier says they do not support 
> Samba 4, they
> > do support AD.
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
> 
>

On Tue, 2021-07-20 at 08:45 +0200, L.P.H. van Belle via samba
wrote:
> Exacly what i mean. ... Kopano, i was already thinking is kopano.. 
> But there is one big difference i think. With your setup and mine
> 
> I think you run Kopano from within UCS 4.3 
> ( and i tested also USC 5.0, no kopano there, the move to licenced
> kopano. ) 
> I run Kopano on clean Debian 10 install.
> 
> And what version kopano/web app is running because
> i dont see that here. IO looks normal. 
> 
I downloaded UCS5 and tried to install it in a VM. The first attempt
failed when I tried to set up a DC, there is (apparently) no way to
change the dns and when it did finish the set up, there was no Samba!

Second attempt, I went for what it described as a join to Active
Directory, I ended up with what I call a Unix domain member, but with
what I would call a borked smb.conf, it was using the ldap backend for
the '*' domain and the nss backend for the 'DOMAIN' domain.
There were
numerous other parameters that I wouldn't use.

All in all, I wouldn't recomend UCS5 to my worst enemy, but this is
just my personal opinion.

Rowland

I have a plain UCS4.4-8 (latest) with kopano inside correct. Joined it 
initially to a server 2012 windows DC.

Later migrated that windows 2012 domain to a UCS domain. Kopano is still 
a member-server.

/etc/kopano/ldap.conf contains:

ldap_uri = ldap://kopano01.procorp.local:7389/

(this is a the local ldap on kopano server)

As the local ldap seems to do not get passwords synced UCS-master to 
local LDAP. Due to this the system is doing huge amount of 
kerberos-connections to the UCS Samba-DC to validate the user 
credentials all the time (25k/minute)

09:28:55.718262 IP kopano01.procorp.local.42559 > 
adm-ucs0.procorp.local.kerberos:? v5
09:28:55.723467 IP adm-ucs0.procorp.local.kerberos > 
kopano01.procorp.local.59107:
09:28:55.726673 IP kopano01.procorp.local.33144 > 
adm-ucs0.procorp.local.kerberos: Flags [S], seq 1850537116, win 29200, 
options [mss 1460,sackOK,TS val 279012852 ecr 0,nop,wscale 7], length 0

Due to this, the authentication time is very bad:

root at kopano01:~# kopano-stats --system | grep ldap_avg_auth
 ??? ldap_avg_auth??? ??? Average duration (?s) of authentication made 
to LDAP server??? ??? ??? 276250

(connection to local ldap -> kerberos request to UCS-master and return)

I'm running:

Webapp 5.1.0.0+167.1
Kopano core 8.7.20


On 20.07.21 08:45, L.P.H. van Belle via samba wrote:
> Exacly what i mean. ... Kopano, i was already thinking is kopano..
> But there is one big difference i think. With your setup and mine
>
> I think you run Kopano from within UCS 4.3
> ( and i tested also USC 5.0, no kopano there, the move to licenced kopano.
)
> I run Kopano on clean Debian 10 install.
>
> And what version kopano/web app is running because
> i dont see that here. IO looks normal.
>
> Greetz,
>
> Louis
>
>> -----Oorspronkelijk bericht-----
>> Van: samba [mailto:samba-bounces at lists.samba.org] Namens
>> Stefan Bauer via samba
>> Verzonden: dinsdag 20 juli 2021 7:57
>> Aan: samba at lists.samba.org
>> Onderwerp: Re: [Samba] howto optimize samba/kerberos for 20k
>> requests per minute - help needed
>>
>> Hi,
>>
>> it is kopano. They state on their website?:
>>
>> -----------------
>>
>> Note
>>
>> Please note that due to performance problems in Samba 4,
>> Samba 4 is not
>> supported as a user source for setups larger than 50 users.
>>
>> -----------------
>>
>> And that is indeed what i notice. The high amount of
>> kerberos-requests
>> between samba-DC and kopano server is causing a very high
>> io-load on the
>> samba system - renders the system unstable. Also the time an
>> authentication request takes, is between 200-400ms.
>>
>> As Kopano is doing an authentication every time a user clicks
>> a single
>> mail and no caching is possible, it is hitting the samba
>> system too hard.
>>
>>
>> Stefan
>>
>>
>> ?
>> https://documentation.kopano.io/kopanocore_administrator_manua
>> l/user_management.html
>>
>>
>> On 19.07.21 11:27, Rowland Penny via samba wrote:
>>> On Mon, 2021-07-19 at 11:13 +0200, Stefan Bauer via samba wrote:
>>>> Hi and thank you for your time.
>>>>
>>>> We got now the confirmation that samba 4 is not supported by
our
>>>> software-vendor.
>>> If I might ask, who is your software vendor and what is the
>> software ?
>>> In most cases, when a supplier says they do not support
>> Samba 4, they
>>> do support AD.
>> -- 
>> To unsubscribe from this list go to the following URL and read the
>> instructions:  https://lists.samba.org/mailman/options/samba
>>
>>
>

Stefan, 

That its so slow, that can be becaused by UCS setup.

Its setup with kopano and AD is like this. 
Samba AD -> ldap -> (kopano server) >  Ldap proxy > ldap DB >
kopano
And UCS is only "syncing" the data to the local kopano server. 
Kopano USC does NOT use the AD connector but that LDAP connector. 

Now, im thinking, your also running this on Xenserver? 8.1-8.2 for example? 

Can your run shortly : tcpdump -nn -vv -i NIC_ethX  |grep incorrect
Do you see lots of incorrects here? 
Because after and update from XenServer, i had lots of delays due bad checksums
on packages.

If thats the case, try this. 
https://github.com/cloudnull/XenServer-Offloading-Off


Greetz, 

Louis

> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Stefan Bauer via samba
> Verzonden: dinsdag 20 juli 2021 9:35
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] howto optimize samba/kerberos for 20k 
> requests per minute - help needed
> 
> I have a plain UCS4.4-8 (latest) with kopano inside correct. 
> Joined it 
> initially to a server 2012 windows DC.
> 
> Later migrated that windows 2012 domain to a UCS domain. 
> Kopano is still 
> a member-server.
> 
> /etc/kopano/ldap.conf contains:
> 
> ldap_uri = ldap://kopano01.procorp.local:7389/
> 
> (this is a the local ldap on kopano server)
> 
> As the local ldap seems to do not get passwords synced UCS-master to 
> local LDAP. Due to this the system is doing huge amount of 
> kerberos-connections to the UCS Samba-DC to validate the user 
> credentials all the time (25k/minute)
> 
> 09:28:55.718262 IP kopano01.procorp.local.42559 > 
> adm-ucs0.procorp.local.kerberos:? v5
> 09:28:55.723467 IP adm-ucs0.procorp.local.kerberos > 
> kopano01.procorp.local.59107:
> 09:28:55.726673 IP kopano01.procorp.local.33144 > 
> adm-ucs0.procorp.local.kerberos: Flags [S], seq 1850537116, 
> win 29200, 
> options [mss 1460,sackOK,TS val 279012852 ecr 0,nop,wscale 
> 7], length 0
> 
> Due to this, the authentication time is very bad:
> 
> root at kopano01:~# kopano-stats --system | grep ldap_avg_auth
>  ??? ldap_avg_auth??? ??? Average duration (?s) of 
> authentication made 
> to LDAP server??? ??? ??? 276250
> 
> (connection to local ldap -> kerberos request to UCS-master 
> and return)
> 
> I'm running:
> 
> Webapp 5.1.0.0+167.1
> Kopano core 8.7.20
> 
> 
> On 20.07.21 08:45, L.P.H. van Belle via samba wrote:
> > Exacly what i mean. ... Kopano, i was already thinking is kopano..
> > But there is one big difference i think. With your setup and mine
> >
> > I think you run Kopano from within UCS 4.3
> > ( and i tested also USC 5.0, no kopano there, the move to 
> licenced kopano. )
> > I run Kopano on clean Debian 10 install.
> >
> > And what version kopano/web app is running because
> > i dont see that here. IO looks normal.
> >
> > Greetz,
> >
> > Louis
> >
> >> -----Oorspronkelijk bericht-----
> >> Van: samba [mailto:samba-bounces at lists.samba.org] Namens
> >> Stefan Bauer via samba
> >> Verzonden: dinsdag 20 juli 2021 7:57
> >> Aan: samba at lists.samba.org
> >> Onderwerp: Re: [Samba] howto optimize samba/kerberos for 20k
> >> requests per minute - help needed
> >>
> >> Hi,
> >>
> >> it is kopano. They state on their website?:
> >>
> >> -----------------
> >>
> >> Note
> >>
> >> Please note that due to performance problems in Samba 4,
> >> Samba 4 is not
> >> supported as a user source for setups larger than 50 users.
> >>
> >> -----------------
> >>
> >> And that is indeed what i notice. The high amount of
> >> kerberos-requests
> >> between samba-DC and kopano server is causing a very high
> >> io-load on the
> >> samba system - renders the system unstable. Also the time an
> >> authentication request takes, is between 200-400ms.
> >>
> >> As Kopano is doing an authentication every time a user clicks
> >> a single
> >> mail and no caching is possible, it is hitting the samba
> >> system too hard.
> >>
> >>
> >> Stefan
> >>
> >>
> >> ?
> >> https://documentation.kopano.io/kopanocore_administrator_manua
> >> l/user_management.html
> >>
> >>
> >> On 19.07.21 11:27, Rowland Penny via samba wrote:
> >>> On Mon, 2021-07-19 at 11:13 +0200, Stefan Bauer via samba
wrote:
> >>>> Hi and thank you for your time.
> >>>>
> >>>> We got now the confirmation that samba 4 is not supported
by our
> >>>> software-vendor.
> >>> If I might ask, who is your software vendor and what is the
> >> software ?
> >>> In most cases, when a supplier says they do not support
> >> Samba 4, they
> >>> do support AD.
> >> -- 
> >> To unsubscribe from this list go to the following URL and read the
> >> instructions:  https://lists.samba.org/mailman/options/samba
> >>
> >>
> >
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
> 
>

Well, my personal view on UCS5 is .. 

IF you setup a new environment and you use only UCS, that its a nice thing to
use.
But if you know how to setup it manual, your servers perform way faster en USC
there setup.

At least, UCS5 is debian10 

Now for the Topic poster. 
Im assuming it webapp that borking his performance, tuning apache/nigx will also
help,
But maybe this is something to offload ldap queries. 
https://github.com/Kopano-dev/ucs-oidc-webapp 
Switch to SSO and OIDC for authentication
! Note, i have NOT tested this. 

Also, if your brave and now married with USC, then i suggest, 
Use my script https://github.com/thctlo/Kopano  and setup local repo for kopano.
Get it, Run it and Setup manually.  It removes the need to use dpkg -i *.deb,
that installs to much..
Just, keep in mind these are development packages, so if everything is running
perfectly,
Dont upgrade, and if you do, do it in test environment. 

Also, if your running kopano 4.3 in samba 4, have you have had a very good look
at you logs.
And how kopano (in UCS 4.3) is linked to AD, its simpley a bad setup. 

If you want to fast test, use the kopano docker packages from Zokradonh.
https://github.com/zokradonh/kopano-docker 

I hope i gave you some ideas..

I dropped UCS (again), i (again) gave it a try but, you get to much overload
stuff in my opinion.
Its a nice product, just, not for me. 

And for adjustint the samba4 indexing, totaly forgot to mention that. Its
something you can try.
ldbedit -v -P -s base -b @INDEXLIST -H /var/lib/samba/private/sam.ldb 



Greetz, 

Louis

> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Rowland Penny via samba
> Verzonden: dinsdag 20 juli 2021 9:07
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] howto optimize samba/kerberos for 20k 
> requests per minute - help needed
> 
> On Tue, 2021-07-20 at 08:45 +0200, L.P.H. van Belle via samba wrote:
> > Exacly what i mean. ... Kopano, i was already thinking is kopano.. 
> > But there is one big difference i think. With your setup and mine
> > 
> > I think you run Kopano from within UCS 4.3 
> > ( and i tested also USC 5.0, no kopano there, the move to licenced
> > kopano. ) 
> > I run Kopano on clean Debian 10 install.
> > 
> > And what version kopano/web app is running because
> > i dont see that here. IO looks normal. 
> > 
> 
> I downloaded UCS5 and tried to install it in a VM. The first attempt
> failed when I tried to set up a DC, there is (apparently) no way to
> change the dns and when it did finish the set up, there was no Samba!
> 
> Second attempt, I went for what it described as a join to Active
> Directory, I ended up with what I call a Unix domain member, but with
> what I would call a borked smb.conf, it was using the ldap backend for
> the '*' domain and the nss backend for the 'DOMAIN' domain.
There were
> numerous other parameters that I wouldn't use.
> 
> All in all, I wouldn't recomend UCS5 to my worst enemy, but this is
> just my personal opinion.
> 
> Rowland
> 
> 
> 
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
> 
>