Hello, I'm running Samba 3.0.6 PDC with OpenLDAP 2.1.25 backend on a Linux machine with RedHat 3.0 ES installed. This is a large installation with separate Samba BDC and 2 file servers. The BDC server uses a replica LDAP server, working as slave for the master LDAP server installed at PDC. The number of domain accounts is about 1850 and at the moment about 500 machines are added to the Samba domain. The number of machines increased slowly since April and for the last few weeks we observed large delays during the domain logons. The logon process for some Windows machines takes as much as 10-20 minutes (!) For most of the users these times are of course unacceptable. Most of the users start their work and logon to the domain between 7:30-8:30 AM. Within these hours the load of the PDC server sometimes exceeds 100-120. About 90% of the CPU time is utilized by slapd. The PDC/BDC machines are HP DL-380 server with single Xeon CPU 2.80GHz, 2,5 GB of RAM, no swap and with Gigabit Ethernet interface. When I turned on the high debug level for both Samba and OpenLDAP daemons and the problem is that during the processing of the logon script Samba orders the LDAP backend to perform multiple searches for all the domain users and repeats it 3 or 4 times. This gives about 8-9 _thousand_ of full LDAP directory searches for single logon session! The small part of slapd debug file follows: Sep 27 15:01:09 umwsap11 slapd[16930]: conn=458 op=65 SRCH base="dc=XX Company,dc=pl" scope=2 filter="(&(uid=umwadd01)(objectClass=sambaSamAccount))" Sep 27 15:01:09 umwsap11 slapd[16930]: conn=458 op=65 SEARCH RESULT tag=101 err=0 nentries=1 textSep 27 15:01:09 umwsap11 slapd[16930]: conn=458 op=66 SRCH base="dc=XX Company,dc=pl" scope=2 filter="(&(uid=umwadd02)(objectClass=sambaSamAccount))" Sep 27 15:01:09 umwsap11 slapd[16930]: conn=458 op=66 SEARCH RESULT tag=101 err=0 nentries=1 textSep 27 15:01:09 umwsap11 slapd[16930]: conn=458 op=67 SRCH base="dc=XX Company,dc=pl" scope=2 filter="(&(uid=umwadd03)(objectClass=sambaSamAccount))" Sep 27 15:01:09 umwsap11 slapd[16930]: conn=458 op=67 SEARCH RESULT tag=101 err=0 nentries=1 text ... and so on, for some reason every user must be found in LDAP several times. All these searches are performed during the logon script processing. Since many of our users are still using Win98 workstations, the system "hangs" for them for several minutes with empty screen and only a logon script window open. What's more confusing, for some of the domain users only about 60 LDAP searches are performed and they are able to log on to the domain in a few seconds. I tried to compare their exported ldif data with users which experience the delays, but there's nothing exceptional, only their names, UIDs and SIDs are different. The problem does not depend on the operating system of the workstation - we've tested Win98, NT, W2000 and XP systems. It seems to be rather user-centric. I tried to increase OpenLDAP and nscd performance by setting the thread number up to 256 and increasing the cache size, but this gives only a small improvement. The indexes in slapd.conf are defined as described in the Samba docs: index default sub index objectClass eq index uidNumber,gidNumber eq index memberUid eq index cn,sn,uid,displayName pres,sub,eq index mail,givenname eq,subinitial index nisMapName,nisMapEntry eq,pres,sub index homeDirectory,sambaLogonScript eq index sambaSID eq index sambaPrimaryGroupSID eq index sambaDomainName eq sizelimit -1 cachesize 100000 dbcachesize 15000000 threads 256 We have BDC server configured as the second logon server, but for some reason only small number of workstation chooses this server as logon server. Perhaps I should increase the "os level" for the BDC from 33 to 255, as it is configured for the PDC? The smb.conf of the PDC server follows: [global] workgroup = XXCOMP security = user server string = XX Company - PDC passdb backend = ldapsam:ldap://127.0.0.1 idmap backend = ldap:ldap://127.0.0.1 idmap uid = 40000-50000 idmap gid = 40000-50000 log level = 1 log file = /var/log/samba/log.%m max log size = 500 time server = Yes socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=8192 SO_SNDBUF=8192 logon path logon drive = K: logon home = \\fileserv02\homes\%U #logon script = %U.bat domain logons = Yes os level = 255 local master = Yes preferred master = Yes domain master = Yes wins proxy = Yes wins support = Yes ldap suffix = dc=XX Company,dc=pl ldap group suffix = ou=groups ldap user suffix = ou=people ldap idmap suffix = ou=idmap,dc=XX Company,dc=pl ldap machine suffix = ou=machines ldap admin dn = cn=Manager,dc=XX Company,dc=pl ldap ssl = no ldap passwd sync = Yes remote browse sync = 10.255.255.255 130.130.255.255 printing = cups hide unreadable = Yes nt acl support = Yes admin users = "Domain Admins" name resolve order = lmhosts wins hosts bcast ldap timeout = 15 I tried to use the idmap feature od Samba, but for some reason after creating the "ou=idmap,dc=XX Company,dc=pl" container in LDAP, Samba does not populate it with SID-GID mappings. Perhaps this is the root cause of our problem. The whole Samba domain worked properly without these logon delays for several months. When the number of users and workstations was small, no performance problems occcured. Now we have serious problem with about 1/3 of the intended number of domain workstations (500 of ~1500). Unless I find the solution, our management will probably decide to migrate from Samba to Active Directory... Thanks in advance, Tomasz Finke
> Sep 27 15:01:09 umwsap11 slapd[16930]: conn=458 op=65 SRCH base="dc=XX > Company,dc=pl" scope=2 > filter="(&(uid=umwadd01)(objectClass=sambaSamAccount))" >> ldap suffix = dc=XX Company,dc=pl > ldap group suffix = ou=groups > ldap user suffix = ou=people > ldap idmap suffix = ou=idmap,dc=XX Company,dc=pl > ldap machine suffix = ou=machinesThese entries make me think you could probably speed things up a bit with a tighter search scope. It looks like you're searching the whole DIT every time since you've got your machine and user accounts split up. I'm assuming you also have nss configured to search dc=XX Company,dc=pl?sub. I'd suggest either merging the user and machine OUs or perhaps putting both of them in a container OU you can search in, rather than doing the whole LDAP tree. True, that won't solve the multiple searches problem, but it should help along the speed of the searches that it does do. -- Paul Gienger Office: 701-281-1884 Applied Engineering Inc. Information Systems Consultant Fax: 701-281-1322 URL: www.ae-solutions.com mailto: pgienger@ae-solutions.com
Marcel de Riedmatten
2004-Oct-08 08:24 UTC
[Samba] Samba 3.0.6 and OpenLDAP performance problem
Le jeu 07/10/2004 ? 23:05, Tomasz Finke a ?crit :> Marcel de Riedmatten wrote: > > > You aren't running winbind aren't you ? > > No, I'm not, just slapd, nscd and Samba.I have looked in my log and i see something similar as you, except this append after the logon script is closed. I see at least three time the same enumeration request: Oct 8 09:56:08 sarge slapd[25437]: conn=266107 op=401 SRCH base="dc=nofida,dc=ch" scope=2 filter="(&(uid=*)(objectClass=sambaSamAccount))" Oct 8 09:56:08 sarge slapd[25437]: conn=266107 op=401 SRCH attr=uid uidNumber gidNumber homeDirectory sambaPwdLastSet sambaPwdCanChange sambaPwdMustChange sambaLogonTime sambaLogoffTime sambaKickoffTime cn displayName sambaHomeDrive sambaHomePath sambaLogonScript sambaProfilePath description sambaUserWorkstations sambaSID sambaPrimaryGroupSID sambaLMPassword sambaNTPassword sambaDomainName objectClass sambaAcctFlags sambaMungedDial sambaBadPasswordCount sambaBadPasswordTime Oct 8 09:56:08 sarge slapd[25437]: conn=266107 op=401 SEARCH RESULT tag=101 err=0 nentries=37 text With 37 entries i don't see performance issue. This is with samba 3.05 (debian). I'all check later if this is realy has to do with the logon process. Another question: have you replicated your ldap server ? -- Marcel -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e=2E?Url : http://lists.samba.org/archive/samba/attachments/20041008/d16b39a3/attachment.bin
Marcel de Riedmatten
2004-Oct-08 09:37 UTC
[Samba] Samba 3.0.6 and OpenLDAP performance problem
Le ven 08/10/2004 ? 11:01, Tomasz Finke a ?crit :> Marcel de Riedmatten wrote: > > > Another question: have you replicated your ldap server ? > > Yes, I have BDC server with Samba and slave slapd installed. But > more than 90% of users choose PDC as their logon server. The > "os level" at PDC is set to 255 and on the BDC to 33. Perhaps > I should set equal values for both servers?I made some search on that one. You should probably use a 2 PDC setup like the one discribed at http://www.unav.es/cti/ldap-smb/smb-ldap-3-howto.html toward the end of the page. Both PDC have domain master = yes local master = yes preferred master = yes os level = 255 What change is the netbios name and the ldapsam entry doesn't have the same order. -- Marcel de Riedmatten -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message =?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e=2E?Url : http://lists.samba.org/archive/samba/attachments/20041008/02c758d7/attachment.bin
Andrew Bartlett
2004-Oct-08 13:02 UTC
[Samba] Samba 3.0.6 and OpenLDAP performance problem
On Wed, 2004-10-06 at 21:06, Tomasz Finke wrote:> Hello, > > I'm running Samba 3.0.6 PDC with OpenLDAP 2.1.25 backend on a Linux > machine with RedHat 3.0 ES installed. This is a large installation > with separate Samba BDC and 2 file servers. The BDC server uses a > replica LDAP server, working as slave for the master LDAP server > installed at PDC. The number of domain accounts is about 1850 and > at the moment about 500 machines are added to the Samba domain. The > number of machines increased slowly since April and for the last few > weeks we observed large delays during the domain logons. > > The logon process for some Windows machines takes as much as 10-20 > minutes (!) For most of the users these times are of course > unacceptable.I looked at deploying Samba 3.0.6 at my site, and found that I could not upgrade past the particular Samba 3.0.3 pre-release that we had at the time. I found that certain windows clients would want to know who was in certain groups, and if there were a lot of people in those groups, then all hell broke loose. On the samba-technical list, we have been looking at one potential solution, but I think the patch needs more work to make it robust. Part of the problem is that it looked for 'primary' group members, by scanning the entire password database. This, and possibly the gid->sid lookups, cause the performance issues. At one point I thought that get_sid_list_of_group() in groupdb/mapping.c was the problem, but it's unused now, so you could try current SVN. Really, we need to look at the incoming SMB requests, and what LDAP traffic it produces. With that data, we should be able to pin down what's killing things at your site (which may very well be different to what my problems were). Andrew Bartlett -- Andrew Bartlett abartlet@samba.org Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College abartlet@hawkerc.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.samba.org/archive/samba/attachments/20041008/87b80e83/attachment.bin