Jared B. Griffith
2007-Dec-13 19:36 UTC
[Fedora-directory-users] LDAP Accounts for large website
I was wondering if anyone here has ever used LDAP for a website, that will potentially have millions of LDAP accounts. If so, are you experiencing slow query responses or other issues? If you were experiencing slow query responses, and were able to rectify the issue, how did you do this? We are currently using FDS for our main website for customer accounts. We currently have over 52,000 accounts in LDAP and have only been using this for 3 months. We are now experiencing extreme slow down in query response when getting customer data into and out of the LDAP servers. Any help would be greatly appreciated. -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542
Chris G. Sellers
2007-Dec-13 19:47 UTC
Re: [Fedora-directory-users] LDAP Accounts for large website
Performance is often impacted greatly by 1) Memory on the LDAP server. Make sure you can store as much of your directory data store in RAM for fast access 2) Indexing. Make sure attributes that you search on freqently are indexed. Also, limit what fields you search on to avoid having a heavy indexing tax. 3) Make sure your network connections are stable, and your not connecting on a 100MB half duplex connection while your network equiptment is expecting a full duplex connection. Once you have auditing those situations, please check your performance again. Sellers 50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core 2 system with 1GB-2GB of RAM should perform okay. On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote:> I was wondering if anyone here has ever used LDAP for a website, > that will potentially have millions of LDAP accounts. > If so, are you experiencing slow query responses or other issues? > If you were experiencing slow query responses, and were able to > rectify the issue, how did you do this? > We are currently using FDS for our main website for customer > accounts. We currently have over 52,000 accounts in LDAP and have > only been using this for 3 months. We are now experiencing extreme > slow down in query response when getting customer data into and out > of the LDAP servers. > Any help would be greatly appreciated. > > -- > - Thank you, > - Jared B. Griffith > - Farheap Solutions, Inc. > - Lead Systems Administrator > - California IT Department > - Email - jared.griffith@farheap.com > - Phone - 949.417.1500 ext. 266 > - Cell Phone - 949.910.6542 > -- > Fedora-directory-users mailing list > Fedora-directory-users@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-users______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com
Steven Jones
2007-Dec-13 20:04 UTC
RE: [Fedora-directory-users] LDAP Accounts for large website
Hi, For applications there are lots of ways to improve performance....with no information supplied there is no where to start... Ie, look at your disk i/o...use iostat are the disks LDAP sits on at 100% utilisation? Even 80% is not good....if so make bigger raid sets and/or distribute the database over differing raid controllers and disk sets....get the utilisation down to 30%.... Is you memory full and you are into swapping? Make sure the issue is not disk i/o bottlenecks....if not add more ram.....make sure your swap is not exhausted....add more swap until you can get more ram... Are your switches or routers too slow? Check their utilisation..... CPU maxed out? Where is it going? On wait? Check disk i/o....consider dual Quad core machines.....or even 4 way quad core machines....Dell R900s are seriously grunty boxes, have 16 cores and hold 64gig of ram cheaply, then attach them to a SAN.... regards Steven Jones Senior Linux/Unix/San/Vmware System Administrator APG -Technology Integration Team Victoria University of Wellington Phone: +64 4 463 6272 ________________________________ From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Jared B. Griffith Sent: Friday, 14 December 2007 8:37 a.m. To: fedora-directory-users Subject: [Fedora-directory-users] LDAP Accounts for large website I was wondering if anyone here has ever used LDAP for a website, that will potentially have millions of LDAP accounts. If so, are you experiencing slow query responses or other issues? If you were experiencing slow query responses, and were able to rectify the issue, how did you do this? We are currently using FDS for our main website for customer accounts. We currently have over 52,000 accounts in LDAP and have only been using this for 3 months. We are now experiencing extreme slow down in query response when getting customer data into and out of the LDAP servers. Any help would be greatly appreciated. -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542
Jared B. Griffith
2007-Dec-13 20:04 UTC
Re: [Fedora-directory-users] LDAP Accounts for large website
Here are the specs on the server: 2 x 2.0Ghz Intel Dual Core Xeon 4 x 1Gb Registered ECC RAM 2 x 74Gb Western Digital Raptors Given that, the hardware issue should be more than sufficient. Network connections are Gigabit full duplex throughout our cage network. How would we go about indexing the attributes? Me and another sys admin have a distinct feeling that it is an issue with the query, but they are going to point blame at us, so we want to make sure that we are golden before saying it''s the code. ----- Original Message ----- From: "Chris G. Sellers" <chris.sellers@nitle.org> To: "Jared B. Griffith" <jared.griffith@farheap.com>, "General discussion list for the Fedora Directory server project." <fedora-directory-users@redhat.com> Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800) America/Los_Angeles Subject: Re: [Fedora-directory-users] LDAP Accounts for large website Performance is often impacted greatly by 1) Memory on the LDAP server. Make sure you can store as much of your directory data store in RAM for fast access 2) Indexing. Make sure attributes that you search on freqently are indexed. Also, limit what fields you search on to avoid having a heavy indexing tax. 3) Make sure your network connections are stable, and your not connecting on a 100MB half duplex connection while your network equiptment is expecting a full duplex connection. Once you have auditing those situations, please check your performance again. Sellers 50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core 2 system with 1GB-2GB of RAM should perform okay. On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote: I was wondering if anyone here has ever used LDAP for a website, that will potentially have millions of LDAP accounts. If so, are you experiencing slow query responses or other issues? If you were experiencing slow query responses, and were able to rectify the issue, how did you do this? We are currently using FDS for our main website for customer accounts. We currently have over 52,000 accounts in LDAP and have only been using this for 3 months. We are now experiencing extreme slow down in query response when getting customer data into and out of the LDAP servers. Any help would be greatly appreciated. -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542 -- Fedora-directory-users mailing list Fedora-directory-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users ______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542
Chris G. Sellers
2007-Dec-13 20:12 UTC
Re: [Fedora-directory-users] LDAP Accounts for large website
if your queries are something like ldapsearch ''(&(uid=%)(cn=%*)(objectClass=account))'' then you would want to make sure uid, cn are indexed. maybe even objectClass. It depends on what you are searching those are the attributes that you index. Read up in Indexing. It''s very much like a database, you have to index the attributes you search often. http://www.redhat.com/docs/manuals/dir-server/ag/7.1/index1.html#996824 Sellers On Dec 13, 2007, at 3:04 PM, Jared B. Griffith wrote:> > Here are the specs on the server: > 2 x 2.0Ghz Intel Dual Core Xeon > 4 x 1Gb Registered ECC RAM > 2 x 74Gb Western Digital Raptors > Given that, the hardware issue should be more than sufficient. > Network connections are Gigabit full duplex throughout our cage > network. > > How would we go about indexing the attributes? > > Me and another sys admin have a distinct feeling that it is an issue > with the query, but they are going to point blame at us, so we want > to make sure that we are golden before saying it''s the code. > > > > > > > > ----- Original Message ----- > From: "Chris G. Sellers" <chris.sellers@nitle.org> > To: "Jared B. Griffith" <jared.griffith@farheap.com>, "General > discussion list for the Fedora Directory server project." <fedora-directory-users@redhat.com > > > Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800) America/ > Los_Angeles > Subject: Re: [Fedora-directory-users] LDAP Accounts for large website > > Performance is often impacted greatly by > > 1) Memory on the LDAP server. Make sure you can store as much of > your directory data store in RAM for fast access > 2) Indexing. Make sure attributes that you search on freqently are > indexed. Also, limit what fields you search on to avoid having a > heavy indexing tax. > 3) Make sure your network connections are stable, and your not > connecting on a 100MB half duplex connection while your network > equiptment is expecting a full duplex connection. > > Once you have auditing those situations, please check your > performance again. > > Sellers > > 50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz > Core 2 system with 1GB-2GB of RAM should perform okay. > > > On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote: > > I was wondering if anyone here has ever used LDAP for a website, > that will potentially have millions of LDAP accounts. > If so, are you experiencing slow query responses or other issues? > If you were experiencing slow query responses, and were able to > rectify the issue, how did you do this? > We are currently using FDS for our main website for customer > accounts. We currently have over 52,000 accounts in LDAP and have > only been using this for 3 months. We are now experiencing extreme > slow down in query response when getting customer data into and out > of the LDAP servers. > Any help would be greatly appreciated. > > -- > - Thank you, > - Jared B. Griffith > - Farheap Solutions, Inc. > - Lead Systems Administrator > - California IT Department > - Email - jared.griffith@farheap.com > - Phone - 949.417.1500 ext. 266 > - Cell Phone - 949.910.6542 > -- > Fedora-directory-users mailing list > Fedora-directory-users@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-users > > ______________________________________________ > Chris G. Sellers | NITLE Technology > 734.661.2318 | chris.sellers@nitle.org > AIM: imthewherd | GTalk: cgseller@gmail.com > > > > -- > - Thank you, > - Jared B. Griffith > - Farheap Solutions, Inc. > - Lead Systems Administrator > - California IT Department > - Email - jared.griffith@farheap.com > - Phone - 949.417.1500 ext. 266 > - Cell Phone - 949.910.6542______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com
Jared B. Griffith
2007-Dec-13 20:14 UTC
Re: [Fedora-directory-users] LDAP Accounts for large website
Resources are fine, and if we really need a 16 way machine this early in the game, then we should just forgoe all of this and stick it in our regular db that sits on a 16way with 32g of mem and a fiber channel san attached to it. iostat is fine, 2% usage CPU is at 0, RAM is not even being used, so of course there is no swap going on. I think we need to do some of this indexing on particular attributes, then address the possible query issue that we think is happening. ----- Original Message ----- From: "Steven Jones" <Steven.Jones@vuw.ac.nz> To: "General discussion list for the Fedora Directory server project." <fedora-directory-users@redhat.com> Sent: Thursday, December 13, 2007 12:04:36 PM (GMT-0800) America/Los_Angeles Subject: RE: [Fedora-directory-users] LDAP Accounts for large website Hi, For applications there are lots of ways to improve performance….with no information supplied there is no where to start… Ie, look at your disk i/o…use iostat are the disks LDAP sits on at 100% utilisation? Even 80% is not good….if so make bigger raid sets and/or distribute the database over differing raid controllers and disk sets….get the utilisation down to 30%.... Is you memory full and you are into swapping? Make sure the issue is not disk i/o bottlenecks….if not add more ram…..make sure your swap is not exhausted….add more swap until you can get more ram… Are your switches or routers too slow? Check their utilisation….. CPU maxed out? Where is it going? On wait? Check disk i/o….consider dual Quad core machines…..or even 4 way quad core machines….Dell R900s are seriously grunty boxes, have 16 cores and hold 64gig of ram cheaply, then attach them to a SAN…. regards Steven Jones Senior Linux/Unix/San/Vmware System Administrator APG -Technology Integration Team Victoria University of Wellington Phone: +64 4 463 6272 From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Jared B. Griffith Sent: Friday, 14 December 2007 8:37 a.m. To: fedora-directory-users Subject: [Fedora-directory-users] LDAP Accounts for large website I was wondering if anyone here has ever used LDAP for a website, that will potentially have millions of LDAP accounts. If so, are you experiencing slow query responses or other issues? If you were experiencing slow query responses, and were able to rectify the issue, how did you do this? We are currently using FDS for our main website for customer accounts. We currently have over 52,000 accounts in LDAP and have only been using this for 3 months. We are now experiencing extreme slow down in query response when getting customer data into and out of the LDAP servers. Any help would be greatly appreciated. -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542 -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542
Steven Jones
2007-Dec-13 20:31 UTC
RE: [Fedora-directory-users] LDAP Accounts for large website
Operating system? CPU and Ram look good.... Iostat? Hi, Where is your basic fault finding data? We run a database backend for spam control and find that cleaning up / indexing the database has dramatic effects....like after 6 months our Dell 6850 is screwed....clean it up and its disk i/o is 2% again....but this shows up under iostat....using mrtg you can see the graph climbing steadily over the months..... A pair of SATA disks (I assume raid 1) is not very fast (on board raid?...shudder...)....also these are SATA, SATA sucks for random i/o and guess what you have a database doing random i/o........ In terms of code, yes it is often the code at fault. Somehow developers who have written sucky code expect sys admins to spend serious time and money on hardware compensating for their bad code....it does not work. This could be the hard thing to prove....ie....is your disk i/o inadequate or is their code so bad its causing the i/o! If its disk i/o...... Generally, you are going to be spending most of your time reading from disk....so you need to be optimising for reads....raid5 is ideal but testing will prove this....LDAP can be distributed over disk sets...so would 4 disks in two raid1s out perform a R5 3+1? From my experience a r5 3+1 for databases is 20% faster than raid 1s.... I suspect you are very budget conscious and have no-name white boxes....so articles like this, http://www.tomshardware.com/2007/11/30/more_serial_raid_controllers_from _amcc/ Can give you good pointers as what to get....generally avoid SATA, look at SAS....for databases the LSIMegaraid SAS 8888ELP in a raid 5 looks worth buying....maybe a very small disk strip is in order so small raid5 sets.... You are ahead of me at present I''m still piloting FDS.... Regards Steven Jones Senior Linux/Unix/San/Vmware System Administrator APG -Technology Integration Team Victoria University of Wellington Phone: +64 4 463 6272 ________________________________ From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Jared B. Griffith Sent: Friday, 14 December 2007 9:05 a.m. To: Chris G. Sellers Cc: General discussion list for the Fedora Directory server project. Subject: Re: [Fedora-directory-users] LDAP Accounts for large website Here are the specs on the server: 2 x 2.0Ghz Intel Dual Core Xeon 4 x 1Gb Registered ECC RAM 2 x 74Gb Western Digital Raptors Given that, the hardware issue should be more than sufficient. Network connections are Gigabit full duplex throughout our cage network. How would we go about indexing the attributes? Me and another sys admin have a distinct feeling that it is an issue with the query, but they are going to point blame at us, so we want to make sure that we are golden before saying it''s the code. ----- Original Message ----- From: "Chris G. Sellers" <chris.sellers@nitle.org> To: "Jared B. Griffith" <jared.griffith@farheap.com>, "General discussion list for the Fedora Directory server project." <fedora-directory-users@redhat.com> Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800) America/Los_Angeles Subject: Re: [Fedora-directory-users] LDAP Accounts for large website Performance is often impacted greatly by 1) Memory on the LDAP server. Make sure you can store as much of your directory data store in RAM for fast access 2) Indexing. Make sure attributes that you search on freqently are indexed. Also, limit what fields you search on to avoid having a heavy indexing tax. 3) Make sure your network connections are stable, and your not connecting on a 100MB half duplex connection while your network equiptment is expecting a full duplex connection. Once you have auditing those situations, please check your performance again. Sellers 50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core 2 system with 1GB-2GB of RAM should perform okay. On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote: I was wondering if anyone here has ever used LDAP for a website, that will potentially have millions of LDAP accounts. If so, are you experiencing slow query responses or other issues? If you were experiencing slow query responses, and were able to rectify the issue, how did you do this? We are currently using FDS for our main website for customer accounts. We currently have over 52,000 accounts in LDAP and have only been using this for 3 months. We are now experiencing extreme slow down in query response when getting customer data into and out of the LDAP servers. Any help would be greatly appreciated. -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542 -- Fedora-directory-users mailing list Fedora-directory-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users ______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542
Steven Jones
2007-Dec-13 20:35 UTC
RE: [Fedora-directory-users] LDAP Accounts for large website
Hi, Sounds like the box is asleep....if the i/o is 2% this suggests nothing is getting to the box.....not being a LDAP expert, but if that was a "straight" database like Oracle or MYSQL I''d think that the problem was external to the LDAP server....if the indexing was bad, I''d expect high disk i/o as a result/indicator.... Be interesting to see if indexing does a thing.... regards Steven Jones Senior Linux/Unix/San/Vmware System Administrator APG -Technology Integration Team Victoria University of Wellington Phone: +64 4 463 6272 ________________________________ From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Jared B. Griffith Sent: Friday, 14 December 2007 9:15 a.m. To: General discussion list for the Fedora Directory server project. Subject: Re: [Fedora-directory-users] LDAP Accounts for large website Resources are fine, and if we really need a 16 way machine this early in the game, then we should just forgoe all of this and stick it in our regular db that sits on a 16way with 32g of mem and a fiber channel san attached to it. iostat is fine, 2% usage CPU is at 0, RAM is not even being used, so of course there is no swap going on. I think we need to do some of this indexing on particular attributes, then address the possible query issue that we think is happening. ----- Original Message ----- From: "Steven Jones" <Steven.Jones@vuw.ac.nz> To: "General discussion list for the Fedora Directory server project." <fedora-directory-users@redhat.com> Sent: Thursday, December 13, 2007 12:04:36 PM (GMT-0800) America/Los_Angeles Subject: RE: [Fedora-directory-users] LDAP Accounts for large website Hi, For applications there are lots of ways to improve performance....with no information supplied there is no where to start... Ie, look at your disk i/o...use iostat are the disks LDAP sits on at 100% utilisation? Even 80% is not good....if so make bigger raid sets and/or distribute the database over differing raid controllers and disk sets....get the utilisation down to 30%.... Is you memory full and you are into swapping? Make sure the issue is not disk i/o bottlenecks....if not add more ram.....make sure your swap is not exhausted....add more swap until you can get more ram... Are your switches or routers too slow? Check their utilisation..... CPU maxed out? Where is it going? On wait? Check disk i/o....consider dual Quad core machines.....or even 4 way quad core machines....Dell R900s are seriously grunty boxes, have 16 cores and hold 64gig of ram cheaply, then attach them to a SAN.... regards Steven Jones Senior Linux/Unix/San/Vmware System Administrator APG -Technology Integration Team Victoria University of Wellington Phone: +64 4 463 6272 ________________________________ From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Jared B. Griffith Sent: Friday, 14 December 2007 8:37 a.m. To: fedora-directory-users Subject: [Fedora-directory-users] LDAP Accounts for large website I was wondering if anyone here has ever used LDAP for a website, that will potentially have millions of LDAP accounts. If so, are you experiencing slow query responses or other issues? If you were experiencing slow query responses, and were able to rectify the issue, how did you do this? We are currently using FDS for our main website for customer accounts. We currently have over 52,000 accounts in LDAP and have only been using this for 3 months. We are now experiencing extreme slow down in query response when getting customer data into and out of the LDAP servers. Any help would be greatly appreciated. -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542 -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542
Satish Chetty
2007-Dec-13 20:42 UTC
Re: [Fedora-directory-users] LDAP Accounts for large website
Jared, I would check to see if the problem if it is on the LDAP or website. Ex. LDAP response may be good from the ldap host using ldapsearch. However the LDAP response by the ldap plugin/api from the webserver/appserver may not. Some additional steps I would do... * Run the ldap query using ldapsearch on the ldap host and then machine on same subnet and then from webserver/appserver machine. Both with and without website load. This will help me identify if the problem is on the network or LDAP server itself or load. * Check on indexing, number of concurrent requests when server is fully loaded, FDs, swap space, TCP/IP settings etc. The load may be high enough that you need additional replicas or even masters. With additional masters you need to be watchful on how load balancing works when writing to different masters. -Satish. Jared B. Griffith wrote:> I was wondering if anyone here has ever used LDAP for a website, that > will potentially have millions of LDAP accounts. > If so, are you experiencing slow query responses or other issues? > If you were experiencing slow query responses, and were able to rectify > the issue, how did you do this? > We are currently using FDS for our main website for customer accounts. > We currently have over 52,000 accounts in LDAP and have only been using > this for 3 months. We are now experiencing extreme slow down in query > response when getting customer data into and out of the LDAP servers. > Any help would be greatly appreciated. > > -- > - Thank you, > - Jared B. Griffith > - Farheap Solutions, Inc. > - Lead Systems Administrator > - California IT Department > - Email - jared.griffith@farheap.com > - Phone - 949.417.1500 ext. 266 > - Cell Phone - 949.910.6542 > > > ------------------------------------------------------------------------ > > -- > Fedora-directory-users mailing list > Fedora-directory-users@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-users
Jared B. Griffith
2007-Dec-13 21:31 UTC
Re: [Fedora-directory-users] LDAP Accounts for large website
Centos 4 - would have been Gentoo 2007 if I could have gotten Directory Server working correctly on it prior to having to have it up and running. Iostat - 2 % Raid 1 on the drives, SATA 3 drives. ----- Original Message ----- From: "Steven Jones" <Steven.Jones@vuw.ac.nz> To: "General discussion list for the Fedora Directory server project." <fedora-directory-users@redhat.com> Sent: Thursday, December 13, 2007 12:31:11 PM (GMT-0800) America/Los_Angeles Subject: RE: [Fedora-directory-users] LDAP Accounts for large website Operating system? CPU and Ram look good…. Iostat? Hi, Where is your basic fault finding data? We run a database backend for spam control and find that cleaning up / indexing the database has dramatic effects….like after 6 months our Dell 6850 is screwed….clean it up and its disk i/o is 2% again….but this shows up under iostat….using mrtg you can see the graph climbing steadily over the months….. A pair of SATA disks (I assume raid 1) is not very fast (on board raid?...shudder…)….also these are SATA, SATA sucks for random i/o and guess what you have a database doing random i/o…….. In terms of code, yes it is often the code at fault. Somehow developers who have written sucky code expect sys admins to spend serious time and money on hardware compensating for their bad code….it does not work. This could be the hard thing to prove….ie….is your disk i/o inadequate or is their code so bad its causing the i/o! If its disk i/o…… Generally, you are going to be spending most of your time reading from disk….so you need to be optimising for reads….raid5 is ideal but testing will prove this….LDAP can be distributed over disk sets…so would 4 disks in two raid1s out perform a R5 3+1? From my experience a r5 3+1 for databases is 20% faster than raid 1s…. I suspect you are very budget conscious and have no-name white boxes….so articles like this, http://www.tomshardware.com/2007/11/30/more_serial_raid_controllers_from_amcc/ Can give you good pointers as what to get….generally avoid SATA, look at SAS….for databases the LSIMegaraid SAS 8888ELP in a raid 5 looks worth buying….maybe a very small disk strip is in order so small raid5 sets…. You are ahead of me at present I’m still piloting FDS…. Regards Steven Jones Senior Linux/Unix/San/Vmware System Administrator APG -Technology Integration Team Victoria University of Wellington Phone: +64 4 463 6272 From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Jared B. Griffith Sent: Friday, 14 December 2007 9:05 a.m. To: Chris G. Sellers Cc: General discussion list for the Fedora Directory server project. Subject: Re: [Fedora-directory-users] LDAP Accounts for large website Here are the specs on the server: 2 x 2.0Ghz Intel Dual Core Xeon 4 x 1Gb Registered ECC RAM 2 x 74Gb Western Digital Raptors Given that, the hardware issue should be more than sufficient. Network connections are Gigabit full duplex throughout our cage network. How would we go about indexing the attributes? Me and another sys admin have a distinct feeling that it is an issue with the query, but they are going to point blame at us, so we want to make sure that we are golden before saying it''s the code. ----- Original Message ----- From: "Chris G. Sellers" <chris.sellers@nitle.org> To: "Jared B. Griffith" <jared.griffith@farheap.com>, "General discussion list for the Fedora Directory server project." <fedora-directory-users@redhat.com> Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800) America/Los_Angeles Subject: Re: [Fedora-directory-users] LDAP Accounts for large website Performance is often impacted greatly by 1) Memory on the LDAP server. Make sure you can store as much of your directory data store in RAM for fast access 2) Indexing. Make sure attributes that you search on freqently are indexed. Also, limit what fields you search on to avoid having a heavy indexing tax. 3) Make sure your network connections are stable, and your not connecting on a 100MB half duplex connection while your network equiptment is expecting a full duplex connection. Once you have auditing those situations, please check your performance again. Sellers 50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core 2 system with 1GB-2GB of RAM should perform okay. On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote: I was wondering if anyone here has ever used LDAP for a website, that will potentially have millions of LDAP accounts. If so, are you experiencing slow query responses or other issues? If you were experiencing slow query responses, and were able to rectify the issue, how did you do this? We are currently using FDS for our main website for customer accounts. We currently have over 52,000 accounts in LDAP and have only been using this for 3 months. We are now experiencing extreme slow down in query response when getting customer data into and out of the LDAP servers. Any help would be greatly appreciated. -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542 -- Fedora-directory-users mailing list Fedora-directory-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users ______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542 -- - Thank you, - Jared B. Griffith - Farheap Solutions, Inc. - Lead Systems Administrator - California IT Department - Email - jared.griffith@farheap.com - Phone - 949.417.1500 ext. 266 - Cell Phone - 949.910.6542