I am still playing around with load/stress testing my FDS install. I am using slamd to run the tests, and I am running the basic load, and the basic search tests against my test boxes, The slamd client and the FDS server are on identical hardware running RHEL4 with 2G memory, and 2 Intel Xeon 3.6GHz processors. The issue I am seeing is that I am getting "cannot connect to the ldap server" errors once I push the "threads per client" past 2400. My question is am I hitting a limit on the OS (max tcp connections)? Or a FDS limit? My assumption was that given appropriate hardware, and proper configuration, I could expect FDS to handle more than 2400 concurrent connections. Any insights as to what I might be missing would be greatly appreciated. Thanks FDS Config: Performance settings are: Time Limit: 3600 Max file descriptors: 65535 Database Link Connection Management Settings Max TCP connections: 6 Bind Timeout: 15 Max binds per conn: 20 Timeout before Abandon: 1 Max LDAP Conn: 40 Max bind retries: 6 Max operations/conn: 2 Database Settings: Max Cache Size: 66060288 Mem Avail for cache: 105622733 Note: uidnumber is indexed SLAMD test parameters: Scheduled Start Time 06/05/2007 13:10:46 Scheduled Stop Time (not specified) Scheduled Duration 300 seconds Number of Clients 1 Wait for Available Clients true Monitor Clients if Available false Threads per Client 2400 Thread Startup Delay 0 milliseconds Statistics Collection Interval 60 seconds Job Dependencies (none specified) Notify on Completion (none specified) Parameter Information Directory Server Host app2 Directory Server Port 389 Bind DN (not specified) Bind Password (not specified) Search Base dc=calpers,dc=ca,dc=gov Search Scope Whole Subtree Search Filter uidnumber=50000 Filter File URL (not specified) Attributes to Return uidnumber Warm Up Time 0 Cool Down Time 0 Search Size Limit 0 Search Time Limit 0 Time Between Requests (ms) 0 Use SSL false Blindly Trust Any Certificate true SSL Key Store (not specified) SSL Key Store Password (not specified) SSL Trust Store (not specified) SSL Trust Store Password (not specified) Number of Iterations -1 Always Disconnect false Follow Referrals false Cary Anderson, Systems Software Specialist UNIX/Linux Services Information Technology Services Branch Technology Services & Support Division / Data Center Section System Software & Storage Infrastructure fCalPERS Phone: (916) 795-2588 Fax: (916) 795-2424
On Wed, 06 Jun 2007, Anderson, Cary wrote:> I am still playing around with load/stress testing my FDS install. I am using slamd to run the tests, and I am running the basic load, and the basic search tests against my test boxes, The slamd client and the FDS server are on identical hardware running RHEL4 with 2G memory, and 2 Intel Xeon 3.6GHz processors. The issue I am seeing is that I am getting "cannot connect to the ldap server" errors once I push the "threads per client" past 2400. My question is am I hitting a limit on the OS (max tcp connections)? Or a FDS limit? My assumption was that given appropriate hardware, and proper configuration, I could expect FDS to handle more than 2400 concurrent connections. Any insights as to what I might be missing would be greatly appreciated.Log entries from your server should give you some insight into why connections are being refused. There''s a good chance you haven''t allocated enough file handles, but that''s a guess.
Thanks for the feedback, I have turned on more extensive logging on the FDS server, Hopefully that will turn something up, but based on the settings in my sysctl.conf, limits.conf, and ulimit, I should be able to handle more than 2500 connections, unless I am missing a configuration parameter. Ulimit 805696 Sysctl.conf ############################## # Set Kernel Parameters # semaphores, SEMMSL, SEMMNS, SEMOPM, SEMMNI: kernel.sem = 256 32000 100 142 kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.msgmax = 8192 kernel.msgmnb = 65535 kernel.msgmni = 2878 fs.file-max = 131072 net.ipv4.ip_local_port_range = 32000 65000 Limits.conf # Custom setup for Global System Limits: * hard nproc 32767 * soft nproc 32767 * hard nofile 65536 * soft nofile 65536 * hard fsize 5000000 * soft fsize 1000000 Cary Anderson, Systems Software Specialist UNIX/Linux Services Information Technology Services Branch Technology Services & Support Division / Data Center Section System Software & Storage Infrastructure fCalPERS Phone: (916) 795-2588 Fax: (916) 795-2424 -----Original Message----- From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Patrick Morris Sent: Wednesday, June 06, 2007 12:33 PM To: General discussion list for the Fedora Directory server project. Subject: Re: [Fedora-directory-users] Load Testing question On Wed, 06 Jun 2007, Anderson, Cary wrote:> I am still playing around with load/stress testing my FDS install. I > am using slamd to run the tests, and I am running the basic load, and > the basic search tests against my test boxes, The slamd client and > the FDS server are on identical hardware running RHEL4 with 2G memory,> and 2 Intel Xeon 3.6GHz processors. The issue I am seeing is that I > am getting "cannot connect to the ldap server" errors once I push the > "threads per client" past 2400. My question is am I hitting a limit > on the OS (max tcp connections)? Or a FDS limit? My assumption was > that given appropriate hardware, and proper configuration, I could > expect FDS to handle more than 2400 concurrent connections. Any > insights as to what I might be missing would be greatly appreciated.Log entries from your server should give you some insight into why connections are being refused. There''s a good chance you haven''t allocated enough file handles, but that''s a guess. -- Fedora-directory-users mailing list Fedora-directory-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users
On Wed, 06 Jun 2007, Anderson, Cary wrote:> Thanks for the feedback, I have turned on more extensive logging on the > FDS server, Hopefully that will turn something up, but based on the > settings in my sysctl.conf, limits.conf, and ulimit, I should be able to > handle more than 2500 connections, unless I am missing a configuration > parameter. > > Ulimit 805696 > > Sysctl.conf > ############################## > # Set Kernel Parameters > # semaphores, SEMMSL, SEMMNS, SEMOPM, SEMMNI: > kernel.sem = 256 32000 100 142 > kernel.shmall = 2097152 > kernel.shmmax = 2147483648 > kernel.shmmni = 4096 > kernel.msgmax = 8192 > kernel.msgmnb = 65535 > kernel.msgmni = 2878 > fs.file-max = 131072 > net.ipv4.ip_local_port_range = 32000 65000 > > Limits.conf > > # Custom setup for Global System Limits: > * hard nproc 32767 > * soft nproc 32767 > * hard nofile 65536 > * soft nofile 65536 > * hard fsize 5000000 > * soft fsize 1000000You may be hitting a limit in the FDS configuration. Again, a quick glance at your LDAP logs (the error log in particular) should tell you if that''s the case.