samba - Jan 2002 - Samba and Out Of Connection Structures on Solaris (again)

Hi Samba gurus,

Recently we are running in to out of connection structures woe's again.

Symptom:
Not able to get a previously succesfully mounted samba share.

Samba.log:
[2002/01/09 12:37:31, 1] smbd/conn.c:(104)
  ERROR! Out of connection structures
[2002/01/09 12:37:31, 0] smbd/service.c:(325)
  Couldn't find free connection.

On tips of this list, and the Sun site, we did a couple of things:

- Running Solaris 8 in 64 bit mode

- samba 64 bit compiled
   - samba 2.2.2
   - Sun CC cc: Sun WorkShop 6 update 1 C 5.2 2000/09/11
   - environment flags:
	setenv CC cc
	setenv LDFLAGS "-fast -xtarget=ultra2i -xarch=v9a -xO5"
	setenv CFLAGS "-fast -xtarget=ultra2i -xarch=v9a -xO5"
	setenv CXXFLAGS "-fast -xtarget=ultra2i -xarch=v9a -xO5"
	setenv CPPFLAGS "-fast -xtarget=ultra2i -xarch=v9a -xO5"

- Raise limit for number of open files systemwide
  - set rlim_fd_cur=2048
  - set rlim_fd_max=8192

- number of filedescriptor limits upped with plimit:
        MOO=`ps -ef | grep mbd | grep pro64 | grep -v grep | awk '{print
$2}'`
        for pid in $MOO; do
                plimit -n 51200,52224 $pid
        done

We are running out of clue's :-( and Im still looking at Out of Connections 
every day.

Any clue or help would be enormously appreciated.

Regards Richard van Beers

richard van beers wrote:
> Recently we are running in to out of connection structures woe's again.
	Can you do a 
		pfiles <pid of smbd>
	please?

	It should report, as it's second line, 
	soemthing like "Current rlimit: 10010 file 
	descriptors" (suggestion dourtesy of Thomas 
	Tornblom)

--dave
-- 
David Collier-Brown,           | Always do right. This will gratify 
Americas Customer Engineering, | some people and astonish the rest.
SunPS Integration Services.    |                      -- Mark Twain
(905) 415-2849                 | davecb@canada.sun.com

Jasper Jans wrote:
|plimit -mk:
| 8592:   /home/sw/sparc-SunOS-5.8-pro64/samba-2.2.2/sbin/smbd -D
|   resource              current         maximum
|  time(seconds)         unlimited       unlimited
|  file(kbytes)          unlimited       unlimited
|  data(kbytes)          unlimited       unlimited
|  stack(kbytes)         8192            unlimited
|  coredump(kbytes)      unlimited       unlimited
|  nofiles(descriptors)  51200           52224
|  vmemory(kbytes)       unlimited       unlimited
|
| pfiles:
| 8592:   /home/sw/sparc-SunOS-5.8-pro64/samba-2.2.2/sbin/smbd -D
|  Current rlimit: 51200 file descriptors

	Ok, it's more likely to be a Samba problem
	than Slolaris...  Off to the Samba-Technical
	list we go.

--dave
--
David Collier-Brown,           | Always do right. This will gratify 
Americas Customer Engineering, | some people and astonish the rest.
SunPS Integration Services.    |                      -- Mark Twain
(905) 415-2849                 | davecb@canada.sun.com

Hi Richard,
           while i am our local samba guru, i consider myself
not a real one, so sorry somehow this rather a request for 
help/information than some aid to you.

- I tried running a Samba 2.2.2 configured as PDC on an E220
  with Solaris 8. (Samba from ftp-site, compiler gcc)

- It failed badly, when the number of Users/Clients exceeded 200.

- It failed on Solaris 2.6 (UE3000) too, with the common symptom:

  - Enormous Overload of the system (Load-Indicater > 64)

  - Samba 2.0.6 worked and works just fine, (The system is 
    almost idle)

  - on other oddity, i am using:

    log file                = /var/samba/log/%M.%m

    and with samba-2.2.2 (in the PDC-configuration, i am running
    other samba-2.2.2's without obvious problems) i get log-files
    like this:

samba/log> more 0.0.0.0.pc2863
[2002/01/07 09:10:32, 0] lib/util_sock.c:get_socket_addr(1038)
  getpeername failed. Error was Transport endpoint is not connected
[2002/01/07 09:10:50, 0] smbd/connection.c:yield_connection(63)
  yield_connection: tdb_delete for name  failed with error Record does not exist

    i can interpret this as a result of the overload (the client has
disconnected
    up, due to timeout)

I feel beeing far away from the limits you mentioned, but i considered your 
hints very helpful. Nevertheless i am strongly interested in more information
about your workload and configuration of your system (and could provide the
same,
if you wish). I have checked for the errors you mentioned
in the logs: they don't occur.


Regards & Good Luck
Gunther Hess
-- 
Dr. Gunther Hess           Elmos Semiconductor AG
Tel.: ++49 +231 7549-520   Heinrich Hertz Str. 1
Fax:  ++49 +231 7549-109   D-44227 Dortmund
WWW: http://www.elmos.de   Opinions are strictly of my own

David,

Thanks for your input. Debug will be enabled today in the
timeframe that we usually experience the problems. However
I got an other reply that seems to indicate where we run
into problems. Our environment uses NT4 terminal servers
to connect to the samba fileserver - and this person had
the same problems as we are seeing. 

----
Subject: Antwort: Samba and Out Of Connection Structures on Solaris (again)
Date: Wed, 9 Jan 2002 16:33:33 +0100
From: hassfeld@xenario.de
To: <richard.vanbeers@novaxess.net>

Hi there!

Are you running a Windows NT4 Terminal Server Edition or Windows 2000
Server with Terminal Services enabled out there? I had the same problem
with shares used by Windows 2000 Terminal Server sessions, because all
shares on your file server used by that server are connected through a
single SMB connection, which runs out ouf connection structures. I raised
MAX_CONNECTIONS in source/smbd/conn.c to 512 and compiled again and it
worked...
----

The idea is the following - run debug during the "regular" problem
times and see what that gives us. Recompile samba with the MAX_CONNECTIONS
set to a higher level (using the 512 mentioned above) and repalce the
current daemons with the new ones after working hours tonight.

So hopefully we will know by tomorrow if this has fixed our problem.
If it does - it might be a good idea to either up this limit in the
new release - or put a warning in the release notes that if you are
running terminal services you might want to up this level...

Sincerely,
Jasper


| -----Original Message-----
| From: davecb@scot.canada.sun.com [mailto:davecb@scot.canada.sun.com]On
| Behalf Of David Collier-Brown
| Sent: Wednesday, January 09, 2002 7:02 PM
| To: samba-technical@lists.samba.org
| Cc: David.Collier-Brown@Sun.COM; Jasper Jans;
| richard.vanbeers@novaxess.net
| Subject: Re: Samba and Out Of Connection Structures on Solaris (again)
| 
| 
| on the general samba list, Jasper Jans wrote:
| |plimit -mk:
| | 8592:   /home/sw/sparc-SunOS-5.8-pro64/samba-2.2.2/sbin/smbd -D
| |   resource              current         maximum
| |  time(seconds)         unlimited       unlimited
| |  file(kbytes)          unlimited       unlimited
| |  data(kbytes)          unlimited       unlimited
| |  stack(kbytes)         8192            unlimited
| |  coredump(kbytes)      unlimited       unlimited
| |  nofiles(descriptors)  51200           52224
| |  vmemory(kbytes)       unlimited       unlimited
| |
| | pfiles:
| | 8592:   /home/sw/sparc-SunOS-5.8-pro64/samba-2.2.2/sbin/smbd -D
| |  Current rlimit: 51200 file descriptors
| 
| I then wrote: 
| >         Ok, it's more likely to be a Samba problem
| >         than Slolaris...  Off to the Samba-Technical
| >         list we go.
| 
| 
| He's getting 
| Samba.log:
| [2002/01/09 12:37:31, 1] smbd/conn.c:(104)
|   ERROR! Out of connection structures
| [2002/01/09 12:37:31, 0] smbd/service.c:(325)
|   Couldn't find free connection.
| 
| This is on Solaris 8, 64-bit, with the rlim_fd_cur=2048 and
| rlim_fd_max=8192 and the fd's per process to 51200 via plimit,
| which has set the system limits properly.
| 
| We use getrlimit(RLIMIT_NOFILE,...), which seems to return
|     rlim_cur = 256U
|     rlim_max = 1024U
| with the default rlimit: 10010 file descriptors
| 
| Suggestion to Mr Jans: print the values with DEBUG
| found in 
|   File           Function        Line
| 2 util.c         set_maxfiles    1516 if(getrlimit(RLIMIT_NOFILE,
| 
| 
| This may be a Solaris issue, or a bug in the logic
| that sizes samba's file bitmap. If it's the former,
| I can officially report it.
| 
| --dave
| 
| -- 
| David Collier-Brown,           | Always do right. This will gratify 
| Americas Customer Engineering, | some people and astonish the rest.
| SunPS Integration Services.    |                      -- Mark Twain
| (905) 415-2849                 | davecb@canada.sun.com
|

On Wednesday 09 January 2002 12:52, Gunther Hess wrote:
> Hi Richard,
>            while i am our local samba guru, i consider myself
> not a real one, so sorry somehow this rather a request for
> help/information than some aid to you.
>
> - I tried running a Samba 2.2.2 configured as PDC on an E220
>   with Solaris 8. (Samba from ftp-site, compiler gcc)
>
> - It failed badly, when the number of Users/Clients exceeded 200.
Hi,

while we have none of your performance issues (yet), here is our setup:

4 Win NT Terminal Services  + Citrix servers Compaq DL360 dualprocs, serving 
about 40/50 people average per server.
samba 2.2.2 running from Solaris 8

Apart from the file_decriptor problem, everything has always worked 
flawlessly, and in fact we think we have found a patch to be applied to fix 
our problem too. 
MAX_CONNECTIONS in source/smbd/conn.c to 512 instead of default 128 (which is 
per deamon, which are shared because we use terminal server; 40/50 ppl login 
from the same machine.

HTH Richard