CentOS - Sep 2009 - Fork: Resource Temporarily Unavailable

Hey all,

I installed a while ago two CentOS 5.3 servers. One of them was
running Samba and another OpenLDAP. I configured Samba to act as a PDC
on the network and everything was working fine, I was already on
policies rules and desktop migrations.

One day, another analyst changed our subnet, and the domain was
implemented only on our sector for start. After the subnet change, we
had a comunication problem and we figured out it that we needed to
configure samba as WINS server too. After a while, he changed Proxy
requests for WINS to samba, only as a test (some of the requests).
Before we could configure samba, BUM, the samba server crashed and
after some crash/restart it showed this on boot time (of course when
sshd was starting)

* /etc/rc3.d/S55sshd: fork: Resource Temporarily Unavailable

After that, ALL CentOS installations have this problem after yum update:

* /etc/init.d/functions: fork: Resource Temporarily Unavailable

I have only one clue: EPEL repository. I installed and configured
yum-priorities of course.

I?m on my last test installation and I am sending now this e-mail
because if it crashes and i have no resolution, they are going to ask
me for another distro install, and I like CentOS too much to do that.

Just one last thing: its a 6000-7000 user environment and we're on a
hurry, so I can't do long time testing or stay with something that may
one day crash.

Thanks,

Gustavo Pastorino

Gustavo Pastorino wrote:
> * /etc/rc3.d/S55sshd: fork: Resource Temporarily Unavailable
Sounds like too many processes are running, you try to run top
(assuming it even works in that condition), if the problem
doesn't happen right away I'd cron a script to run once a
minute and log the contents of ps auxw, and see what chews up
all of the resources on the box.

You can also try adjusting the limits in /etc/security/limits.conf
it seems the root user is "limited" to roughly 36,000
processes, which to me is more than enough, I don't think I've
ever seen more than 3-4k on a box and that was typically when
the box was horked.

nate

Hi,

On Thu, Sep 10, 2009 at 14:10, Gustavo Pastorino
<gustavopastorino at gmail.com> wrote:
> * /etc/rc3.d/S55sshd: fork: Resource Temporarily Unavailable
>
> [...]
>
> I?m on my last test installation and I am sending now this e-mail
> because if it crashes and i have no resolution, they are going to ask
> me for another distro install, and I like CentOS too much to do that.
>
> Just one last thing: its a 6000-7000 user environment and we're on a
> hurry, so I can't do long time testing or stay with something that may
> one day crash.
Very hard to "guess" what is wrong with your machine just with that
information above...

Please post:
* What kernel version are you using? (uname -mri)
* What were the last updates to this machine? (tail -50
/var/log/yum.log; rpm -qa --last | head -50)
* What are the hardware specs of that machine? How many CPUs? How much
RAM? (cat /proc/cpuinfo; cat /proc/meminfo)

These above might help figure out what is happening with your machine.

HTH,
Filipe

On Sep 10, 2009, at 2:10 PM, Gustavo Pastorino <gustavopastorino at gmail.com
 > wrote:
> Hey all,
>
> I installed a while ago two CentOS 5.3 servers. One of them was
> running Samba and another OpenLDAP. I configured Samba to act as a PDC
> on the network and everything was working fine, I was already on
> policies rules and desktop migrations.
>
> One day, another analyst changed our subnet, and the domain was
> implemented only on our sector for start. After the subnet change, we
> had a comunication problem and we figured out it that we needed to
> configure samba as WINS server too. After a while, he changed Proxy
> requests for WINS to samba, only as a test (some of the requests).
> Before we could configure samba, BUM, the samba server crashed and
> after some crash/restart it showed this on boot time (of course when
> sshd was starting)
>
> * /etc/rc3.d/S55sshd: fork: Resource Temporarily Unavailable
>
> After that, ALL CentOS installations have this problem after yum  
> update:
>
> * /etc/init.d/functions: fork: Resource Temporarily Unavailable
>
> I have only one clue: EPEL repository. I installed and configured
> yum-priorities of course.
>
> I?m on my last test installation and I am sending now this e-mail
> because if it crashes and i have no resolution, they are going to ask
> me for another distro install, and I like CentOS too much to do that.
>
> Just one last thing: its a 6000-7000 user environment and we're on a
> hurry, so I can't do long time testing or stay with something that may
> one day crash.
Boot into run level 1, bring up auxillary services one by one waiting  
to make sure each works until you reach the one that borks, then  
reboot into level 1 disable problem service, go into run level 3, then  
diag config file and figure out what broke.

My bet is samba and some script that is kicked off when user's connect.

-Ross