CentOS - Feb 2012 - Race Condition In PHP Session Handler

php-5.3.3-3.el6_2.6.x86_64
2.6.32-220.4.2.el6.x86_64

After some testing we moved our production PHP intranet site from an old
server to a new CentOS 6.2 instance running the mentioned versions.  At
first it worked well and then user's started to experience a hung site.
Upon some digging it appears that the PHP module is getting stuck in a
race condition regarding the sessions file -

[root at yannigan-orange fd]# strace -p  22607
Process 22607 attached - interrupt to quit
flock(53, LOCK_EX^C <unfinished ...>

where file handle 53 is 

...
lr-x------. 1 root root 64 Feb 17 15:24 52
-> /opt/informix/msg/en_us/0333/cli.iem
lrwx------. 1 root root 64 Feb 17 15:24 53
-> /var/lib/php/session/sess_ulgsl9joplobb9o4fue3n2e1k0
l-wx------. 1 root root 64 Feb 17 15:24 6 -> pipe:[660246]
....

The PHP session file.  Originally I had a tmpfs mounted for the session
files.  I removed that and used the underlying filesystem.  Same issue.
I disabled APC.  Same issue.

Switching the the memcache session handler resolved the issue,  although
that may be less optimal.  But this seems like a PHP bug.

Tips, comments, pointers?

Backtrace of a hung httpd -
#0  0x00007fa175af9be7 in flock () from /lib64/libc.so.6
#1  0x00007fa16cb59dfc in ?? () from /etc/httpd/modules/libphp5.so
#2  0x00007fa16cb5a047 in ?? () from /etc/httpd/modules/libphp5.so
#3  0x00007fa16cb56e4c in php_session_start ()
from /etc/httpd/modules/libphp5.so
#4  0x00007fa16cb57559 in ?? () from /etc/httpd/modules/libphp5.so
#5  0x00007fa16cc9afe8 in ?? () from /etc/httpd/modules/libphp5.so
#6  0x00007fa16cc72320 in execute () from /etc/httpd/modules/libphp5.so
#7  0x00007fa16cc4ca5d in zend_execute_scripts ()
from /etc/httpd/modules/libphp5.so
#8  0x00007fa16cbfacf8 in php_execute_script ()
from /etc/httpd/modules/libphp5.so
#9  0x00007fa16ccd59a5 in ?? () from /etc/httpd/modules/libphp5.so
#10 0x00007fa17751da60 in ap_run_handler ()
#11 0x00007fa17752131e in ap_invoke_handler ()
#12 0x00007fa17752c990 in ap_process_request ()
#13 0x00007fa177529858 in ?? ()
#14 0x00007fa177525568 in ap_run_process_connection ()
#15 0x00007fa177531767 in ?? ()
#16 0x00007fa177531a7a in ?? ()
#17 0x00007fa177531dab in ap_mpm_run ()
#18 0x00007fa177509900 in main ()



-- 
System & Network Administrator [ LPI & NCLA ]
<http://www.whitemiceconsulting.com>
OpenGroupware Developer <http://www.opengroupware.us>
Adam Tauno Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.centos.org/pipermail/centos/attachments/20120217/df89ab7e/attachment-0005.sig>

<http://debuginfo.centos.org/6/x86_64/>

My CentOS server has php-5.3.3-3.el6_2.6.x86_64 but it appears the
latest debuginfo package is php-debuginfo-5.3.3-3.el6_2.5.x86_64.rpm

yum doesn't want to install this due to "Package does not match
intended
download"  [yes, I did the obligatory "clean metadata"].

Are debug info packages not current?  Does the .6. vs. .5. matter?

On Fri, 2012-02-17 at 16:51 -0500, Adam Tauno Williams
wrote:
> php-5.3.3-3.el6_2.6.x86_64
> 2.6.32-220.4.2.el6.x86_64
> 
> After some testing we moved our production PHP intranet site from an old
> server to a new CentOS 6.2 instance running the mentioned versions.  At
> first it worked well and then user's started to experience a hung site.
> Upon some digging it appears that the PHP module is getting stuck in a
> race condition regarding the sessions file -
> 
> [root at yannigan-orange fd]# strace -p  22607
> Process 22607 attached - interrupt to quit
> flock(53, LOCK_EX^C <unfinished ...>
> 
> where file handle 53 is 
> 
> ...
> lr-x------. 1 root root 64 Feb 17 15:24 52
> -> /opt/informix/msg/en_us/0333/cli.iem
> lrwx------. 1 root root 64 Feb 17 15:24 53
> -> /var/lib/php/session/sess_ulgsl9joplobb9o4fue3n2e1k0
> l-wx------. 1 root root 64 Feb 17 15:24 6 -> pipe:[660246]
> ....
> 
> The PHP session file.  Originally I had a tmpfs mounted for the session
> files.  I removed that and used the underlying filesystem.  Same issue.
> I disabled APC.  Same issue.
> 
> Switching the the memcache session handler resolved the issue,  although
> that may be less optimal.  But this seems like a PHP bug.
> 
> Tips, comments, pointers?
> 
> Backtrace of a hung httpd -
> #0  0x00007fa175af9be7 in flock () from /lib64/libc.so.6
> #1  0x00007fa16cb59dfc in ?? () from /etc/httpd/modules/libphp5.so
> #2  0x00007fa16cb5a047 in ?? () from /etc/httpd/modules/libphp5.so
> #3  0x00007fa16cb56e4c in php_session_start ()
> from /etc/httpd/modules/libphp5.so
> #4  0x00007fa16cb57559 in ?? () from /etc/httpd/modules/libphp5.so
> #5  0x00007fa16cc9afe8 in ?? () from /etc/httpd/modules/libphp5.so
> #6  0x00007fa16cc72320 in execute () from /etc/httpd/modules/libphp5.so
> #7  0x00007fa16cc4ca5d in zend_execute_scripts ()
> from /etc/httpd/modules/libphp5.so
> #8  0x00007fa16cbfacf8 in php_execute_script ()
> from /etc/httpd/modules/libphp5.so
> #9  0x00007fa16ccd59a5 in ?? () from /etc/httpd/modules/libphp5.so
> #10 0x00007fa17751da60 in ap_run_handler ()
> #11 0x00007fa17752131e in ap_invoke_handler ()
> #12 0x00007fa17752c990 in ap_process_request ()
> #13 0x00007fa177529858 in ?? ()
> #14 0x00007fa177525568 in ap_run_process_connection ()
> #15 0x00007fa177531767 in ?? ()
> #16 0x00007fa177531a7a in ?? ()
> #17 0x00007fa177531dab in ap_mpm_run ()
> #18 0x00007fa177509900 in main ()
-- 
System & Network Administrator [ LPI & NCLA ]
<http://www.whitemiceconsulting.com>
OpenGroupware Developer <http://www.opengroupware.us>
Adam Tauno Williams