Roland RoLaNd
2014-Jun-20 08:21 UTC
nf_conntrack: table full, dropping packet - Apache server with shorewall
I just added a new server to my web cluster, at low load all is good but at peek time i get this : kernel: [321835.288989] net_ratelimit: 6 callbacks suppressedkernel: [321835.288992] nf_conntrack: table full, dropping packet.kernel: [321835.289119] nf_conntrack: table full, dropping packet.kernel: [321835.289638] nf_conntrack: table full, dropping packet.kernel: [321835.289659] nf_conntrack: table full, dropping packet.kernel: [321835.289676] nf_conntrack: table full, dropping packet.kernel: [321835.289693] nf_conntrack: table full, dropping packet.kernel: [321835.289940] nf_conntrack: table full, dropping packet. and obviously, my web server starts dropping connections... this server has a 10 Gbps NIC + 15 GB of ram 13 out of which are assigned to apache... the shorewall config i used is out of : /usr/share/doc/shorewall/examples/one-interface with the following changes: rules : SECTION NEW#allowing office to do whtever.ACCEPT:info net:X.X.X.X $FW#accepting http/sWeb(ACCEPT) net $FW and my policy : #this server can connect to wherever$FW net ACCEPT #anything except rules allowed in rules is droppednet all DROP info# The FOLLOWING POLICY MUST BE LASTall all REJECT info My system is set by default to this:sysctl net.netfilter.nf_conntrack_maxnet.netfilter.nf_conntrack_max = 65536 When i experience high load, i max out the above (number changes) /sbin/sysctl net.netfilter.nf_conntrack_countnet.netfilter.nf_conntrack_count = 64946 researching the subject i found the two solutions: - i can change the max conntrack number but it should never be above the allowed opened files limit which is :cat /proc/sys/fs/file-max1534427 but this may freeze the system. so i don't want to risk it. - the other solution is to disable "natting" as this post suggests : http://www.pc-freak.net/blog/resolving-nf_conntrack-table-full-dropping-packet-flood-message-in-dmesg-linux-kernel-log/ but i'm not confident with this step, which is why i'm reaching out to you. Any advice or alternative solution would be appreciated. ------------------------------------------------------------------------------ HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing & Easy Data Exploration http://p.sf.net/sfu/hpccsystems