Ive been googling quite a bit now for problems with running out of mbuf clusters. Im basically sending a 30k datachunk down 1000-4000 connections, but 1000 is more than enough to quickly fill upp 8192 mbuf clusters. I also tried setting maximum amount of mbuf clusters to 65536, but that only made the box hard-wire 86MB of 96MB RAM, making it just as unsuable as a dead machine. Of course, when the machine runs out of mbuf clusters, it dies. I also found this with google: "Finally, the fact that FreeBSD 3.x panics when it runs out of mbuf clusters is a well-known problem. The solution is to not let it run out of mbuf clusters by configuring a sufficient number for them.">From this it sounds as it is a problem that should befixed, but it obviously isnt in 4.8. Is this behaviour now considered acceptable? And if so, doesnt this make FreeBSD extremely easy to kill using a simple DOS-attack? Is this "fixed" in any way on 5.1? --- John B?ckstrand
On Thu, Jun 26, 2003 at 03:03:20AM +0200, John B?ckstrand wrote:> Ive been googling quite a bit now for problems with > running out of mbuf > clusters. Im basically sending a 30k datachunk down > 1000-4000 connections, > but 1000 is more than enough to quickly fill upp 8192 > mbuf clusters. I also > tried setting maximum amount of mbuf clusters to 65536, > but that only made > the box hard-wire 86MB of 96MB RAM, making it just as > unsuable as a dead > machine. > > Of course, when the machine runs out of mbuf clusters, > it dies. I also found > this with google: > > "Finally, the fact that FreeBSD 3.x panics when it runs > out of > mbuf clusters is a well-known problem. The solution is > to not > let it run out of mbuf clusters by configuring a > sufficient > number for them." > > >From this it sounds as it is a problem that should be > fixed, but it > obviously isnt in 4.8. Is this behaviour now considered > acceptable? And if > so, doesnt this make FreeBSD extremely easy to kill > using a simple > DOS-attack? Is this "fixed" in any way on 5.1? > > --- > John B?ckstrandIt's not panicking, it's running out of resources. Whenever you have this sort of problem you need to provide more information, there is absolutely no way I can help you like this. You need to, at the very minimum, give us 'netstat -m' output and make a serious attempt at figuring out what is consuming so many clusters. You could be running out of clusters but you could also be running out of memory before you run out of clusters, in which case you should probably _not_ increase nmbclusters and instead fix the underlying problem instead (re-work your application). In such a scenario, blindly bumping up nmbclusters can make the problem worse. Even if you had 'unlimited' (or dynamically growing) nmbclusters, you'd _still_ have the same problem and, what's more, it could actually render your system even more unusable as the machine would not be able to allocate memory for other more important uses. -- Bosko Milekic * bmilekic@technokratis.com * bmilekic@FreeBSD.org TECHNOkRATIS Consulting Services * http://www.technokratis.com/
> From this it sounds as it is a problem that should be > fixed, but it > obviously isnt in 4.8. Is this behaviour now considered > acceptable? And if > so, doesnt this make FreeBSD extremely easy to kill > using a simple > DOS-attack? Is this "fixed" in any way on 5.1?Afaik, this bug was recently fixed in both of HEAD and STABLE. Eugene
John Bdckstrand wrote:> > Ive been googling quite a bit now for problems with > running out of mbuf > clusters. Im basically sending a 30k datachunk down > 1000-4000 connections, > but 1000 is more than enough to quickly fill upp 8192 > mbuf clusters. I also > tried setting maximum amount of mbuf clusters to 65536, > but that only made > the box hard-wire 86MB of 96MB RAM, making it just as > unsuable as a dead > machine.It isn't the amount data you are sending but the overhead required for each network connection. I would recommend adding more RAM. Bump it up to 256MB. With mbuf set at 65536 and using 86MB, you should still have plenty left for your application. If that doesn't solve your problem or you can't add more memory, then you'll want to look at controling the number of simultaneous connections to a number that your box can handle.> > Of course, when the machine runs out of mbuf clusters, > it dies. I also found > this with google: > > "Finally, the fact that FreeBSD 3.x panics when it runs > out of > mbuf clusters is a well-known problem. The solution is > to not > let it run out of mbuf clusters by configuring a > sufficient > number for them." > > >From this it sounds as it is a problem that should be > fixed, but it > obviously isnt in 4.8. Is this behaviour now considered > acceptable? And if > so, doesnt this make FreeBSD extremely easy to kill > using a simple > DOS-attack? Is this "fixed" in any way on 5.1?Yup, that is what DoS attack is... exhaustion of one or more resources of the victim. P2P software is an easy way to exhaust mbuf buffers on a box. P2P software(e.g. edonkey) can be a useful network stress tool; opens lots of connections and pushes a lot of data. My experience with mbuf exhaustion on a 4-stable boxes has been the box basically loses network connectivty until it can recover some buffers. The box is still responsive from the console and killing the offending application from the console will free up the mbufs and restore network connectivity. good luck, greg
> > >From this it sounds as it is a problem that shouldbe> > fixed, but it > > obviously isnt in 4.8. Is this behaviour nowconsidered> > acceptable? And if > > so, doesnt this make FreeBSD extremely easy to kill > > using a simple > > DOS-attack? Is this "fixed" in any way on 5.1? > > Yup, that is what DoS attack is... exhaustion of oneor more resources> of the victim. > > P2P software is an easy way to exhaust mbuf bufferson a box. P2P> software(e.g. edonkey) can be a useful network stresstool; opens lots> of connections and pushes a lot of data. Myexperience with mbuf> exhaustion on a 4-stable boxes has been the boxbasically loses network> connectivty until it can recover some buffers. Thebox is still> responsive from the console and killing the offendingapplication from> the console will free up the mbufs and restorenetwork connectivity. Ah, unfortunately my box doesnt respond even to keyboard events (caps lock etc). The behaviour you describe I find totally acceptable on the other hand. And the software Im writing happens to be p2p-related, but its not a edonkey server. :) --- John B?ckstrand
> > From this it sounds as it is a problem that shouldbe> > fixed, but it > > obviously isnt in 4.8. Is this behaviour nowconsidered> > acceptable? And if > > so, doesnt this make FreeBSD extremely easy to kill > > using a simple > > DOS-attack? Is this "fixed" in any way on 5.1? > > Afaik, this bug was recently fixed in both of HEADand STABLE.> > EugeneYes, you could well be right. The fix about not checking return values from m_copy() when receiving broadcast messages I think, sure seems to fit what is happening to me. I disconnected my three machines from the rest of the LAN, and tried exhausting mbuf clusters again. This time, it didnt hang, and no rl0 watchdog messages either. I am ugprading the box now and will try a new kernel. --- John B?ckstrand
> > From this it sounds as it is a problem that shouldbe> > fixed, but it > > obviously isnt in 4.8. Is this behaviour nowconsidered> > acceptable? And if > > so, doesnt this make FreeBSD extremely easy to kill > > using a simple > > DOS-attack? Is this "fixed" in any way on 5.1? > > Afaik, this bug was recently fixed in both of HEADand STABLE.> > EugeneIndeed, I upgraded my system and it now works as advertised: If I use up too much resources, netowork connectivity is lost, but once I disconnect those connections, it comes back. (it even stops responding to pings =) --- John B?ckstrand