BTW, there seems to be an issue with the yp servers traffic handling. After motherfish came back up today (from 3 days of downtime) mysql drove the load average up to 200, with the maximum number of mysqld instances all spinning in some kind of active wait. I attached to several but strace showed no activity during the period I was watching. This is quite bad; we assumed something was wrong and took the site down to investigate, but I don't know much about mysql and didn't discover anything obvious. Was this just a spike from all the icecast servers trying to connect at once? If so the code needs tuning. This is a serious scalability limit. It seems to be coming down slowly on its own now, which most of the mysqld processes going to sleep. -r --- >8 ---- List archives: http://www.xiph.org/archives/ icecast project homepage: http://www.icecast.org/ To unsubscribe from this list, send a message to 'icecast-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.
On Tue, Jun 01, 2004 at 02:11:03PM -0700, Ralph Giles wrote:> It seems to be coming down slowly on its own now, which most of the > mysqld processes going to sleep....or not. Apparenlty Jack noticed the load climbing again and killed mysql. I've restarted it with a connection limit of 10; this gives much the same results but limits the load to a more acceptable ceiling. Let me know when we can bring it back up. :) -r P.S. I'll be offline for much of wednesday and thursday. --- >8 ---- List archives: http://www.xiph.org/archives/ icecast project homepage: http://www.icecast.org/ To unsubscribe from this list, send a message to 'icecast-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.
I looks like the issue here was a corrupt database. Oddsock cleared the tables and everything is running normally again. (Thanks!) I've restored the normal connection count, so we should be back to regularly scheduled service. Anyone know if spinning like this is a known mysql bug? It's a nasty failure mode. -r --- >8 ---- List archives: http://www.xiph.org/archives/ icecast project homepage: http://www.icecast.org/ To unsubscribe from this list, send a message to 'icecast-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.