No, really, service http stop. Srsly.

[adric]

Load was in the 570s on this poor thing, so I stopped www and sql services to try and save it. After a bit of killall httpd, killall -9 httpd and the like, things are finally starting to calm down (300s, 200s). Then something spawns some more apaches, half of which immediately hang, and load hits the 700s again. I kill and kill -9 , and I try to find the source of the new apaches, not in screen, not in cron ... fuckit:

mv /etc/apache/bin/httpd /etc/apache/bin/httpd.stopthat

Problem solved. Now then, where was I... I let the load settle down to approximately zero, start sql and mv apache back into place. I didn't even get a chance to start it as while I was checking the logs they spawned again and hello load of 681.36. WTF? So,

mv /etc/apache/bin/httpd /etc/apache/bin/httpd.srsly

And a great many slow-acting killall -9 httpds later the load is settling down, into the 30s.., 20s .. good server, niiiiice server. 7.75 174.54 263.47, 2.85 142.77, 246.98 ... good girl ...

And now I turn an even more suspicious eye to that cgi I saw in the logs...

Epilogue? : /var was completely full. Apaches could not add to logs and therefore hung without logging an error..