DDoS Use Case - How we mitigated a 9Mbps DDoS attack?

One of our sister website hosted on cloud server was recently hit by a 9Mbps DDoS, and the apache web server ran out of memory and crashed. The attack lasted more than 2-months with no known reason. We've taken a number of mitigation steps including installation of mod_security with mod_evasive, APF, BFD, DDoS Deflate and Rootkit and Traffic Control, but none came to rescue. Use of Linux provided WAF will mitigate the DDoS to the extent where CPU, Memory and Bandwidth are allowed; and in our case a single CentOS server with 4GB RAM wasn't sufficient to mitigate DDoS.

Since our initial mitigation effort failed, we cloned the server and added 4 additional VPS servers with a node balancer from Linode. Having 5-node balancer wasn't' sufficient to handle 9 MBps bot traffic. The trouble was that Apache would run "Out of Memory" due to fake bot traffic, and no user traffic were served. I thought it was the Apache Killer, which caused my Apache to crash but that wan't the case as my version of Apache was patched and yet I was experiencing the same problem.

To alleviate "Out of Memory" problem, I've also tried Nginx setup but nginx wasn't able to handle the traffic either. With a load balancer with 5 nodes of Nginx servers, the webapp was throwing 503 Service Unavailable although I wasn't seeing the "Out of Memory" error.

We, then tried DDoS mitigation service provided by Incapsula, which took only 5-minutes to setup. To mitigate DDoS, We had to signup with a 7-day Trial of Business Plan. Incapsula's DDoS mitigation service worked great, but it would cost us $299 per month. For websites generating over $300 in revenue, paid service is a great way to protect your site from DDoS but for smaller websites it may not be cost-effective. Here is the DDoS stats collected from Incapsula.