I switched my web servers over to an AWS t2.micro instance in 2014, which has been adequate for my needs until late afternoon on December 16th when I started to get a flurry of "high CPU" alerts. I noticed a large number of queued web requests that seemed slow to complete. Web server responsiveness was extremely slow. A quick check of logs suggested a slightly higher than normal amount of web traffic, but I have seen similar spikes in the past without impacting web server performance.
I suspected a hardware issue, possibly the storage system. However, even simple console commands were taking much longer than normal. It turns out that I had exhausted my CPU Credit Balance (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html), causing my instance to be throttled. For a t2.micro, I earn credits if CPU utilisation is below 10% and consume credits if it is about 10%. The decline in my balance started around December 8th. I had a credit balance alert set that notified me on December 14th, but it was in a bunch of other overnight alerts and I missed it.
I took key web servers offline to reduce load. Further log analysis showed a large volume of web requests from a Chinese web hosting company that I blocked and also from addresses in the Google hosting range. I was seeing traffic from sets of sequential IP addresses in the 66.249.64.x, 66.249.65.x, 66.249.66.x, 66.249.68.x, 66.249.73.x, 66.249.75.x, and 66.249.79.x ranges - I blocked all traffic from these address ranges. By the morning of December 17th, my credit balance was recovering and is continuing to improve, so I was able to bring my web servers back online. I implemented operational changes to reduce the number of AWS alerts so that if this happens again, I will be more likely to catch the early signs.
I run web traffic filters that block attempts to access files that do not exist, are hidden. or are only used for site administration. I also monitor for high traffic rates which catch web scrapers and other miscreants. The traffic I was seeing from Google did not meet any of these criteria - the only oddity was similar traffic originating from small ranges of IP addresses. Sadly, web hosting security is a "whack-a-mole' activity.
Blog comments