Earlier this week, NatureMapr suffered a large scale denial of service style attack, causing a sustained outage over several days.
It is important to stress that sensitive data was not at risk at any stage of this attack - our sensitive data handling framework continued to do exactly what it was designed to do, safeguarding sensitive information through multiple layers of robust controls at both the infrastructure and application layers.
This attack was an attempt to overload our infrastructure and/or make the platform unavailable for legitimate users.
We don't have any explanation as to the reasoning. Hackers sometimes get off on the challenge of bringing down busy web sites as an accomplishment, other times they do it to try to find and discover weaknesses in busy applications to test out their skills. Other times, zombie servers try to overload every web site until they find one that falls over. Regardless, this is a standard part of doing business on the internet these days and we've only been lucky it hadn't happened earlier.
As NatureMapr has grown over time, the overall footprint of the number of pages and assets that we have "dangling out on the internet" as potential targets has also grown. Every single one of these assets costs money to build, host and maintain over time and ultimately increases our risk and exposure.
Example numbers during the peak of the attack were:
- 143,000 images downloaded within 60 minutes (40 image download requests per second sustained)
- Over 6 million requests to user species and user sighting pages within 24 hours (70 requests per second sustained)
- Spikes of up to 500 requests to user species and user sighting pages per second
- Over 1000 requests per second across the platform in aggregate
This attack led to 2 very positive improvements for the platform:
- Rapid provisioning of enterprise grade web application firewall infrastructure and policies which give us much greater control over who and what we allow to send requests to the platform over the internet. While this was a layer 7 (application level) attack, we now also benefit from network level DDOS protection by having all NatureMapr traffic routed (and monitored) through the edge network of a major provider.
The downside is that we now get charged per gigabyte of traffic to have this infrastructure and we are not looking forward to the bill. Regardless, this is now absolutely mandatory, non-negotiable infrastructure.
We continue to tune a number of defensive policies to further reduce the number of rogue requests that are still hitting the platform
- Over time, the attacks also exposed a low level weakness in our application structure that caused web page load times to respond erratically and unreliably under these extremely high loads.
Requests to user species and user sighting pages were becoming blocked and banked up until all previous requests to these pages to completed. This quickly became a very large bottleneck that resulted in our infrastructure running out of CPU, followed by one or more outages shortly later.
This weakness in our application structure was refactored and remediated early yesterday which has liberated these affected "bottleneck" pages to be able to better handle extremely large numbers of requests in parallel, without causing thread starvation (an outage).
These changes have proven to have had a positive effect on platform stability and we are now better placed for the future.
I want to thank and acknowledge the support of my team during this very testing period.
Onwards and upwards for NatureMapr.
