This is the same issue that happended to us on 8th and 14th of May 2015 and appears to have resurfaced again on one particular server.
Firstly, we can only deeply apologise for any inconvenience.
Secondly, I have informed the firewall vendor again about this issue in the hope they can provide a permanent solution.
What has happended last night at appx 1230am gmt, the firewall simply stopped accepting requests from outside or from ip's that are whitelisted. The effect of this is that the server is still up, but noone from any outside ip's can connect to any web sites.
Server Monitoring
In order to determine if a server goes down, we do use server monitoring software which checks the server and various ports every 60 seconds. In fact we use two different companies for this.
Data Centre Monitoring (where server is hosted)- they ping various ports and services every 60 seconds
Uptimerobot.com (external 3rd part service) - they monitor our servers every 60 seconds.
However, those services are whitelisted on the firewall so did not detect any failure of connections, so noone at the data centre was alerted and uptime robot did not report any issues.
Since the last time this issue has happended on the 14th of May, I put in place a third monitoring option which is not whitelisted and picks up connection issues as if they were accessing like everyone else, and yes, when I woke this monirng I had 348 alerts telling me there was an issue, hence I went and restarted the firewall this morning and everyone from outside could conenct again.
Why does this keep happening?
The ASL firewall provides absolutely excellent security including bruteforce protection on individual clients sites. Other hosting companies do not provide such protetcion and usually you have to use something like suscuri.net which costs $200 a years per site. Since using ASL Firewall 3 years ago, we have not experienced any clients sites being hacked.
Its only these 3 instances this year that we have had major issues with the firewall closing all outside connections and we feel terrible its happended but I dont think switching firewalls is a great diea as it has served everyone so well so far..
What next?
I have raised a support ticket with the ASL team and wil work with them to find out a permanent solution to this.
Again, I can only apologise for this issue last night.
Monday, July 20, 2015
