August 22nd 2011
0915 CET - We have just noticed an issue with permissions on the server and all sites server wide are showing 403/500 errors.We are currently working on the issue as quickly as possible in order to get all sites up and running again.
update 1149 CET - we have a senior admin at the data centre looking into the issue. seems that cpanel and ftp is also being affected by this mysterious permissions issue. We apologise to our customers and we hope to have it resolved as soon as possible.
update 1203 CET - we are still in close contact with the main data centre admin and they are looking at this issue.
Update 1321 CET - have just been informed by the data centre that there are some corrupted binaries on the server so they are trying to recover them.
Update 1428 CET - have been informed by the data centre that the server will need to be rebooted after they reintsall perl in order for them to check the permissions situation. Am currently waiting for their action.
Update 1627 CET - we are still waiting for the data centre to give us a ETA as to when the web services will be back online. We sincerley apologise to our clients for this downtime.
Update 1733 CET - the datacentre has just informed us that the matter related to the server has been sent to Cpanel Support Team and is being investigated.
Update 2254 CET - we have again spoken to the data centre in the uk asking what is happening with the server. We have seen that cpanel.net support staff have gone into the sevrer earlier on but we have not received the update as promised from the uk data centre. We (ukcheaphosts) are not very happy with this situation and can only imagine the inconvenience its causing to clients. After shouting at the uk data centre I have been promised an update in 30 minutes time.
August 23rd 2011
Update 0021 CET - we were promised an update from the data centre at 2330 CET but it has not materialised. we are about to phone the data centre once again to find out what is going on. I think at this point its safe to say that we will more than likely be changing data centres after the issue is resolved as we cannot handle any more of these types of problems. the amateur management of our servers by the data centre reflects on us and this will not do. We deeply apologise to any of our uk cheap hosts clients for the serious inconvenience caused so far, now reaching over 15 hours.
Update 0235 CET - we still do not have updates from the uk data centre.
Update 0810 CET - seems that the issue with permissions was solved at 0439 CET this morning. We are launching a full investigation into what has happended and will report things to clients as when when we have more information.
Update 1346 CET - we have discovered that there are many wordpress sites with vulnerabilities leading to malicious users uploading files to the server in attempts to compromise the server. What we are doing at the moment is running various malware scanner and temporarily suspending users web sites until the files are checked. We are finding ALOT of wordpress sites with folder permissions (particularly /wp-content/uploads) being set to 0777, when the safest setting is 0755.
The issue yesterday between 0915CET and 0439 CET was caused by a particular nasty script being uploaded to the server via one of our clients and when executed it changed the root permissions on the server itself, taking all web sites offline, stopping all email services, cpanel, mysql etc. We still have people investigating how this has been allowed to happen, but the fact is that badly managed web sites with incorrect permissions on their folders are to blame here.
We are running a shared hosting environment and some responsibility for security is shifted to the customer when they load up open source software on their web sites or scripts which are not secure.
IF YOU ARE A WORDPRESS USER, READ THIS:http://codex.wordpress.org/Hardening_WordPress
For the time being we are going through around 50 + sites and their files and are bit by bit putting them back online again aonce the site is deemed to be safe.
We will update this message further throughout the day.
Monday, August 22, 2011
