In the annals of web hosting history, this particular TITSUP (Total Inability To Support Usual Performance) is right up there with the best. On the morning of Saturday 16 April 19, 2016, someone at the 123-Reg hosting provider ran a script that had a catastrophic error resulting in it deleting the servers and websites of many customers.
It was probably done with the best of intentions: routine maintenance to make the web server work more efficiently or perhaps to free up some space. However they ended up freeing a lot more space than they bargained for, with 67 web servers deleted.
This was 123-Reg’s initial statement:
“We suffered a technical fault, which has affected a minority of 123 Reg customers. The fault was limited to 67 servers out of 115,000 (across Europe) and it is important to note that only a selection of customers on the 67 servers affected has been impacted.
We are investigating the restoration of each VPS on a case-by-case basis and are working individually with customers to keep them informed of the website recovery process. We are working as quickly as we can to restore service to normal.”
The problem affected their VPS servers rather than shared hosting. Shared hosting is normally used by small businesses or hobbyists as they provide an entry level route into getting a website online and tend to benefit from dedicated support from the web hosting company’s technical support teams.
VPS (Virtual Private Servers) offer greater speed, security and features and are often setup and configured by a website owner’s chosen web developers or web hosting company. It is a common scenario where a web design and hosting company such as ourselves will have a VPS with several unconnected websites hosted. As long as the number is controlled and the web server has enough memory, diskspace and processing power to cope then this is common practice.
The issue with a VPS is that they tend to be unmanaged services which means that unless you, as the client, have specifically asked for it, website backup is an extra service which may or may not have been automatically configured or may not have been setup at all. That is why many are still scratching their heads on what to do next.
It might be possible to rebuild the server but if you haven’t got a recent backup of the website files, database and email accounts then you face major problems in getting back online. One high profile victim was the Scottish Premiership football team Ross County whose website was unavailable for over 72 hours – http://www.rosscountyfootballclub.co.uk/
123 Reg have called in data recovery experts and it looks like they are gradually recovering and restoring the deleted websites.
What lessons can learned from the 123-Reg website deletion incident.
There are lessons to be learned from any incident such as this. No one is immune from catastrophic events such as this so you and your web developer need a backup plan where you consider what would happen if you experienced this type of incident and how you would get back online. While you shouldn’t over-engineer your backup solution there needs to be some sort of proportionate, costed plan in place so you can get back online quickly after an incident.
There are several options for backing up your website that are both manual and automatic. While some options may need a bit of investment in mountable back drives, cloud storage and time to configure this is considerably cheaper than the consequences of losing all your data and being offline for long periods.
Please see our earlier blog post on backup types and solutions – Website backup plan, how often and how much
