Site Outage Friday 3rd February 2023

[MagnetON]

At approximately 10:40am on Friday morning the platform providing all the servers for TenTenths, along with other projects was subjected to a "ransomware" attack.

Hundreds of servers around the world have been impacted, not just Tenths. You can read more about it here.

This attack effectively wiped out the Tenths web and database servers.

Shortly after analysing the attack I commissioned new hardware for Tenths and started the process of re-creating the servers.

We have a very robust backup / disaster recovery plan for Tenths with multiple generations of backup stored off-site from our hosting provider (including a 20Tb NAS in my attic!)

It takes a bit of time to create new database and web servers, once these are created the backups have to be uploaded to them, the database restored, and all the rest of the web server files put back in to place.

It's time consuming because there's a fair bit of data to ship around.

From experience it takes around 24 hours to get everything moved and done.

In this case we returned the site to functional in around 19!

There's more work to be done, but hopefully it won't involve any long interruptions.

[E.B]

Wondered what the issue was when regardless of browser or device I couldnt access, but knew the return to life was in good hands and would happen as soon as it could.

Thanks "Magneton" and any others involved for your effort and time involved returning access and seemingly normal navigation to all us site users.

Much appreciated.

[Mike E]

Thank you!

[Mondeo Man33]

Excellent work getting back online so quickly!

[gary396]

Well done to all concerned in restoring service - obviously a well drilled back-up plan. Great work.

[S griffin]

I knew something was up when I came on here just now and found a lot of my posts (and other people's) had disappeared.

[Aysedasi]

Fantastic work Grant, let's hope you don't get any chance to break the 19 hour record!

[GORDON STREETER]

Well done, I suppose that all you had to do was switch it off and back on

[VIVA GT]

Well done & thank you. (I actually got quite bored at work yesterday without this distraction!)

[Malcolm Smith]

Many thanks for all your efforts, I'm sure everyone appreciates them.

[Gerard C]

Thank you and well done, very much appreciated.

[Adam43]

Well done and thank you

[JT240Z]

Very thankful for all the hard work involved. You folks did a great job.

[Lancsbreaker]

Busy yesterday so didn't notice the outage, but thanks for your quick and effective work - sure it was a stressful time.

[chillibowl]

Missed everyone yesterday. Thanks for fixing!

[FIRE]

Thanks

[skycafe]

Thanks for all the effort!

[Tourer]

Thanks for getting us all back on the air so quickly - 19 hours sounds like a damn fine benchmark!

[MagnetON]

I'm sure I can improve on that time.

[275 GTB-4]

Originally Posted by MagnetON

I'm sure I can improve on that time.

Its NOT a contest! DON'T even think about improving your time!!!!

[MagnetON]

Well, I do have to move everything back to the original hardware at some point...

[MagnetON]

I've been asked if there's anything can be done to guard against this happening again.

In this specific attack, yes. The underlying vulnerability (which was the hosting platform itself) has been patched.

However it's impossible to protect against everything as new exploits are found every day.

I have a robust backup strategy that lets me recover the site in around 12-16hrs to a new server with the loss of, at most, a day worth of posts. Going forward I hope to reduce the time to recover, and the data loss range. This strategy will be tested very shortly when there will be an outage to move the site back to the original hardware.

Tenths doesn't make any money and is entirely funded by myself. To provision the temporary server it's currently running on will cost around £100/month for 6 months (minimum contract) in addition to the normal £150/month hosting bill.

It's always possible to build in more resilience, have multiple servers on multiple locations, or move the whole lot to cloud based infrastructure such as AWS but the money isn't there.

Follow up question:
"What about putting advertising back?"
Well, at the point the advertising was removed it was generating about a tenth (ironically!) of the monthly hosting bill. I figured if I was paying 90% anyway then the extra wasn't going to bankrupt me and you folks could have an increasingly rare thing.... A popular website without adverts.

So unfortunately there may be occasionally outages when major things happen.

[Richard C]

Thanks for all the hard work. I think your strategy is the right one. Right level of cost vs. risk. You can't protect against everything and having a good recovery strategy allows you to recover from things that slip through the protection model.

Originally Posted by MagnetON

Follow up question:
"What about putting advertising back?"
Well, at the point the advertising was removed it was generating about a tenth (ironically!) of the monthly hosting bill. I figured if I was paying 90% anyway then the extra wasn't going to bankrupt me and you folks could have an increasingly rare thing.... A popular website without adverts.

Funny, I had been subscribing for years, so I didn't notice the advertisements had gone for everyone! While you might be able to self fund this, have you considered opening up something like Patreon for those who might just want to contribute even if they don't get anything "special" back in return. Then again, I guess if more wanted to do help fund the site, you would have been more than 10% funded on the subscription model!

Richard

[Mike Harte]

May I add my thanks to Grant for all his hard work getting us back online almost before my withdrawal symptoms had kicked in.

As a previous subscriber, I would be more than happy to carry on the £10.00 subscription, and I would like to think that many more of us keyboard warriors would do so also.

[Aysedasi]

Originally Posted by Mike Harte

As a previous subscriber, I would be more than happy to carry on the £10.00 subscription, and I would like to think that many more of us keyboard warriors would do so also.

Absolutely. I contributed for a number of years when I didn't actually have to and a tenner a year is an incredibly small price to pay for everything I get out of Tenths.