How Heart Internet web hosting almost destroyed my business

If you’re a regular reader of any of my websites,  you’ve probably already noticed that all of the sites have been down for most of today.

You’ve probably also noticed that, as of the time of writing (8pm, Friday), although most of the sites are now back up, they’re all displaying posts from September, with over a month’s worth of content missing.

This is because, at about 2pm today, our web hosts, Heart Internet, called us to say that while they were working on our server (our sites are huge and take up a lot of space, so we have our own server with them), they had somehow managed to irretrievably lose all of the sites. As in, they were gone forever, with no way of restoring them.

Our server was mirrored – there were two discs, each containing the same information. They had both suffered the same catastrophic event (we still don’t know what that was, exactly), and the data on both of them was lost. Forever.

Just to be clear what we’re talking about here, they told us they had lost this site, The Fashion Police (over 600,000 page impressions per month), Hey, Dollface! (around 90,000 per month) and  Shoeperwoman (70,000 page impressions per month), PLUS all of the websites Terry has designed and hosts for our clients. Gone. All gone.

I realise a lot of people don’t take blogging seriously, or understand how someone can make a living from it, but the fact is: we do. Those sites are my livelihood. They are my full-time job. They’re what pays our mortgage, and all of our other bills. They are three years’ worth of 12-hour work days, blood, sweat and tears.

And now I was being told that they were gone for good. My whole livelihood, wiped out in the time it took me to eat lunch. ( And, OK, to watch Neighbours.)

To answer the question I know everyone will ask: YES, WE HAVE BACKUPS. Obviously, given the importance of these sites to us, they are backed up. Even with back-ups though, it would’ve taken us a long time to restore the sites to the way they were.  During that time, we would be losing money: we started losing money as soon as the sites went down, and will continue to lose money until they’re fully restored. This is not a few pennies we’re talking about: it’s our livelihood.  As it is, we’re going to have to manually restore the last month’s worth of posts to all of the sites, and all of the advance posts I had been writing for my holiday in December are probably gone for good. Which means no holiday in December, clearly.

You can only imagine the day Terry and I have had.

With Heart Internet absolutely insistent that there was nothing they could do to restore the sites, I did the only thing I could think of in my panicked state: I started venting on Twitter. Naturally, Heart Internet didn’t like that much. In one of the many conversations we had with them this afternoon, they asked Terry to tell me to stop tweeting about them, “to allow them to concentrate on fixing the problem.” Obviously this begs the question: what had they been concentrating on BEFORE that? Naively, we’d assumed they were concentrating on restoring the data from our ruined server. The only inference I can really take from “stop tweeting so we can concentrate”, though, is that they were actually concentrating on… Twitter.

I did not stop tweeting. I WILL not stop tweeting, or blogging, or speaking freely about the things that happen to me, because that is my right. I understand that it wasn’t pleasant for Heart Internet to read the hundreds of re-tweets and comments about them. But the fact is, it’s not my job to protect Heart Internet from criticism by keeping quiet about something that was going to have such a huge impact on my business and life.  I related what was happening, as it happened, and I got a huge amount of support and advice from the people who follow me (and many who I’d never even heard of until today) – and that was a huge help to me, so thank you to those people.

Obviously, given that the sites are now (mostly) back up and running, albeit with posts missing, it turned out that Heart Internet WERE able to restore the data after all. Why did they call us at 2pm today, then, and say they were all gone forever, and there was absolutely nothing they could do about it? I’d love to know.

I will be writing more about this (um, probably much, much more) once the dust has settled and I know exactly what has happened. For now, our priority is to get the sites back to normal, so it looks like we have a long, hard slog ahead of us.

And I never did find out what happened at the end of Neighbours

COMMENTS
  • I’m glad to hear you’re back up and running but have seen your tweets and these people clearly have no idea how to treat their clients. I’m kind of shopping around for a hosting company for mysf at the mo. No prizes for guessing which bunch of morons have made it to the top of my “Not if they were the last company on earth” list!

    I hope you guys get up straight soon.

    Bx

    October 30, 2009
  • Bless you. I remember how awkward A Small Orange were when I left them and went to another hosts – I lost a lot of images and odd posts here and there but to lose EVERYTHING is devastating, and the thing is, its out of your hands for a lot of it.

    Hope you get back up and running soon! *goes to do a back up*

    October 30, 2009
  • Glad things are getting back to normal and things could at least be partially restored.

    Moving forward, you need to rework your Disaster Recovery strategy and I’d recommend disk image based “bare metal” backups be taken, if possible. This allows you to backup not just the data but all the hosting server’s configuration etc.

    To put this into perspective, last time I had a “catastrophic event” I was back up and running within one hour…including the trip to PC World to buy a new HDD. As I also employ Transactional Logging, I lost no data at all – not even the stats from the page impression I got 1 second before the disk failed!

    In short, there are some good solutions out there. Feel free to ping me on the included email if I can offer any advice.
    .-= Ben Rose´s last blog ..Ecotronic Penguin Torch – No Batteries! =-.

    October 30, 2009
  • UGH! I’m so sorry to hear all of this! I’m glad you got most of it back. I’m REALLY hoping you find the advance posts because you need a vacation more than anything now!

    I think I need to back up my site tonight.

    Please pour yourself a big glass of wine! Hang in there!
    .-= Kristabella´s last blog ..DINAO Week 15 – The Halloween Edition =-.

    October 30, 2009
  • Can’t believe them asking you to stop tweeting – what did they expect to happen?! That you’d just go “oh well never mind, I’ll start up again!” Some people will never understand that bloggers make money, but Heart Internet *Should*. They should be thoroughly ashamed of how they treated you and the entire situation.

    As for Neighbours… Donna’s “maybe brother” appeared. That was it. Granted it would have been more fun to watch that than have the nightmare you did, but you didn’t miss much 🙂
    .-= siany´s last blog ..Lazy Girls’ Guide to Halloween =-.

    October 30, 2009
  • I haven't heard of Heart Internet until now, but just for you I hate them. They are poo-poo heads.
    .-= Tracey´s last blog ..The Parsley Pot has moved =-.

    October 30, 2009
  • Am absolutely livid on your behalf. Make sure if you see a drop in blog income this month, that you forward the bill onto Heart Internet. Utterly disgusting!
    .-= Erin Le Clerc´s last blog ..Pretty Vintage loves Snow White make-up tutorials! =-.

    October 30, 2009
  • I can't believe the web hosting company treated you that way! It's pretty apalling, especially since even though they recovered most of the data and you have all the backups, it's affecting your winter holiday… maybe you should start inviting people to do guest posts for FP and your other blogs so you can have some posts stockpiled while you recover everything else?
    .-= Veta´s last blog ..music video features me [wearing vintage!] =-.

    October 31, 2009
  • Hey Amber

    I don't have any knowledge about any of this stuff but I can understand your frustration at the situation. I can't believe they ask you to stop tweeting so they can concentrate – was anyone from there even replying to you? Surely if they lose so much data twitter should be that last place they are looking too. I haven't been on twitter for a few days because of my work schedule so surely they can manage to tear themselves away from it to try and restore what they broke!

    Keep tweeting and keep blogging and I hope everything gets resolved sooner rather than later.
    .-= Mhairi´s last blog ..Weight Loss Update – the first in a while =-.

    October 31, 2009
  • Wow, this is B-A-D. I noticed that the site was down and that my reader got all confused declaring “old” posts as new ones. Thank heavens for backups. As the poster says: (Try to) Keep calm and carry on (and have some tea!) Best wishes!!
    .-= Nina´s last blog ..Things I have learned today =-.

    October 31, 2009
  • Phew thank goodness you got them back (although this really does beg the question of how they managed to do it when the sites were supposedly lost. Are other, less brave tweeters, perhaps just being told it's game over?)

    This all got me wondering how much responsibility a web hosting company actually has for your content/earnings. Really, they ought to be liable for anything you lost during this time, I think, but how much could they be held to account for if they really did NEVER return the sites? It's s a difficult question!

    October 31, 2009
  • Rock Hyrax

    REPLY

    Wow – can't believe you got it together to write a piece after that day you had.

    Call me naive, but I'd have thought it was the web hosting company's responsibility to look after your data as well as the server hardware. Clearly they believe they might just be a tiny bit responsible for your data as it appears they do monthly backups. What's the point of paying for a business web hosting service if it's your own responsibility to keep a copy of your entire data? I mean what are you supposed to do, have a script to download all the comments every day? (I can't imagine transferring all those GB on a regular basis.)

    And the lackadaisical attitidue they had makes me suspect they thought you'd go away quietly and expect them to do their jobs properly – heaven forfend that they shouldn't be able to b*gger off home early on a Friday.

    In my opinion those people behaved amazingly unprofessionally and deserve any bit of damage that may have been caused to their reputation. Trouble is, I have a feeling this attitude towards backups is more common than we think – this isn't the first time I've been surprised by poor backup practices…

    And to think I've been considering setting up a blog and thinking, "I wonder who Amber uses, they can't be bad if she has several sites with them?"

    October 31, 2009
  • That's terrible news. Glad they were able to fix it in the end. It's really annoying when people say it's x, and then if you rant enough they're able to fix it. They should just fix it in the first place. I wondered if they would have tried as hard if it wasn't for your twitter ranting.

    October 31, 2009
  • Hoshi

    REPLY

    Good news- I can see your images!

    November 1, 2009
  • Amy

    REPLY

    Wow, what a nightmare. Wonder what their response would have been if they had just deleted a obscure blog and not a full-fledged business? Customer service is a dead concept.

    November 2, 2009
  • How's it been since then? I'm with Heart as well and mostly they've been great but I lost all my sites for 6 1/2 hours last night – had some nail biting moments watching them come back bit by bit. There was nothing on the web hosting status page about it, which didn't help.
    .-= annabelt´s last blog ..Bizarre screenshot of the day: RSS directory? =-.

    November 8, 2009
  • Yet your still with Heart Internet, interesting

    January 7, 2013
  • Mike

    REPLY

    I am sorry you have had these problems.

    You are not alone. I too have had abysmal service from Heart Internet and lost a lot of information that they now want to charge for to retrieve!

    You are right to be vocal.

    October 10, 2017
POST A COMMENT