Response to Outage
First we would like to thank everyone for your patience through this ordeal. We would also like to thank the tech team over at The Planet for working so hard day and night to get the servers back online. With such a terrible event we are just happy no one was hurt.
In the last two days we have frequently been asked why we didn't have backups. We do have backups and they are performed nightly. But in a case like this where the data center that houses thousands of servers, including those of popular hosting companies such as hostgator.com, suffers an explosion and a resulting fire the backups are not helpful as the server itself was still intact and working. This is a major data center in Texas which is why this event made news across the country in various tech articles and websites. This is not a common issue at all, as a fire, let alone an explosion, is rare. The Planet also did have redundancy in their power to avoid downtime in a crisis, however, with the fire department not allowing them to utilize their backup generators they were not able to use any emergency contingencies.
The next question we've been asked quite a bit is why we do not have a secondary server in case of emergencies like this, where all traffic could be pointed to another part of the country. This is not a common feature on hosting plans like ours due to the cost of such a setup. We are currently looking into what options we have for providing a remote backup that would limit the downtime in a rare occurrence like this, but so far we haven't been able to find any that would allow us to maintain a reasonably priced package of under $50/mo. Some larger hosting companies may be able to provide this service by spreading the cost over several thousands of sites and many servers, but at this time we are not able to offer this.
For this reason we are offering all clients (until Wed night) who would like to convert their service to an individual template purchase the ability to do so for $100 rather then $250. We will transfer your site as is with all your content to the server of your choice. If anyone has any recommendations for hosting companies that have a remote server backup for immediate restoration of sites in this type of situation feel free to add it to this post so others can see if it is the right solutions for them. The individual template would have the same limitations it currently does, no template switching and $60 upgrades, but you will not need to start from scratch as we will move your template with all your info in it. This offer does expire this Wednesday night, June 4th, 2008.
Our first priority is customer service so we need to make sure all customers are taken care of and do not experience any delays in support due to this past weekend. Jamie and I will be researching all options and data centers this week to determine whether a server switch is necessary. Without being able to offer an affordable package with remote server backups and secondary system there is not a solution that we could have used to prevent the outage. We can, and do, have measures in place to prevent data loss. We also have high quality servers that we do not oversell and that are monitored 24/7 to minimize any downtime caused by the server itself. That being said we have received some recommendations for other data centers that we will be looking into to ensure we have the most reliable system possible. Currently we believe the likelihood of a fire and explosion happening again in our current data center is about as likely as the same thing happening in any data center that could hold our servers.
In the last two days we have frequently been asked why we didn't have backups. We do have backups and they are performed nightly. But in a case like this where the data center that houses thousands of servers, including those of popular hosting companies such as hostgator.com, suffers an explosion and a resulting fire the backups are not helpful as the server itself was still intact and working. This is a major data center in Texas which is why this event made news across the country in various tech articles and websites. This is not a common issue at all, as a fire, let alone an explosion, is rare. The Planet also did have redundancy in their power to avoid downtime in a crisis, however, with the fire department not allowing them to utilize their backup generators they were not able to use any emergency contingencies.
The next question we've been asked quite a bit is why we do not have a secondary server in case of emergencies like this, where all traffic could be pointed to another part of the country. This is not a common feature on hosting plans like ours due to the cost of such a setup. We are currently looking into what options we have for providing a remote backup that would limit the downtime in a rare occurrence like this, but so far we haven't been able to find any that would allow us to maintain a reasonably priced package of under $50/mo. Some larger hosting companies may be able to provide this service by spreading the cost over several thousands of sites and many servers, but at this time we are not able to offer this.
For this reason we are offering all clients (until Wed night) who would like to convert their service to an individual template purchase the ability to do so for $100 rather then $250. We will transfer your site as is with all your content to the server of your choice. If anyone has any recommendations for hosting companies that have a remote server backup for immediate restoration of sites in this type of situation feel free to add it to this post so others can see if it is the right solutions for them. The individual template would have the same limitations it currently does, no template switching and $60 upgrades, but you will not need to start from scratch as we will move your template with all your info in it. This offer does expire this Wednesday night, June 4th, 2008.
Our first priority is customer service so we need to make sure all customers are taken care of and do not experience any delays in support due to this past weekend. Jamie and I will be researching all options and data centers this week to determine whether a server switch is necessary. Without being able to offer an affordable package with remote server backups and secondary system there is not a solution that we could have used to prevent the outage. We can, and do, have measures in place to prevent data loss. We also have high quality servers that we do not oversell and that are monitored 24/7 to minimize any downtime caused by the server itself. That being said we have received some recommendations for other data centers that we will be looking into to ensure we have the most reliable system possible. Currently we believe the likelihood of a fire and explosion happening again in our current data center is about as likely as the same thing happening in any data center that could hold our servers.
posted by Portfoliositez.com at
6:20 PM
8 Comments:
I just wanted to say that I think you girls handled the situation very well. Stuff happens and there was nothing on your end that you could prevent, you kept us all well informed of what was going on, and I appreciated it very much.
ditto
I agree completely! Stuff happens and even the internet can prove to have problems. Thanks for your great customer service!
You'll have to kick me out!
You are an awesome team and I could not imagine taken my business elsewhere...I don't believe that I would get any better service than what you all give anywhere else! You have proven time over and again that you care for your customers!
i wasnt affected, as i already have my site hosted by other domain (i had only just purchased a year with them, as i was refered to your site).
I work in IT - and we do offer same kinda deal - with backup and other stuff, we have organised downtime every month for a few hrs, to basically check if its all running smoothly, and also, to prevent things in future.
so, downtime - well, u guys sorted it out, and as i can tell, blu did suffer a LONGER down time :)
so, u girls - rock :)
You guys were fantastic. Anyone wanting to view my site was completely understanding. Thank you for the great communication.
I think you handled the outage just fine - though I would have appreciated receiving this blog post as a blast to my email account, how would I have known that I could purchase the template at a discount without thinking to check your blog?
I don't want to do that anyhow... but just think your customer service model should be a little more proactive (just as I recommended that we should have received a notice about the outage as an email when it happened).
Regardless, your work is beautiful and your prices can't be beat. Thank you.
Post a Comment
<< Home