The site was down for over a day. I want to start by apologizing, this length of down time is not acceptable and I am taking steps mitigate the future risk. The website side of the EMC platform lives in the Slicehost/Rackspace cloud. The reason I chose to do this is that you can scale it easily and it is very reliable (hah). I will say I have used them for over 2 years and never had a problem before. During the downtime the game server were NOT affected, as they live on their own dedicated beastly hardware in a different data center. What happened is that the machine the website was living on crashed hard. It ended up with corrupt file system and took them over a day to recover. Once it was back up I had the wonderful task of reparing database corruption. Some I had to patch in from backups. The only data (that I can see at this time) that was lost was rupee transactions (history). These were purposely not backed up because they are large (millions and millions) and loss of them does not hurt us. For those of you that have been here for awhile you know that one of my top goals is to provide as much uptime as possible. I plan feature releases and updates very carefully so we don't need downtime, especially on the game servers. You can be assured I was going nuts and could barely sit still over the last 24 hours. I wrote more angry emails and had more angry phone calls than I think the rest of my life combined. Once again I apologize, and thank you all for sticking around.
I would also like to add that during this time the staff did not have Square. So they were operating in the stone age without things like /report to help Special thanks to them for working harder than normal. Also a big thanks to the players that spread the news about what was up with the site.
Yay, thanks for bringing it back to life! Btw: its hard to host more than 16k accounts and thousands of active threads and posts, as also a system like TEXP and all the rupees transactions right?
Tis okay Justin. You do such a great job, one little website malfunction isn't gonna scare us away that easy
You did all you could Justin, just tell Rackspace to go find there business elsewhere and get a new host
Thank you Justin, We do not care about a few days of downtime, it's to be expected. Your doing such an amazing job and thank you for fixing it so quickly, I'm sure that the domain hosts are hiding behind an obsidian wall
Broadcast is great - shame it doesnt broadcast to all the servers at once tho. Iv never switched servers so many times
Thanks for the thanks and don't beat yourself up about it Justin. You did all you could, at least everyone could still play.
I used to use slicehost. they were ok, but i switched to linode for vps hosting after rackspace stated changing things. linode's pretty good, they've a few data centers around the globe, and you can chose which one you make your vps on, so you can have a couple in different parts of the world and balance them or whatever it is you do with high traffic sites. Plus there control panel is much more indepth than slice host, and they're cheaper
Yeah I am still playing with the idea of going with another "cloud" (vps), or just giving in and putting a pair of load balanced dedicated web servers in the same data center (Florida) that the game server are in. I mean the site is pretty busy, about 5k unique visitors and 30k page views daily.
We had similar problems at ZBS (Zerobullshit) (TF2 and latterly Minecraft) - hosting services can disappear (with your cash sometimes) only to re-surface under a different name. My mates finally gave up the ghost trying to run it, understandable but such a shame for oldies like me who enjoyed the forums PS Though pretty useless at TF2 lol - By the By ZBS mates won the TF2 world championship (800+ teams competing to the best of my knowledge)
haha i'm on the tiny plan because i keep my website a secret I often look at the massive plans and think at what point is a vps just silly. you know if you can only fit one vps on a physical server what's the point i'm sure you'll figure out the best way to go, but i had to give linode a little plug i'm pretty happy with them
