8/6/2022 Website Crash

Discussion in 'Empire Help & Support' started by Aikar, Aug 8, 2022.

  1. On 8/6/2022 the server that powers the EMC website had a catastrophic drive failure (even under RAID 1, both failed at same time).

    It really doesn't surprise me, as this server was our oldest one, and had been due to be migrated off before this happened, but sadly it got me...

    This server had quite a bit of responsibility for my infrastructure, so it was quite a hit for me, taking out my email, internal DNS for emc network (But thankfully I had the database address hard coded anyways on 9 of the 12 servers so they stayed up, and I fixed it for the last 3 saturday to get them up too), as well as my primary DNS control (DNS is replicated, but I controlled it from my server)

    So my first priority was regaining control of domains and email and that was pretty much most of saturday, movin them over to cloudflare (been a TODO for me anyways to reduce how much responsibility this machine had)
    as well as working with host to get replacement machine ready.

    By sat night I was able to get some of my web services restored.

    But EMC website was a bit rougher.... As you know our software is on older versions, and this new server came with a new version of PHP programming language. This has required me to do a lot of fixes to get code up to par with the newer version.

    Some configs also were not part of the version control, and I'm having to restore those. This has impacted the ability to buy items from supporter store as the configs for what they are in game are missing, so I've disabled that part of the page for now.

    Subscription based ranks work differently and should still work - chicken has tested it with a voucher.

    A few supporter rank renewals were held up for Sunday but Paypal should be retrying them now the site is up but I will make sure things are fixed if anything does go wrong there.

    As you may notice, forum avatars are missing. Those are semi recently backed up to my local home PC, but I recently had a failure on my home pc and was down for a while, so it's likely around 1~ month old now but I am still in progress restoring everything to my home pc too so I will try to get what I can restored from that soon, but in mean time you can just go re-set your avatar.

    But with that, bugs are expected! Please report issues you've seen on this thread if it hasn't been reported yet.
    You may also notice the site is a lot faster now. The PHP upgrade did come with some nice performance boost, but sure is looking snappy! It's also on newer hardware, newer OS, so it's def a lot faster than the old server.

    Sorry for the interruption with all this. Def been a chaotic weekend for me but I will keep working through things as fast as I can.
    Egeau, ChumMiner, Fred_TWK and 14 others like this.
  2. Known Issues:
    • Blog/Landing page images are halfway borked. Must be manually fixed.
    JNightwind, MoreMoople, 607 and 2 others like this.
  3. Thanks for the hard work!
    Tuqueque likes this.
  4. The wiki is also faster. :)
  5. Thanks for your hard work!!!

    Vote bonus is not showing. o.(o)
  6. A bit of a weird one, probably unrelated, but when clicking the Events button, it automatically moves it over to the wiki button and stays stuck there, even when clicking other buttons.
  7. Does this mean the forums could use newer versions now or still limited due to ties with payments, etc.?

    So fast now though, especially the wiki! A fortunate side effect of a bad situation :D

    Also would this help us get PM, forum post email notifications again?
  8. What a time to return after a long break. :rolleyes:
    Thanks for your efforts getting everything back up and running! And I've never doubted that you make regular backups.
  9. Nope. The code has been updated to work with a newer version of PHP, but the main issue with updating EMC is that there is a lot of custom code that would not work with newer versions of XenForo (the forum software).
  10. If you mean the tab highlighting, its always done that because the events page is under the wiki, and i wasn't able to make it fake highlighting that when under a specific page of the wiki.
  11. Email sending is something that is down yeah and will try to get up soon as I can.
    ThaKloned likes this.
  12. Hmm, the earlier behaviour has stopped, it was that the tab highlighting got stuck on wiki, even when clicking on empire, for example.
  13. I didn't log in for five days and came back to the website having suffered an amount of devastation not seen since the Battle of the Somme ended.

    Haven't suffered an EMC-related shock like this since the summer of 2018. Nature is healing.
    wafflecoffee likes this.
  14. thanks for saving us upon a double disk failure!

    also thanks for the new php version wow wiki loads super fast (will it stay fast tho?)
    FadedMartian and UltiPig like this.
  15. The battalion of creeper explosions failed to take down the site :p.

    Thanks for your hard work into getting the site back up!
    KatydidBuild likes this.
  16. the minecraft skin avatar 3D head and 3D helm head display the others image than the one that is above the label.
    3D head displays 3D helm head instead of just the head and the opposite for the 3D helm head.
  17. yo aikar, having an IT background, i realize double hard drive failure is no fun, so I am glad and appreciate your hardwork on this one…

    this low key could be a “blessing” in disguise as the Wiki is faster than ever.. heck yeah!
    Fred_TWK, ThaKloned and Kryarias like this.
  18. For sure, glad it worked out. In terms of a blessing in disguise - that improvement would have happened when he migrated the server regardless. Just it forced his hand when it needed done. Whoops.
  19. Great to see things back so quickly, thanks Aikar and chickeneer and whoever else is involved in the restoration.
    607 likes this.
  20. So you are aware, you can also check your vote bonus on your rupees page. Showing fine there atm :)
    FadedMartian likes this.