Everyone, EMC will be going down Wednesday night at 11:45PM EST (EMC Time) Our data center has this as a maintenance window to replace a power supply on one of the servers. However, the recent cause of the SMP1-3 downtime was due to the system unexpectedly running out of file system space due to the latest mc versions using more size per chunk (I do have monitoring on this but due to reasons of configuration, I hadn't received another warning that it was now dangerously low). I was hoping to solve this issue with some code level changes (such as improving compression format and changing how local backups are stored), but it hit critical before a solution could be put in place. To resolve this, I've ordered new hard drives to be installed on all the game machines so I can split some of the data to the other drives to solve this issue, and have this planned to be done by the data center while they are having them offline for their own maintenance so we can get by with 1 maintenance window. Since the work is out of my control, I can't give a real estimate, but the DC has quoted 30min to 1 hour (Starting at midnight) so estimate up to 1 to 2 hours of potential downtime. Sorry for the inconvenience. This only impacts the game servers. The website will still be up. Technically, games and Utopia will be up as they are on a different server than the main 9 but the proxy servers that let you access them will then be down... so access to them will still be difficult. games.emc.gs may work if you want to try to get on games. I might update DNS to point that to the backup (and typically unused) proxy.
How will losses from transferring items onto these 3 servers before a backup be dealt with? Will they be recreated with sufficient evidence, or are they simply lost forever?
i see kicking the server racks didn't help. joking aside, thanks for the updates. hope you went western digital rather than seagate for the hard drives.
what do you mean by that. i do not believe the servers are being transferred. he is adding hard drives.
some items were lost due to the following, the system unexpectedly going down without a backup being made means that time was rewound for one server, but not the others- things put into the rewound area will disappear- including EMC items.
if there is a distant back up, and they needed to utilize that, i am sure they would of course use the most up to date backup. the items you speak of will probably be lost forever, even if a back up was needed. From my understanding of his post, it sounds like the only data that would be lost was that since the last backup. I think the servers back up automatically around the time of restarting each night.
Items lost during that period should be reported to https://pmss.emc.gs. Add Aikar to the conversation as well. A relatively low number of items were lost, and only about 5 players were actually affected, is my estimate. (myself being one of them )
While frustrating, this is not as bad as it could have been. Is there a known time frame where said items were lost?
Thanks for the update and happy there is a solution in place! I hope nothing happens in the meantime before the maintenance. I know I was one of the affected players during this time but I don't remember if I was even carrying anything. Once again great work getting these issues solved!!!
The nature of loss varies during the window. Since the issue crashed the servers, its not like it was a long window of gameplay that was "exposed". The servers crashed due to it, but I was away from home so took me a while to get home to resolve it is why the downtime was longer. During the time, anytime a chunk or player data tried to save, it occasionally failed. In some cases, this means the player data was left as the last time it was saved and any item a player acquired after last successful save was lost. Another case is where the player data saved partially in a corrupt state and the entire player inventory was lost. Please let us know ASAP if this happened as the backups are about to be gone for this to be restored! I did make a copy of the backups of potentially impacted players so that if the backups do expire I have a copy, but if it hit someone that didn't hit my logs I won't have your backup after Tuesday. Another case is where a chunk partially wrote itself, but couldn't finish, and resulted in the chunk corrupting and wiping out a chunk, only had a single case of that so far though and thankfully was an easy restore. The final case is like player data, chunk didn't save so it was left as the last successful save, so if a player put items in a chest they would be lost, or alternatively, if a player took items out of a chest, they may of came back. If you ended up with dupes, please report to SS too.
We all know why EMC is closing down This week I am working on assignments so I don't think the shutdown will affect me as much. Thanks Aikar for keeping the servers from dying haha
Looked pretty funny when a chunk in the middle of my creeper farm was gone when I did a driveby. Got some new 1.15 stuff in that chunk tho . Again thanks for the quick restoration of the chunk, and well done doing it without any complications!
