I found the problem, Minecraft decided it couldn't open up region files anymore, so when it couldn't open a region it was just creating a new one in its place lmao (actually not so funny, pretty bad code choice imo). The last backup was taken at 6:20am ET, about 3 hours before I stopped Utopia to investigate. Their is a possibility we will have to restore from that backup and thus lose about 3 hours of work on Utopia. Currently I am looking to see if we can get around that, depending on how badly the world files were corrupted.
I didn't take damage when I tried to drown myself, but I didn't mess with any blocks. Also, the enter/leave messages for residences still appeared when you crossed the boundaries...
loadOrGenerateChunk() ... actually quite what the caller wants. You could attach a custom WorldGenerator for town, one that stops the server when it's asked to generate a chunk. Won't help for the wild, though.
So if the other servers weren't full of people keeping chunks loaded, would this be happening on them too?
Ok I did not have to restore from a backup, it appears none of the original region files were overwritten by the new "wild" ones. However players may have lost a small amount of progress that was not saved before the issue started. Please let me know if see any problems, I am opening the server now. Please also be aware that the live map is still a little jacked up, it will take an hour or two for it to repair itself.
I am not 100% sure what caused the issue, I only know what the result was. Without trying to decompile and understand the actual Minecraft code it is hard to point at exactly what it was. My best guess is that it was leaking file pointers (keeping files open longer than it should) and thus ran out of the ability to open any more. Then when it couldn't open it, it just figured it didn't exist so it generated a new one, but it also couldn't save that new one to disk because of the same problem.
I'm glad the problem was identified! Three hours isn't too bad, if it comes to that. Thanks Justin for all your work! And everyone else for keeping the suspense up with me, lol. I kind of wish I was in town when this happened, actually. I'd be interested to see my lot all wilded up. Ah well!
Sounds logical but scary at the same time that we dont know what caused it to happen... hope it doesn't happen again :3
Well the good news is that we do multiple local and offsite backups. The absolute most we could lose (per server) is 24 hours of work, and that would be an extreme multiple disk failure or something of that sort.
