this roll back was a lot more than an hour. I crafted 10DCs of sea lanterns only to find them in crystal and shards form again. and i finished crafting those like 4 hours ago.
It appears that i still have a very valuable item missing, i can't find my life anywhere and I'd appreciate its speedy recovery. Oh, wait...i haven't had one of those since discovering video games. Carry on. Joking aside, thank you so much for working to fix the issue so quickly! Now go to bed, you need your rest so you can heal!
Phew, I thought it was just me. I found an area I had mined out completely reverted, and I started it hours ago.
Yeah, same here. I appreciate the quick work but I'm now missing an addition to my sugarcane farm I'd made earlier which is really my biggest concern because it was a pain to build. Edit: I'm also missing 2 of my op swords and my op bow :/
Thankfully I don't seem to be missing anything tho I had noticed that the restore had given me a duplicate of my diamond voters pick axe and messaged SS+ to take the duplicate from my vault.
If you find yourself with copies of an item that you shouldnt of had due to the rollback, please report it to SS! As for rollback time... even if timezone played a difference here, im not sure how it wouldnt of created an error instead of using a different time. rdiff-backup --force --restore-as-of '2016-06-07T02:20:00+04:00' /opt/minecraft_backups/server_$1/$file $file
ok well i dont think theres anything left we can do from an automation/technical standpoint here now. Going to get some sleep, and well have to work out recovering from this w/ SS now. We will make things right where we can, just please send a PM.
Wouldn't it be -04:00 instead of +04:00 ? IIUC, you have restored 8 hours older data (?) IIUC, not only players who were online are affected, but also all shops they have used. That's the nightmare situation for an IT admin. I'd like to express my moral support: good luck! Try to get another pair of eyes with anything regarding backup and restoring. It's especially hard with a service that should be available 24/7 ...
Ouch! I hope this can get fixed enough, but well, things like this happen, and partly we'll just have to deal with it!
Wow..item replacement happened quick..I expected 2-3 days before the time was made for me...thanks a lot to the staff.
It has been brought to the SrStaff's attention that the backup may have been later older than we had hoped. This will open up the time frame significantly for potential issues. Contact us if needed!
Maybe you should consider to apply ZFS to that server, if possible of course. Allows for instantaneous snapshots ("backup") as well as very easy recovery; either a full rollback or per-file. The advantage of a snapshot is that its pretty instantaneous and doesn't gobble up too much diskspace because only the changes after a snapshot has been created get stored. And of course it also provides plenty of options for offline backups. Either through commands native to ZFS or your regular tools such as rdiff, tar, etc.
Man Aikar, you're probably the most dedicated dev I've seen. Working to fix a critical issue on low sleep AND recovering from surgery? Most programmers would be taking it easy.
This is essentially what rdiff backup also does, but adds gzip compression. For our 50gb~ world folder, the backup folder which encompasses 1 week of backups at 1hr increments only takes like 70gb. It's very efficient and very good for restoration too. I simply was panicking to get things online and normally used relative time formats for restore, but I was worried if I did that, I would accidentally cross into the bad data time frame so I wanted to use exact. def looks to be the case... I must of accidently typed + in the panic I had also just taken a damn prescription pain killer and was drowsy, so that surely didn't help. It was suppose to be a no risk deployment... just 1 line of code put in the wrong place that that related to another line of code that was in the right place, and I tested the good one, and thought "Hey, the logic change is good" and cut a corner of testing.
I have lost my voters armor, and part of a tower I built yesterday.... The one time I choose to play, and move my voters armor......
Message us at pmss.emc.gs and we'll get these things replaced. Accidents happen, even as a developer, but we'll make things right.
I don't think anyone has mentioned this here yet but even though the world was "restored" rupees have not been. This basically means that any purchases were completely reverted (so the shop owner gets there items back) but the rupees have not been reverted. I spent roughly 4k on hoppers yesterday only to find them completely gone and to find that my rupees have not been refunded aha. It also seems that player inventories were not synced. I also lost hours worth of hardened clay and endstone but no proof of that. If the rollbacks are done by the hour then how come another rollback couldn't be done after the first mistake was seen for those of us who didn't log in until after the mistake was realized?
