So I checked my vote bonus and it was at 64 (haven't voted in a few days) so I voted and checked again, and now I'm at 60. wut.
it doesn't take it away until you next vote as it runs the penalty calculations on vote. however when I do voting results for contests, I always run a check "and last vote was in past 30 hours"
I have had a similar problem. I can't vote on Sundays, so I had never been able to big vote streaks. Since awhile ago I could go without voting and keep my streak.