We had some badges that we removed which would have caused drops. Also there's a few gaps in the system where it doesn't give/take away rep as it should. It's still a work in progress system.
Don't worry about pushing that button, I push the button for everyone every now and then so it would have happened anyway!
For what it's worth, your rep looks about right at the moment :)