I don't think a game with 1 vote should be in top rated games list at all. You can submit a game, rate it 5 stars and have it jump into the top 6 list. That's not right is it?
After reading this post last night I gave 5 stars to a new unrated game to see what effect it would have and it's now 2nd in the top games list with 3 votes. Maybe there should be top new games list and top overall for games with more than x votes.
Some sites don't show a rating until the game receives a certain number of votes. I think that's a good idea because the first few votes are influential and you get people thinking a game is worth 3 stars but giving it 1 to counter the 5 it already has.
You're right about people not voting though. It's the same with tutorials. The top rated (> 1 million views) has only 379 votes.