There are a million possible reasons why this may happen and without seeing your project it's impossible to tell.
The most common mistake is creating lots of objects over time and not destroying them. Run the game in Debug Mode, play it a bit and check the list of objects on the left, see if there are any unusually big numbers there.