Then yes, I think you're approaching this the wrong way. You're making the WHOLE game go faster. You should only change the parameters of the elements that have any incidence in increasing the game's difficulty. Such as enemies, platforms, etc. So you should do this in a "case per case" basis instead of changing the whole game.
You have to change your approach and this problem that you're having won't happen anymore. Good luck