I seem to have found and fixed the problem.
Originally I paused the player sprite's animation while not on a platform and setting the speed to 40 while on a platform.
It seems that it sets this speed every tick and additionally this seems to cause issues when the performance drops.
I assume it is in fact linked to what you mentioned regarding the sprites collisions during frame changes.
My fix was to remove the "pause animation" mechanic and the setting of the speed while on a platform.
The animation now unfortunately does not pause, but at least the player doesn't teleport anymore