As you mentioned, your collision polygon changes each frame of the animation.
Consider making it as consistent as possible between frames. Try to have it the same shape and surface in order for the collision to be "consistent".
You also should post your project for investigation, that would makes things easier to see the issue and be able to find out what is exactly going on, in regards to how you did set up your project.