Personally I think the OP should try this out. I'm sure this could work great to an extent. It's just that I think this would be best used for high frame count objects, that are few and far between. CPU/GPU acceleration. In either even there is a reason why the industry doesn't use video's as the primary form of animation.
As an example. This could work well for animated backgrounds. Interactable characters in adventure games. Monsters in Final Fantasy combat.
However I don't think these would work well in games where it's used for the player in a platformer. I think the buffering for the video would causes visual latency.
Also here is a question. What about transparency? Can video's have transparency?
I am pretty sure that videos can not have transparency once rendered.