Invisible objects are used all the time for such things as waypoints, collisions etc, but if you don't specifically test for such a collision, then it should have no impact.
If you're worried, then makes some tests, use debug etc, or post a .capx for others to look at.