I have some invisible text objects in the same area, but they are all on the same layer so clicks should pass through, right?
Why would you think that?
Using invisible sprites as triggers is a fundamental part of programming. In your case it might be OK but impossible to say.
Without seeing your Events, .capx etc it is not really possible to offer any meaningful guidance.
Just in case, check you collision polygons are correct and not overlapping.