I can think of a lot of reasons for this not working the way you describe it, it would be far easier to solve if you could provide the capx..
First thing is using the pin behaviour combined with the physics behaviour, physics has joints, why not use those?
Second thing is disabling the collision for all objects that are not active (if they are all on screen (only z order controls visibility) they could all be interacting)
Third thing is the placeholder (does it itself have physics behaviour), how have you set it up?