Thanks for replies.
What I'm trying to do is have ladder which the character can climb up but you can use your mouse by pressing mouse left button to drag and drop it and when dropped it uses physics to fall down, and I have pushable boxes where character can push boxes around, and when I have the ladder being dragged with the mouse and collides with the box it flings the box away, which I do not want. But thanks for feedback and I'll keep trying to figure something out.