You need to share a small example so someone can investigate it and help you as it's not clear what is happening here.
In general (Bound to Layout or Dragg) it shouldn't interfere to destroy an object as long both are overlapping.
It's hard to say without looking at an example.