Yeah the trick was to only make the ball joints visible out of the physics objects. The connectors have no physics and are simply based on the position of those ball joints - hiding the sometimes erratic physics that lies beneath. Works much better than I expected.
Thanks for the feedback :)