Okay, I don't have a great solution for you. This one works but it's a bit of a cheat and it isn't as smooth.
I used a particles object and zeroed out all the random behaviour, gave the particles gravity, and then positioned it to my draggable circle.
It takes a bit of tweaking to get it to work right since the particles and the physics engine use different scales for velocity and gravity.