it goes left according to his point of view (it goes to the left of the down gravity, so to the right of the up gravity, not sure I make much sense, note this occurs the same way with gravity to the sides), so this is the expected behavior.
you will have to hard code the controls depending on the gravity using the simulate control actions rather than using the default controls.