Hard to help without seeing your code.
It might be due to the host syncing the peer's position. In the official examples the host moves the peer. So the peer will always update its position according to the host's game state. If you try to move on the peer directly, the peer will be synced back to the position of the host's game state. Peers only send their inputs, then the host moves the peer.