Yes, thank you I completely forgot about that.
Yet, the problem with the restarting audio still persisted. I didn't mention that my prototype is an audio-only game. Since the player won't see the objects anyway, I ended up rotating the whole world around the player, this way the audio doesn't restart.
This might not be a solution if you are working with visuals, though.