Aaand after asking, I think I've got it, and I think I've found a bug:
Playing audio at position seems to use the position relative to the layout, rather than the position relative to the visible window area, to determine position. So since one layout I'm at the bottom of, and the other layout I'm at the top of...