This seems to be an issue with particular devices, especially Samsung devices. On my phone, for example, I've tried some AAA sort of games that also have this delay. There are also some, like Cut The Rope, that don't have this delay, which means they used Android audio a different way either intentionally (to avoid the delay) or it just happen that way.
Unfortunately, I don't think this is ever going to be resolved for our phones, so we'll have to accept it. Don't worry, half the people I've tested my game on don't even notice that anything's delayed