Unfortunately, I haven't found an actual fix, but instead a weird workaround.
I moved all my audio files into the sounds folder. Yes that means that music files aren't being streamed which is less efficient, but it doesn't seem to have made any noticeable impact.
Then I moved every sound effect I want to play at timescale 0 to the music folder, and preloaded them.
While this is really stupid as now Sounds are in the Music folder and Music is in the Sounds folder, everything is placed in sub-folders so it doesn't really matter.
Then its just a matter of changing the Audio object's property for "Timescale Audio" to "On (Sounds Only)."