This is for Chrome (latest versions always), as it has the best HTML5 audio support right now.
What you want to do is tell audio to mute playback, NOT set silent. Tag ALL of your audio files on playback (e.g. fx, music, ambient, etc.). For one time sounds (like a gunshot or car door), just tell it to stop when you press the button. For looping sounds (like ambient background sounds such as street noise), tell it to mute when you press the button.
For music, make sure you actually have it in the music folder. Music in Chrome WILL follow the timescale (make sure you have music timescale on). Make audio playback rate "0" when you press the button, and then "1" when you press it again.
<img src="http://wayofthesax.com/img/audio%20pause%20screenshot.png" border="0" />