I'd suggest making families of objects that you want to pause, which don't include the player, and setting their time scale to zero.
You might be able to set the entire time scale to zero, and then restore the player object time scale only, but I'm not sure if that works.