I hope there's an easier method but I found a workaround.
At the start of each layout, I load a stylesheet. And the stylesheet contains the "html" and "body" background color of the layout I'd like. It's a bit cumbersome because I have to create a new stylesheet for each / most layouts.