The layout size is just a number. The memory footprint is directly related with the objects that are loaded in each layout. Also, the cpu/gpu strain is being used only by the objects that are on screen or have some motion/opacity/rotation/etc change (and so those properties have to be calculated).
An empty layout of any size should not have any memory impact at all.