Can't that somehow be streamed? I mean, always load the current layout and always already start streaming the previous and the next layout into the vram - and kick out everything else.
If you jump to a layout that's not the next, you'd get choppyness.
That said, it would be cool to hint a preload.
Like an action: load layout x into VRAM
you could trigger it with enough time and if there's not enough time, well, choppyness.