If you can find a better solution to the problem of scaling, I'd say go for it; this method works, but only for static frames and it can be temperamental for reasons I can't fathom.
I just found this thread, so this might come too late.
I made a very similar approach, but with another "anchor".
Please ignore the german comments, I made them for my own use.
My idea behind it is to start with a virtual screen, independant from the window or the layout size. In this case that virtual screen was determined to have a size of 1920x1080. The layout may have any size (although it doesn't make much sense to have it being smaller than the virtual screen), e.g. 4000x4000. The window may also have any size, in this event sheet it is 1024x768. The black bars are on an inherited layer with 0% scroll and zoom ratio. All calculations refer to the virtual screen size (width=1920, height=1080) and the display width and height. I then switch to fullscreen in another event sheet.
The example above works with scrolling and I couldn't find any issues yet. But I could only test it on my own pc/monitor, so maybe there are issues.
EDIT: Forgot the important information that the calculations assume an aspect ratio where the width value is higher than the height value. 16:9, 16:10, 4:3 etc works, but for aspect ratios like 4:5, 1:2 etc you need to adapt the calculations to be based on the height instead of the width.