This seems to work well:
Set scale to 0.5*OriginalWindowWidth/min(min(camera.X, layoutwidth-camera.X), min(camera.y, layoutheight-camera.y)*OriginalWindowWidth/OriginalWindowHeight)
min(camera.X, layoutwidth-camera.X)
and
min(camera.y, layoutheight-camera.y)
are the closest x and y distances to the edge of the layout
using the aspect ratio I can then convert the y distance to a x distance:
y*OriginalWindowWidth/OriginalWindowHeight
So now there are two x distances, and we can get the smaller one with another min.
min(x, y*aspectratio)
and to convert the x distance to a scale we multiply by 2 and divide by the window width:
2*xdist/originalWindowWidth
As an additional idea you can also limit the zoom if it gets much too close near edges.
min(scale, 4)
or whatever max scale you want.