One idea is to define my own borders within a layout at max size but that method would be very complicated and tedious to implement...
I'm not sure why that would be very complicated and tedious. Can you explain the issues you have with that method?
I think that is your easiest route. Make the layout very large. What is the downside of having large parts of the layout unused?
You could cover the unused area with a tiled background, block it off with solid objects, or whatever else you need to do, depending on your game mechanics.
If you are thinking of issues more so with the level editor part, you could set the area of the tilemap, which would stop them from trying to edit the tiles outside the level. If not using a tilemap and something like drag and drop, you could just have an event to check on drop if the dropped position is greater than the level width/height.
Maybe there is a way to change the layout size now, but I know in the past it was not possible. Even if it is, I'm still not sure how it would make anything that much easier for you here.