How do I

does anyone know how to enlarge the dotted black line rectangle on the layout layer (the window size)?

On the right hand side of the screen, there is a folder under projects that is the base of your project. If you select that, the properties panel on the left will show an option called "Window Size."

You can also get these same properties to show by clicking the view link on the left side of the screen next to where it reads "Project Properties."