It shouldn't be any problem. Everything outside the view is not rendered, so will not use draw calls or fill rate, so it shouldn't have any effect on performance. The only thing that could affect performance a little bit is total number of instances in your layout, since picking becomes more heavy with more instances.
But everything in memory is loaded when the layout start, so memory use would be the same regardless of where you scroll to.
Everything outside the layout will be loaded also, but shouldn't affect the fill rate, or draw calls.