Every object that exists on the layout will be loaded into memory upon the layout start, unless you manually manage memory. Which isn't recommended unless you're very familiar with the system and really need it for some reason (you shouldn't).
Basically no swapping objects won't make a difference, because the object you are swapping to is already in memory anyway regardless.