Thanks, Ashley!
I've actually gone through the blogs, very enlightning. I had already applied most of those tips and tricks in there, also using a pre-loader here and the amount of objects is acceptable, the game runs nice and smooth so I had to look further.
I did find the problem a few seconds ago (after a long search...), weirdly there was one object that the "slow loading layouts" had in common, it uses pathfinding and the cell size was set to 1 because I wanted to the object to have a constant y position while randomly moving around.
Changing this to a higher number solved it, I had to wait 10+ seconds for the preloader (Loading...) to just show, now it appears in 2 seconds.
The thing is, I knew this setting was intense for the CPU, but testing it at the time didn't show any frame drops and I mustn't have payed attention to the loading time then.
Anyway, mark it as solved, maybe this information is useful for someone. :)