"or should i just turn it on on all layers?"
You could try, but I doubt it will help much.
The only other suggestion would be to check your collision filtering.
Like if you have a lot of instances checking if they collide with a much fewer amount of some other object.
That leads to a bunch of unneeded checks.
Always try to set the order to fewest first.