The best thing to do would probably be to not use the layout brightness effect. Instead, use two layers and give one of them brightness effect (probably the bottom one). Put all the objects that should be changed in the changed layer and put the object that should be unaffected in the normal layer.