There's a big difference between "bullet on collision with brick", and "brick on collision with bullet".
Can you describe how you mean it? Because I don't see a difference at all.
How do you check the collision? You can try to improve the collision checks, i had same problem with my game which had around 150k collision checks but after messing around with the events i improved it down to 1k only.
I had simple bullet-on collision with dot-destroy dot and bullet
I tried to divide it into sectors, so first I check if bullet is in sector "A" and if yes, check for collision with dots in sector "A" it helped to reduce checks/tick from 12000 to 2000 but I still feel it is too much because performance on mobile is not smooth. Also I'm testing in on low end mobile, with snapdragon 212... still I feel like there should be a smoother way.