I've done this before using multiple bullet behaviors on the same object.
One behavior heads towards the player.
The second bullet behavior, using the enemy object in a family, can be set to -
On family overlap with enemy (or within x distance) - set family bullet behavior angle away from enemy, and set enemy bullet behavior angle from family. Use a high speed, and a high deceleration. You can even set the speed inverse to their distance to each other if you want, so that they repulse each other harder the closer they are to each other. This doesn't necessarily prevent overlap, but neither do monsters in vampire survivors.
This is more akin to steering or boids style maneuvering.
I would also recommend updating steering forces only every x seconds instead of every tick, for performance reasons. You can also see this in action in vampire survivors after you die.
Regarding performance for massive amounts of enemies, you might notice there actually aren't thousands on screen at a time in vampire survivors. Try counting them. The enemies (and xp drops for that matter) are culled when offscreen and xp is combined.