Make sure your physics objects are using the fastest options: World Solver should be Linear (fastest), and World Friction should be Adaptive (also fastest). Using exact is much slower.
Apart from that, physics involves some pretty complex calculations, so it will slow down. You should design your game to use a handful of large objects as opposed to trying to do particles with physics on them, which would be extremely slow.