Countmatching expression picks all selected instances. I don't think the ships were filtered at all, so it picked all. You can guess the outcome.
Try this... create a (global) variable named 'allships', that tracks the total number of ships (though that may not be necessary, if you use Ships.Count, but I'm not sure if that is influenced by selection).
then...
+for each formation sorted by formation.UID
+inverted | ship('idformation)=formation.UID //this should select all ships that are NOT in the current formation)
-set "ships" to 'allships'-countmatching(ship) // all ships minus ships not in current formation = number of ships in the formation
Empty formations should have 0. A bit roundabout, but this should work. Perhaps someone else has a more elegant solution. Also, try using Ship.Count instead of 'allships', if it is not influenced by selection.