looks a bit like z fighting, maybe set the shadow layer z elevation to 0.01 or so
That did help! I didn't realize you could change the Z elevation of layers, thanks for that. The effect is mostly gone at 0.01, but I still get literal edge cases on the shadow edges when it's being cast from an object that is already behind shadow:
I don't think it's z-fighting this time, I pushed the Z elevation to 3 units and still got the same effect.
EDIT: Figured it out, my shadow light was on a separate layer from the shadow casters, once I matched the Z elevations of both layers it started working again.