Yeah layer projection is a pain
I don't use layer angle, but if the center of rotation is the center of the layout, basically the conversion is quite simple
if X and Y are the position in the unrotated layer
rX and rY the same position in the rotated layer
cX = Layoutwidth/2
cY = Layoutheight/2
you just have to take the point and counter-rotate around the center of rotation of the layer, with the same angle.
It's exactly the same formula as the other topic on planete rotating around a sun
rX = cX/2+cos(angle(cX,cY,X,Y)-LayerAngle)*distance(cX,cY,X,Y)
rY = cY/2+sin(angle(cX,cY,X,Y)-LayerAngle)*distance(cX,cY,X,Y)