Just a question, Are you using Box2D Raycast for Shadow Cast?
Because I have seen a lot of examples from Box2D Raycast which tends to work perfect for shadow casting.
Develop games in your browser. Powerful, performant & highly capable.