50-100 layers is way too many in my opinion. Layers should not be used for fine grained Z ordering, that's what Z ordering itself is for. Layers are useful to group large numbers of objects in Z order, like background, foreground, parallax layers, and so on, which means generally you'd have under 10 layers.
Layers are fast if they have no effects (no shaders, and opacity must be 100% and filter white). In this case objects are just drawn directly to the screen as if everything was on one layer. Otherwise, the effect has to be processed on a separate texture, which can be expensive, but is still fully hardware accelerated. Processing 50-100 layers with effects will be very slow, but without any effects they have no performance impact at all.