Construct's effect compositor: part 1

One of the cool things you can do in Construct is add multiple effects to an object. For example you can add a Warp effect to distort the object, and then apply a AdjustHSL effect on top of that to adjust how the colors appear. It all "just works", showing the result of the combined effects. It also works reliably cross-platform everywhere from Xbox One to iOS thanks to the portability of WebGL.

An example of the Water effect, an advanced water-rippling style distortion.

How does this work under the hood? This is actually one of the most complicated parts of the runtime. It's a great example of our philosophy of doing the hard work behind the scenes to provide a simple, intuitive and very cool feature.

We call the code that renders chains of effects the effect compositor. In this blog I'll give an overview of how it works. It's going to get quite technical. Due to its complexity it's difficult to provide simple advice at the end such as rules of thumb for performance, but there are a few useful tips, and hopefully it's interesting to know what's going on behind the scenes. I'll also focus on how it works in Construct 3 - the effect compositor is similar in Construct 2, but not as well optimised. Also since there's a lot to cover it will be spread over two blog posts. This is part 1 and part 2 will follow soon!

Effects vs. blend mode

First of all, it's worth highlighting the difference between effects and the Blend mode property. The blend mode is a fast, built-in feature that GPUs provide for blending images with the background. There's a limited selection, but since they're relatively cheap to use we provide them as some options you can use. For example you can switch the blend mode to "Additive" for a useful lighting effect without the performance overhead of the full effect compositor.

Effects on the other hand are small bits of code called shader programs written to perform a specific visual effect. They run directly on the GPU, making them super fast at processing visual effects. Since shader programs are custom-coded they can do almost anything, including more advanced background blending effects like a "Multiply" blend that you can't find in the blend mode property.

If you use both effects and a blend mode, the blend mode is treated as the last effect in the chain.

Normal rendering

For an overview of how rendering works in the engine, see the blog post How the Construct 2 WebGL renderer works. Despite the name it covers some fundamentals that still apply to Construct 3. The key take-away is that normal rendering is heavily batched: so long as it's drawing lots of the same thing, it can work very efficiently.

Providing the blend mode stays the same that can be handled efficiently too. If you use one effect, it can sometimes also efficiently batch that. Unfortunately it's not straightforward - more on why later.

It's worth noting that normal rendering - i.e. sprites with no effects - are still rendered with a shader program. It's just a default program that says "draw pixels from the texture as they are". This also means if you want to render an effect and then render something without an effect, it's switching between two different shader programs rather than enabling and disabling the effect.

Basic effect rendering

Suppose you add an effect like AdjustHSL which modifies the colors in the image. This effect is simple enough to be directly rendered. In other words the process of rendering it involves:

1. Switch to the AdjustHSL shader program.
2. Draw the object to the screen normally. The shader will alter the rendered pixels.

If the next object in Z order does not use an effect, it will be followed by a "Switch to the default shader program" command. Much like rendering objects with different blend modes, this "breaks the batch" ("break" as in "split", not "broken"!) since it cannot simply make a prior command draw more quads: it has to change settings and then submit a new quad. So just like changing the blend mode there is already some overhead, simply because we are not rendering the same thing as before.

Batching effects

If there are lots of objects using the "AdjustHSL" shader all consecutive in Z order, then they can be batched. For example 100 sprites all using "AdjustHSL" could render with commands like this:

1. Switch to the AdjustHSL shader program.
2. Draw 100 quads.

There is one significant caveat to this though: shader programs can use parameters. For example in this case AdjustHSL has parameters for the amount of hue, saturation and luminance to adjust the colors by. These parameters must be set by another command, which can also break the batch. So if every object has different parameters for AdjustHSL - e.g. every instance specifies a different hue - then the batch becomes inefficient again, along the lines of:

1. Switch to the AdjustHSL shader program
2. Set the hue parameter to 33
3. Draw 1 quad
4. Set the hue parameter to 50
5. Draw 1 quad
6. ...etc...

Therefore for maximum performance, ensure shader parameters are all exactly the same across instances. This means Construct can efficiently batch them all in to one rendering command. Of course, it may be that having different parameters is essential to your game - but you may be able to do something like reduce them to a small set of fixed options and then group them across layers, to extract as much batching as possible.

Background blending effects

AdjustHSL is a simple effect because it doesn't depend on anything else. It just alters the source image according to its parameters, making it pretty much a drop-in replacement for the default shader. Background blending effects are much more complicated because they need to bring in image data from two sources - both the source image and the rendering destination - to combine them in some interesting way, and then draw the result to the rendering destination.

It's difficult enough just getting the foreground and background pixels to line up given the plethora of co-ordinate systems going on (layout, screen and object, in both texture and pixel units, sometimes rather unhelpfully flipped on the Y axis). Still, let's gloss over those particular details to one extra spanner that gets thrown into the works: due to the way GPU hardware works, shader programs cannot both read from and write to an image at the same time. In other words, a background blending shader can't directly render to the background, since it reads its pixels. Instead we have to render it to an intermediate surface performing the background blending effect, and then copy the result to the display.

Let's have a look at how it works if we render a simple explosion graphic with the "Screen" blend shader program.

The high-level steps are:

1. Switch to the "Screen" shader program
2. Switch the render target to an off-screen intermediate surface
3. Draw the sprite quad, with the "Screen" shader program reading both the foreground and background pixels, and writing the blended result to the intermediate surface
4. Switch the render target back to the background
5. Draw the intermediate surface to the background

Notice it only needs an intermediate surface the size of the sprite being rendered - but there are still several steps involved.

Also notice that a side-effect of rendering to the intermediate surface is that the background pixels are now "baked in" to the image. Later effects, such as warp, will now also warp the background pixels as well as the source image, as they have essentially become part of the foreground. Typically this is not quite what you want, so if you use multiple effects it's probably best to put any background-blending effect as the last in a chain.

Background blending performance overhead

The intermediate surface needed to render a background-blending shader is a crucial performance overheard, since it always breaks the batch. There are several rendering commands involved to do the copy, so even lots of objects using exactly the same background-blending effect can never be batched together. This is a fundamental inefficiency about background-blending effects that is difficult to do much about. For this reason you should avoid using background blending shader effects on lots of instances. If you do need something like 100 instances all using a background-blending shader, it may be more efficient to put them all on a layer and apply the effect to the whole layer. It may process a larger area, but it will avoid "batch thrashing" where the renderer is forced to generate thousands of rendering commands with little to no batching.

Reflecting on this, it's clear why the Blend mode property is definitely worth having. While you only get a few limited background-blending options, they're built-in to the GPU hardware in a way that lets it render them directly, so it doesn't need to have this copying overhead. Also since changing it is just a single command, Construct can still batch large numbers of objects all using the same blend mode. This means blend modes are much more efficient than background-blending shader effects - so use them instead of a shader effect wherever possible.

Next time...

As a half-way summary, here are some of the tips mentioned so far:

• Effects can batch efficiently if lots of instances use the one effect with the same parameters - as long as it's not a background-blending effect.
• The 'Blend mode' property is much more efficient than using background-blending shader effects.
• Since background-blending shader effects have a high overhead, avoid using them on lots of individual instances - try using a layer effect instead.
• Background-blending shader effects work best as the last effect in a chain.

We've still not got too far in to the effect compositor yet! The really heavy lifting gets involved when we have chains of multiple effects. There are also some extra complications and special cases that get thrown in along the way. (Hopefully you can start to see how this can end up being quite a headache to work with!) Click here to read part 2 and find out more about the effect compositor.