The surprising difficulty of resizing images on spritesheets

19
Official Construct Team Post
Ashley's avatar
Ashley
  • 6 Jun, 2020
  • 1,992 words
  • ~8-13 mins
  • 4,704 visits
  • 7 favourites

Spritesheets (aka texture atlases) are a common technique in computer graphics to combine multiple images on to a single image. However if an image on a spritesheet is resized smaller, things get surprisingly difficult, due to the way it interacts with mipmaps. This manifests as "seams" or fringes of unwanted content along the edges of an image, such as shown below.

Example of a seam along the edge of an image

I've even spotted it happening on buttons in Google Docs!

A spritesheet seam happening in Google Docs

Here's what's going on and why it's a tricky problem to deal with.

Spritesheets

Spritesheets just mean pasting multiple images on to a single large image. For example a simple game might use a spritesheet like the one shown below, with different object's graphics all placed on to one "sheet".

Example of a spritesheet

Each image can still be rendered separately, since GPUs can efficiently render a smaller area of a source image. Using spritesheets helps improve efficiency since there are fewer images to download and load on to the GPU. It can also save memory and improve rendering performance.

Mipmaps

Mipmaps are a way to improve the performance and quality of resizing images. Resizing an image smaller with good quality is computationally intensive, since you have to consider all the data in the full resolution source image. To make this more efficient, a series of high-quality resizes are calculated ahead of time. The image size is halved each time, at 1/2 scale, 1/4 scale, 1/8 scale, etc. all the way down until the image is a single pixel.

Example of a mipmap from Wikipedia

This uses a bit more memory (a third more, to be exact), but improves performance and quality. For example if the image needs to be rendered at 1/8th scale, the GPU can pick a pre-rendered high-quality resized version of the image from the mipmap, and render that. This improves quality and performance since it doesn't have to do a computationally intensive resize on-the-spot.

There's more to mipmaps than that, but that's the basic principle behind them. It's also worth pointing out mipmaps are often used automatically - for example even browsers use them when resizing images on web pages!

Spritesheets and mipmaps

Here's where things start to get trickier. You may think that spritesheets and mipmaps work independently. Actually the problem is when you resize a spritesheet really small, multiple separate images end up combined together. Take a look at our example spritesheet from earlier, and consider rendering just the spaceship sprite in the bottom left. The GPU will cut out the area for just this image from the spritesheet, shown by the blue dashed rectangle below.

Cutout area for the spaceship on the spritesheet

To demonstrate the problem, imagine this image is being rendered at a really small size - so small, it picks the mipmap that is just 2x2 pixels big. This will have combined the entire spritesheet in to just four pixels! This mipmap, scaled back up to the original size, looks a bit like this.

Spritesheet resized to 2x2

Notice that the entire bottom-left quarter of the spritesheet has been combined in to a single color! If we mark out this area on the original spritesheet, we can see it includes other unrelated image content other than the spaceship we want.

Lower left quartile of the original spritesheet

All the colors in that area have effectively been mixed together in a pot an then painted over the whole quartile of the image. Now the GPU will render the space ship image area from this mipmap, as shown below.

Spritesheet resized to 2x2, with original sprite area shown

Now you can see the problem: the color the GPU will draw from the mipmap is the average of the spritesheet plus other images - so will be slightly the wrong color. In this case it'll be too grey, as some of the grey gun sprite above the spaceship has been mixed in to this pixel of the mipmap. This is sometimes called color bleed, as some content from outside the spaceship image area has "bled" in to it.

This is an extreme example to show the principle of color bleed on mipmaps. It can actually happen at any mipmap level, such as at half size - but it's less likely. It also tends to manifest at lower levels as a "seam", or fringe, along the edge of one side of the image. To see why that's the case, take a look at the spritesheet with a 16x16 grid drawn over it.

Spritesheet with 16x16 grid drawn over it

On the 16x16 mipmap, all image content within each grid cell will be combined in to a single color. Where the image edges don't fit this grid exactly, they will be combined with an adjacent image. As another example, look back at this image I showed earlier:

Example of a seam along the edge of an image

This is an artifical example I made to demonstrate the problem. The source sprite sheet actually looks like this:

Example spritesheet causing seam

Now you can see where the blue fringe comes from - a big blue sprite directly below it on the spritesheet. I deliberately made the source images large so that displaying them at a small size involved resizing them down a lot, therefore using small mipmaps. The images look like they align to each quarter, but they're not actually perfectly aligned; consequently some mipmap levels have the blue sprite bleeding in to the purple circle's image area. This also happens inconsistently across the mipmap levels, manifesting as a flickering line as the circle sprite resizes smaller.

Looking back at the Google Docs example, you can see icons are rendered from a high-resolution spritesheet of lots of different icons, and an adjacent icon bled through.

Demonstration of how seams appeared in Google Docs

Causes of the problem

In practice, this is actually relatively rare. As noted you have to be unlucky enough to have all of the following happen:

  1. Resize an image considerably smaller than its source size
  2. Have the image positioned on a spritesheet unaligned to the mipmap pixels
  3. Have another sprite adjacent that has an obvious and opaque color right up to the edge of the image
  4. Lack enough spacing or transparent borders on images to overcome the resizing

Our game development software Construct automatically arranges images on to spritesheets. By default all images have 1px of transparent border applied, and then on top of that are also spaced out 1px apart on the spritesheets. This means there is usually at least 1-2px of transparency between images, helping prevent color bleed with modest resizing. Experience shows this is sufficient for most games. However sometimes if sprites are resized a lot smaller the fringing effect still happens.

Avoiding the problem

One way to avoid the problem is to use a source image that is smaller. If it's not downscaled, it won't use a mipmap and so color bleed won't happen. However this is not always possible, since the object may be shown at a range of different sizes.

Another way would be to throw out the whole spritesheeting approach completely and go back to using separate images. However this also throws out the efficiency gains. On top of that, since the problem is relatively rare, it's not always necessary. We want to keep things efficient if possible.

The best solution I've found is to align and space out images by power-of-two sizes. For example if an image is 100x100px, it should have a power-of-two size allocated on the spritesheet (128x128px). Then that space itself should be allocated at a power-of-two position, e.g. with its top-left corner at (0, 0), (0, 256), (128, 1024), etc. Since mipmaps reduce the size by powers of two (1/2, 1/4, 1/8...) this ensures all images are aligned with mipmap pixels as far as possible. Technically if you resize an image smaller than a single pixel it will still color bleed as the image area is smaller than a mipmap pixel - but this is hardly noticeable since the image is so small. In practice it solves the problem of fringes appearing along the edges of images.

However this itself still comes with a big downside: since images have to be padded out so much to fit to power-of-two sizes, the spritesheets can easily end up bigger, and that takes up more memory. For example the spritesheet with the spaceship I showed earlier grows from 256x256 to 512x512 when padded out this way - meaning that spritesheet takes up 4x as much memory once loaded!

Managing the problem in Construct

It's difficult to solve this problem automatically: there is a fundamental trade-off between tightly packed spritesheets, which save memory and generally work fine, and padding spritesheets, which uses more memory but in practice completely prevents fringing.

This problem can happen anywhere images on spritesheets are resized smaller - including in web design with major web apps, as noted with Google Docs! It happens in our game development software Construct as well, and some users have the problem of color fringing along sprites. Since it's difficult to see how it could be handled automatically, we added a user-controlled option for it. We named this Downscaling quality. The default medium tightly packs spritesheets, and generally works. However if there are issues with color fringing, the high mode will pad out spritesheets and avoid the problem - at the cost of increased memory use. To discourage unnecessary use of this option and needlessly wasting memory, "high" mode is marked as "not recommended". The hope is nobody will select it just for the sake of it, but anyone having trouble with fringing can opt in to it to fix rendering issues and accept the cost of increased memory use. It's a tricky tradeoff to communicate in a simple dropdown list - hopefully the negative language will at least prompt people to look it up in the manual, which explains in more detail what it's for, and help users make good decisions about using it.

Conclusion

This is a non-obvious rendering artefact that's probably not widely understood, especially in web design. It's interesting to explore the way computer graphics works and explain how it comes about.

However I guess the wider lesson is that features rarely work in isolation. It would be easy to think spritesheets are an optimisation that doesn't really affect anything else, and also that mipmaps are an optimisation that doesn't really affect anything else. This is naive and in fact the combination of these two optimisations causes some awkward rendering problems with difficult trade-offs - even though both features are working correctly and exactly as designed! It's not even a bug, it's an unintended side-effect of two separate optimisations. Even providing an option to users for this is tricky, since it's hard to explain in a brief sentence what the option is for and why you both might want to use it and not want to use it.

Even more generally, a rule I often think about is "everything is always more complicated than it seems". It's easy to look at each feature and optimisation in isolation, but when you think about the whole system, awkward and non-obvious results can emerge. The same point came up in another blog post I wrote about two years ago: Software development is hard: a collision bug post-mortem - that also involved the intersection of various features and optimisations working in an unexpected way, resulting in an unexpected bug.

And as for users - don't change settings you don't understand, or you might end up with negative consequences! Make sure you understand what settings do and why you are changing them - and if in doubt, read the manual 😉

Subscribe

Get emailed when there are new posts!

  • 9 Comments

  • Order by
Want to leave a comment? Login or Register an account!
  • Glad I found this, those little lines were driving me nuts! Thanks for the detailed explanation.

  • Thank you a lot for the explanation. It`s great to see that Contruct`s author is glad to explain how his engine works) I wasn`t even hoping that my post will persuade such big and detailed answear! Thanks a lot for your great job!

  • I have wondered about this for years, thanks for the detailed explanation

  • Thanks. Now I can stop the bleeding.

  • As a NoCoder I was never know why is this bleeding and i thought it is a bug. finally i understand why this bleeding.

    Thanks for this clarification.

  • Thanks for this post Ashley. Another way I'm thinking could be to always have blank (transparent) offset to all our images? So every sprite will be 0.1-1% unnecessarily larger, but all unwanted bleeds will definitely be transparent anyway?

      • [-] [+]
      • 1
      • Ashley's avatar
      • Ashley
      • Construct Team Founder
      • 1 points
      • (0 children)

      A fixed transparent edge does not always solve the problem. As noted in the blog post, the only way to really solve it is with power-of-two sizing and positioning, via the "High" quality downscaling setting.

  • I appreciate the write up. It's much for clear now. Thanks!

  • Thanks Ashley, great explanation. Thanks also for allowing the control in C3, so an informed dev can make the appropriate choice.