The Great HTML5 Gaming Performance Test: 2016 edition

We've done two big performance tests in the past: once in 2012 and again in 2014. Given it's been about another two years and technology is moving as fast as ever, it's time to take another look. As before, we've gone through most of the devices we have available for testing (skipping our growing pile of devices which are either defunct or eventually died), and run our standard performance test (updated to use the latest Construct 2 runtime). We've also tested some of the newer publishing options like Cordova and Crosswalk for comparison.

As we said in 2014, two years of updates across operating systems, drivers, Javascript engines, WebGL, improvements to our own engine, improvements to supporting libraries like Cordova and Crosswalk, and new hardware generations make a big difference - and HTML5 support is now in a great position.

Focusing on mobile

We're focusing on mobile devices because that's the main concern for performance. Mobile devices tend to be weaker (although this is changing!), and in the past have had less mature software (also less of an issue now). Additionally, HTML5 game performance on desktop basically hasn't been a problem for a few years now. The generally more capable hardware combined with the maturity of desktop software meant that even in our 2014 test all our desktop systems basically got top scores. By now it would be surprising to find a desktop-grade device that didn't get a top-score of 60 FPS on our test. I can't remember seeing any complaints from users about performance in a desktop browser in recent times, so we're not even bothering to test desktops this time around. This in itself is a very positive sign of the maturity of HTML5.

The test

As before, our favourite test is the venerable sbperftest. It's based on a real-world game using our engine making it a realistic test, and it's automated so user input cannot vary the results. It measures the frames-per-second (FPS) once a second and the average FPS over the test duration of about one minute is the resulting score. There are some extra measurements for interest relating to jank and CPU utilisation, but these do not affect the score. It's also a relatively demanding test: at some points there are quite a lot of sprites on-screen at once, and a lot of blending explosion effects. As a result we find it is quite a good indicator of a platform's performance - if sbperftest gets a good score, many other Construct 2 games should perform well too.

The results

So, after a day of running the test on lots of devices, here are our results!

Analysis

All of the devices performed well - even old devices such as the 5-year-old iPad 2 and the 4-year-old Nexus 7. The only wart was I was surprised to find the Nexus 9 doing pretty badly with the system webview. None of the other Android devices had trouble with their system webviews, so it seems to be a device-specific issue. I filed bug 594154 so hopefully Google can look in to that. The great thing about the Android system webview is it now auto-updates from the Google Play store, updating regularly like Chrome does. So Google can fix this and have the fix in a later webview update. Gone are the days of waiting for an OS update to update the webview! (At least on newer Androids, anyway.)

I think the great thing about these results (ignoring the Nexus 9 issue) is the consistency. The results are good across a range of different operating systems, browser engines and wrappers. In 2012 many devices failed to reach 30 FPS, in 2014 it was a mixed bag, but now the results are mostly in the 55-60 FPS range. Many of the devices that scored 58 or 59 could in fact generally hit 60 FPS, but timer accuracy, the odd v-sync skip or a one-off jank bumped down the score a little. Whether games are playable is no longer in question; it's only a matter of how smooth it is.

It's interesting to follow some of the devices which have scores in previous tests. In particular the iOS devices improved a lot: even just since 2014 iOS has gained WebGL support and then introduced WKWebView improving Javascript performance. Since 2012 some of the improvements have been incredible: Cordova on the iPad 3 went from 3 FPS to 56 FPS, and on the iPad 2 from 14 FPS to 57 FPS. Safari has also steadily improved, with iPhone 4S scores of 34 FPS (2012), 40 FPS (2014) and 52 FPS (2016); similarly the iPad 3 scored 30 FPS (2012), 42 FPS (2014), and 54 FPS (2016). New hardware like the iPad Air 2 glides through the test perfectly, even as a Cordova app. The aging Nexus 7 seemed to lose a bit of performance (it never seemed quite the same after the Android 5 update), but still performed well with both native app wrappers. The old and low-end Lumia 520 improved from 47 FPS in 2014 (with Windows Phone 8 and IE11) to 53 FPS in 2016 (with Windows 10 and Edge). As ever, software improvements are able to extract more performance out of the same hardware.

The recent generation of devices are already practically perfect. Even the iPad Pro with its gigantic screen aced the test with a steady 60 FPS. It appears around 2014 most hardware reached the point where performance was perfectly sufficient. Devices coming out more recently rival the specs of some laptops, so there is starting to be overlap between mobile and desktop performance. This will only get even better. In future, there will be hardly any reason to consider mobile devices as weaker. They are already closing in on desktop-grade performance!

Webviews are fast now

We are definitely happy to see the end of the non-browser wrappers like directCanvas and Ejecta. These proved very difficult to support and were always missing some major features, and we deprecated them all last year. Now we are pleased to see Cordova with real browser engines actually exceeding the performance of the non-browser engines. A relatively limited directCanvas could score 54 FPS on an iPad 3 in 2012; Cordova with WKWebView now exceeds that with a feature-complete browser engine. Again, Ejecta even with WebGL support scored 51 FPS on an iPhone 4S in 2014; now with Cordova, 56 FPS. So we have reached the point where the non-browser wrappers - which were specifically designed to be faster than webviews - are actually slower than a webview! We are glad to see we made the right move to deprecate the limited non-browser engines.

In the past webviews were often slow and missing major features. This is now history. Webviews across all major platforms - iOS, Android and Windows Mobile 10 - support full-performance Javascript with WebGL, Web Audio and more. In other words webviews are now on par with their browsers, and there is no performance disadvantage to wrapping a HTML5 app with something like Cordova. Crosswalk performed solidly in our test and works with Android 4.x devices, providing a great workaround for older Androids (which still have limited system webviews - although all our tested devices were Android 5+). And the auto-updating system webview on Android 5+ is just great: HTML5 apps can keep improving even if Android itself never gets updated by the carrier. The slow but indefatigable process of Android updates will inevitably have Android 5+ on a high enough majority of devices that Android 4.x compatibility won't matter, so apps will then be able to drop the APK size overhead of Crosswalk.

Conclusion

I don't know if it would be too controversial to call HTML5 performance on mobile a solved problem now. There is still a large but shrinking set of old Android devices, and GPU blacklisting remains an increasingly uncommon but annoying problem. However I don't think it's controversial to say that HTML5 performance on mobile will soon inevitably be a solved problem, just like it is on desktop. With mature software and hardware close to desktop performance, mobile is no longer missing anything. Modern devices are already great and as ever the technology will only continue to improve even further.

I think this proves that HTML5 is a great technology for games. It's a consistently fast and mature cross-platform technology, even when wrapped as a native app. The standards compliance of modern browsers is excellent, too. Construct 2's engine is mature as well, having been one of the earliest-released HTML5 engines in 2011, and now with five years of upgrades and maintenance is probably one of the fastest and most reliable HTML5 engines on the market. It's great to see this carrying over to native apps on both desktop and mobile. We're intrigued to see how things will keep advancing in future!