Hacking something useful out of WKWebView

1
Official Construct Team Post
Ashley's avatar
Ashley
  • 7 Mar, 2016
  • 1,521 words
  • ~6-10 mins
  • 12,260 visits
  • 0 favourites

We recently announced support for WKWebView in iOS apps with significantly improved performance, but some caveats about memory usage and video support. This was a pretty difficult bit of hacking on WKWebView to work around some serious limitations. What we found is probably useful for any Cordova app, and provides the nitty gritty details for technically-minded Construct 2 users, so here's a deep dive in to how to bend WKWebView in to something relatively useful.

As is well-known, the main reason to support WKWebView is the fact it supports JIT compilation of Javascript. This makes JS code several times faster. Obviously for a game engine, this is a must-have feature. So we went in to this with a great motivation to do anything possible to make it work.

Why not use a local web server?

If you can, do. This solves most of the problems. However it's kind of crazy to have to run a HTTP server just to get a local app to work. There may be security implications, there could be conflicts with other apps (e.g. around choosing ports), and there are probably some weird side effects like the user can probably also load your app in Safari. We also want to stick to officially supported features to have better guarantees about things like whether features will build and work, and it's important it works with build services like PhoneGap Build for Construct 2 users. None of the options I could find worked out very well:

  • Cordova Local WebServer appears to be some kind of labs experiment which isn't very actively developed or used (it has no issues, for example). PhoneGap Build didn't seem to know about this plugin, so I moved on.
  • CorHttpd failed to build with the latest Cordova iOS version.
  • The Telerik WKWebView plugin doesn't use the official WKWebView support, which we wanted to use. Apparently it doesn't work with the latest Cordova versions anyway.

In particular, choosing a port for the preview server is surprisingly hard. You have to always use the same port, otherwise stored data disappears (since each port counts as its own origin) - but what if your chosen port is in use? How do we make sure all apps built with Construct 2 always use the same port, and yet guarantee it is never used by any other app? Imagine 10,000 apps in the store using this approach, all needing their own constant but unique port. This still doesn't sound easy to work with. So I decided to try making it work without a local web server.

Loading the index page

Luckily Apple in iOS 9 fixed a problem where you, erm, couldn't load a HTML document in WKWebView. This is the main reason Cordova has been able to add official support for WKWebView in iOS 9. So your index page appears in the WKWebView, then... nothing else works.

Just a brief aside: how do you detect if you're in WKWebView? Apparently there's no obvious way to do this, except rely on the fact WKWebView supports IndexedDB but UIWebView does not. So our detection looks like "if iOS and Cordova and window.indexedDB".

Loading local files

One of the first things Construct 2 does when loading is to load the game's main data JSON file via XMLHttpRequest. This does not work in WKWebView: it appears to treat local files as if they came from a remote server, even though they're in the app itself, and such requests are blocked.

By the way, just to make this more fun, the same thing happens to scripts. Scripts can actually load from "remote" (in this case local) locations and still run, with one difference: to prevent information leaking across domains, Javascript errors are hidden. They just turn in to "Script error" on line 0, regardless of what went wrong. So you cannot get any useful information from errors! So we did a bunch of debugging with "alert" :P

However cordova-plugin-file allows us to access local files in the app. It's hardly documented, but after digging around, you can read a local text file like this:

function fetchLocalFileViaCordova(filename, successCallback, errorCallback)
{
	var path = cordova.file.applicationDirectory + "www/" + filename;

	window.resolveLocalFileSystemURL(path, function (entry)
	{
		entry.file(successCallback, errorCallback);
	}, errorCallback);
};

function fetchLocalFileViaCordovaAsText(filename, successCallback, errorCallback)
{
	fetchLocalFileViaCordova(filename, function (file)
	{
		var reader = new FileReader();
		
		reader.onload = function (e)
		{
			successCallback(e.target.result);
		};
		
		reader.onerror = errorCallback;
		reader.readAsText(file);
		
	}, errorCallback);
};

// e.g.: fetchLocalFileViaCordovaAsText("file.txt", function (text) { ... })

We just manually JSON.parse the resulting text, and it works! We can get the local data file and start loading it. But it still doesn't start up.

Loading images

Similarly to making AJAX requests, images decide that if they load from a local file, they came from a remote location. Remote images can be drawn to a 2D canvas, but Construct 2 uses WebGL rendering for maximum performance. Due to the possibility of timing attacks (extracting image data by writing shaders that take a different amount of time per pixel!) you cannot create WebGL textures from remote images. So let's go back to our Cordova plugin that can load local files. How do we get that to an image?

Blob URLs would do the job, but apparently createObjectURL doesn't work on the Cordova plugin's files - I think they are some kind of fake file object which can just be used in specific ways. Data URIs could work, but they tend to be inefficient (string parsing to load an image?!) and bloat the data size. However... we can read a local file as an ArrayBuffer, then create a real Blob from that!

function fetchLocalFileViaCordovaAsArrayBuffer(filename, successCallback, errorCallback)
{
	fetchLocalFileViaCordova(filename, function (file)
	{
		var reader = new FileReader();
		
		reader.onload = function (e)
		{
			successCallback(e.target.result);
		};
		
		reader.readAsArrayBuffer(file);
	}, errorCallback);
};

function fetchLocalFileViaCordovaAsURL(filename, successCallback, errorCallback)
{
	// Convert fake Cordova file to a real Blob object, which we can create a URL to.
	fetchLocalFileViaCordovaAsArrayBuffer(filename, function (arrayBuffer)
	{
		var blob = new Blob([arrayBuffer]);
		var url = URL.createObjectURL(blob);
		successCallback(url);
	}, errorCallback);
};

Yes, this probably involves copying the data. But it stays binary, and we can create a real blob URL to it. This works - now images can load! But we can't hear anything.

Loading audio

The Web Audio API loads audio by AJAX requesting the audio files as array buffers. Conveniently we can basically hack in an alternate request using fetchLocalFileViaCordovaAsArrayBuffer when we detect as WKWebView. This works.

Music is a little different. Audio elements can stream audio, which the Web Audio API can't do, so we use that to stream music tracks. So, why not try the blob URL trick again? It turns out that there's a 4 year old bug in WebKit that basically means media elements can't load anything from a blob URL, with one terse comment from an Apple engineer saying Safari does not support that, and then nothing.

You could jam in a data URL, but that's pretty inefficient. Luckily in Construct 2 there's a flag we can flip that just goes ahead and loads music with the Web Audio API. Since that loads ArrayBuffers, it works! All the content is local so arguably there's no need to stream. It does hold the entire track decompressed in memory, but a data URI would likely be even worse. Modern iOS devices have gigabytes of memory, so while it's not ideal, we can probably get away with this.

Loading video

Aiming for complete support of Construct 2's features, we move on to video. Like with audio elements, we can't use a blob URL here. You can throw in a data URL, as horribly inefficient as that is - we're talking megabytes-long strings here - but it hardly even works. It staggers along taking ages to load, then manages to eke out a couple of frames per second. It's a real slideshow, and it's not going to work. file:/// URLs don't work either.

It turns out that this time, we're truly stuck. There is no alternative API we can rely on. There's nothing we can do to load a local video file in to a video element. You probably really do need a local server for this.

Sadly, we resolve this by simply saying video is not supported in WKWebView mode. Hopefully most games will be able to get by, but if there needs to be video, it seems sticking with the older and slower UIWebView is the only option.

Conclusion

Apple are trying to replace a major system-level component with a complex feature set. It's not surprising that this is a long-term engineering effort and it has its problems. However it is compounded by Cordova's general patchiness and missing features in Safari. Blob URLs in particular have been around for years now, reaching the point of being a widely-implemented standard, and if Safari supported this it would pretty much provide a general-purpose workaround for loading any local resource. Alas, we must yet again look towards the next iOS release - which hopefully will add local file access too!

Given the struggle to support loading local files, and the various caveats it comes with, I thin

Subscribe

Get emailed when there are new posts!