A new architecture for publishing web content to desktop

41
Official Construct Post
Ashley's avatar
Ashley
  • 25 Sep, 2023
  • 2,306 words
  • ~9-15 mins
  • 13,176 visits
  • 5 favourites

For several years, if you wanted to publish web content made in HTML/CSS/JS as a desktop app, the answer has been to use Electron or NW.js. For us, we've long supported publishing games made in Construct, our fully browser-based game creation tool, to desktop using NW.js. However we recently began to feel like we'd run in to the limits of this approach. In the end we decided to build something new. This blog outlines the problems with the existing desktop publishing frameworks, why we went with our own solution, and how it works.

Limits of Electron/NW.js

Electron and NW.js both have a similar architecture: it's a full copy of the Chromium browser engine (used in the Chrome, Edge and Opera browsers), but also with node.js integrated. Node.js provides access to lots of additional capabilities, ranging from hosting servers to file system access, allowing desktop applications to do more than they otherwise could in a normal browser.

For us are three major downsides with this approach:

  1. As is widely remarked upon in the tech world, shipping a full copy of a browser engine with your app is pretty inefficient. To illustrate this, an empty Construct project exported to HTML5 is just 240 KB compressed, but an empty NW.js export for Windows 64-bit is about 116 MB compressed. I don't think this matters as much as people think in an age of terabyte storage drives and 100 GB game downloads, but it is inefficient if you have loads of different copies of browser engines installed with all your apps, and it also loses the benefits of auto-updating browsers.
  2. Over time, for us, the node.js component has largely become redundant. Years ago we used node.js for things like file system access, clipboard access, and so on. Virtually all of that is now possible in browsers themselves. We moved over to using browser APIs, and now we use node.js for virtually nothing. It doesn't seem great to have a highly complex component integrated unnecessarily. All we really need is the browser engine by itself.
  3. Integrating C/C++ SDKs is tricky. Accessing a C++ component from node.js opens up a whole can of worms involving Application Binary Interfaces (ABIs) - the precise binary data and function calls used to interact with a native addon. This has become important for us as users want to be able to integrate their games with services like Steam and Epic Games. Modern node.js has something like four ways of writing native addons, and the most modern approach looks OK (if you can find the right samples and documentation for it), but is still quite complicated to use. Things get still more complicated when integrating with NW.js, which so far I've been entirely unable to get to work at all. Further, ABI changes between versions mean you have to keep recompiling native addons with every update, which is a maintenance pain. To illustrate the problems with this, we previously used Greenworks to integrate NW.js and Steam; it required regular rebuilding, and is now unmaintained (a risk you take relying on small open-source projects). I don't believe any such equivalent exists for Epic Games either. What we really want is a simple and direct as possible way to integrate a C/C++ SDK with JavaScript.

So in short we want to avoid bundling a browser engine, skip the node.js integration, and have a better way to integrate C/C++ SDKs.

Other frameworks

There don't seem to be any existing frameworks that fulfill all of this. There don't seem to be many desktop web frameworks to begin with, probably because Electron/NW.js are the de-facto standard options.

One option could be Tauri. This looks like it may well be a very good framework for many uses, but two things put me off it. First of all it appears to be based on Rust - a modern alternative to C/C++. Not a bad choice on their part, but in our case we want to integrate with C/C++ as directly as possible. Rust can integrate with C/C++ SDKs, but it adds another layer which can end up getting in the way, similar to the role node.js plays in Electron/NW.js. The second reason is basically that it means relying on another highly complex third-party component. When and whether to rely on third parties is often a difficult judgement and has various tradeoffs in both directions. However in this case I felt most existing frameworks don't give much thought to games, and anyway we already had something pretty close to what we wanted. So I felt it was time we brought things in-house.

Microsoft WebView2

I've been very impressed with Microsoft WebView2. It's a modern web view control for applications to embed web content. It launched in 2020 with the new Chromium-based Edge browser, and uses the same browser engine. This also makes it a relative newcomer compared to Electron and NW.js (which both appear to have got going around 2013-2014). Its features include:

  • It uses a shared browser engine which is installed by default on Windows 10/11
  • It's also based on the Chromium browser engine so has excellent compatibility with existing web content run in Electron/NW.js
  • Automatic updates, just like browsers
  • A comprehensive C++ SDK providing good integration with web content
  • Maintained by Microsoft, regularly updated, well-supported, and used by Microsoft themselves in some of their own apps, so it looks dependable in the long-term.

This does a lot of what we want, and it seemed so promising that we actually already built a Windows WebView2 export option in addition to our Windows NW.js export option. An empty Construct project exported to Windows WebView2 is just 530 KB compressed, as it doesn't have to bundle a whole browser engine or node.js - in fact it's just an overhead of 290 KB over a web export (when compressed).

The main downside is: WebView2 is not a framework. It's more just the browser engine component of a framework (and in fact Tauri uses WebView2 as well for that purpose). In our case, we built a small C++ wrapper application that embeds WebView2 and loads the Construct project. It works great, it's very lightweight, only needs a relatively small amount of in-house code, and gives us full control.

The only thing it's missing? A way to integrate C/C++ SDKs. So we decided: let's build our own extension system for WebView2.

An extension SDK

Building extension systems takes a fair amount of work. We kept things simple. There's an SDK to build a small DLL file for added features, which we call a "wrapper extension". There's a minimal message-passing system for communication, based on exchanging small bits of JSON data. It does everything we need and avoids having to set up complicated bindings for direct calls. The technical details of getting data from a Web Worker (where the engine runs by default), through to a C++ app, and then across a DLL boundary, are all pretty complicated. Here are few of the stages a message sent from JavaScript to C++ goes through:

  1. First post the message from the Web Worker to the main document
  2. The main document can then stringify the JSON data and call chrome.webview.postMessage() to send it to the WebView2 control
  3. The wrapper app listens for the WebView2 WebMessageReceived event, and parses the JSON string
  4. Based on the received JSON data, the wrapper app decides which extension the message should be sent to
  5. The JSON data is packed in to "plain-old-data" for a stable ABI for crossing the DLL boundary, and calls a method in the DLL to send it the data
  6. The method in the DLL unpacks the data back in to more useful STL types like std::string, and then finally calls the message handler

Sending a message from C++ back to JavaScript essentially does that whole process in reverse. Despite all that, we came up with a system that to the furthest extent possible hides all of those details to make it as easy to use as possible for integration, including the ability to make async calls from JavaScript which later resolve with data returned from the extension.

As an example of how we made this easy, here's the code to send a message from JavaScript to a C++ extension from our addon SDK:

this.SendWrapperExtensionMessage("show-messagebox", [message, title]);

Then the C++ extension receives a message like this (slightly edited for brevity):

// C++
void WrapperExtension::HandleWebMessage(const std::string& messageId, const std::vector<ExtensionParameter>& params, double asyncId)
{
	if (messageId == "show-messagebox")
	{
		// Parameters have to be converted to wide strings for Windows APIs
		const std::wstring& message = Utf8ToWide(params[0].GetString());
		const std::wstring& title = Utf8ToWide(params[1].GetString());

		// Call a Windows API directly
		MessageBox(hWndMain, message.c_str(), title.c_str(), MB_OK);
	}
}

That demonstrates how once you've got everything set up, there is minimal code and complexity to get from JavaScript to C++ code (in this case, calling MessageBox). There are no complicated direct bindings and no need for build systems other than a standard Visual Studio solution. As mentioned JavaScript can send async messages, and messages can also be sent from C++ to JavaScript. You can read more about how it works in our wrapper extension SDK documentation, and some starter code for a custom wrapper extension is provided in our Addon SDK.

As this is all in-house code, we also control our own ABI, which is a small interface and easy to keep backwards-compatible. So extensions can be built once and work long-term without needing rebuilding, even when the browser engine is updated, and even if we upgrade the wrapper application itself.

Integrating SDKs

To prove it works, we then integrated the latest Steamworks SDK for a new Steamworks Construct plugin. We were also able to customize our code to make it even easier to integrate than before. Then we went one further and integrated the Epic Games Online Services (EOS) SDK for a new Epic Games Construct plugin, which was not previously supported at all. We've published the source code to both those plugins on our GitHub account to give developers confidence they can tweak and extend them as they need. And then for completeness, we also published a new File Construct plugin using the same extension system for unrestricted file read/write access - as while Chrome supports the File System Access API, it's currently only able to access files and folders that the user chooses through a picker dialog. (If this WebView2 proposal we made is accepted, then even this should be unnecessary as we could do that with the File System Access API as well.)

It all works well! It's been a few week's work to set everything up, but I think it's a solid system and makes it as straightforward as possible to integrate C/C++ SDKs with JavaScript. This unlocks the full capabilities of traditional desktop games and apps for use in Construct projects, even though it uses a HTML5 engine.

Future work

Despite how well this new approach has worked out, there's still one big limitation that remains: it's currently Windows only. However the official WebView2 roadmap lists macOS and Linux support on the way, so it looks like it should be possible to port this approach to both other desktop platforms in future. Particularly intriguing for a game engine, it also lists Xbox support, and at time of writing the corresponding issue has an "Awaiting release" label (described as "dev work has been done and in release pipeline"), so that looks like it might happen soon. We'll definitely be investigating that right away when it's out, aiming to use the same extension system to integrate Xbox features via the GDK.

Another step we are considering is building a native node.js addon that implements our extension architecture as well. This would let the very same wrapper extensions we use in WebView2 also work in NW.js, which would also give us a way to support macOS and Linux in the interim before WebView2 support arrives. That however depends on being able to figure out how to build a native addon for NW.js, which as I mentioned so far I've not managed to get working at all! I'm sure it's possible though, so it'll probably just come down to a matter of eventually figuring out the right configuration. Then even if we have to rebuild the node addon with every NW.js release, the ABI between it and the wrapper extensions is stable, so the wrapper extensions never have to change.

Conclusion

While it doesn't yet have the same platform reach as Electron and NW.js, we feel this is a better approach for the future, especially as we anticipate WebView2 will eventually have equivalent platform support (and perhaps even including Xbox!) It gives us the best of both worlds of JavaScript and C++: the same single codebase running across all platforms with outstanding performance and a broad feature set, and the platform integration of C/C++ SDKs. WebView2 is lightweight and efficient using a system-shared browser engine rather than bundling everything with the app, it auto-updates ensuring the best performance and security, there are no big unnecessary components in the stack, and having the extension system in-house means we have full control over it and the ABI.

I think WebView2 is the way forwards for publishing web content to desktop. In my view it's already the best option for Windows, and it should bring similar benefits to macOS and Linux in future. So anyone else considering how to publish web content to desktop should definitely consider WebView2, or frameworks that use it like Tauri. And my congratulations to the Microsoft WebView2 team, who've done a great job making an incredibly useful and high quality component!

And if you want to try out our extension system and write C++ components for Construct, check out our wrapper extension SDK documentation to get started!

Subscribe

Get emailed when there are new posts!

  • 26 Comments

  • Order by
Want to leave a comment? Login or Register an account!
  • Super nice! I feel so much passion and excitation in this article.

    That's really cool to have a passionate team Scirra. Thanks for all the good stuff and making such a great tool!

  • Love to hear Construct is taking more steps to move away from third party solutions which caused nothing but pain.

  • I see auto-updating as an issue, releasing your game on top of a moving target means it will break

    File size is a complete none issue for the platform it targets

    Thanks for talking about your plans for Linux and Mac, I was worried that NWjs gets dropped in favor of WV2 without having a replacement for them

    Very excited about XBox

    Backwards compatibility for addons is huge, it really helps trust if devs don't have to fear that they could rot away

    Only thing I don't like is the auto updates. Nice for development, but for release I want to nail it all down, lock in the version I did all the testing on. But I did see that there's a fixed version too

    Would be awesome if we could choose between the evergreen and fixed version

    I was still a bit skeptical about WV2 because of some unanswered questions I had, but this blog addressed pretty much all of them and now I'm very excited about it, thanks

      • [-] [+]
      • 4
      • Ashley's avatar
      • Ashley
      • Construct Team Founder
      • 4 points
      • *
      • (1 child)

      Construct content has generally worked fine for years with breakages being very rare, so I think auto-updating is a good default. (Other updates like OS updates or driver updates can always break stuff too - you should always be prepared to do some maintenance!) But if you want to ship a specific browser engine with your app, you still can with WebView2 using "fixed" mode. It's documented in this guide.

      • Thanks for the link, I didn't know there was already a tutorial for it. Bookmarked.

        Yes it is rare but consistent that things just stop working on the web. It happened to me and only with the live browser version, while the NWjs version was still fine.

        Though fixing it was easy because I still have a sub, and Construct and I are still around to update it.

  • Nice work! Great to see exploration of more Export options.

  • This is great news for the future of C3 :)

  • Would be great if it will be supported by Xbox in the future. 🥰👍

  • Webview2 for xbox is slated for a Winter 2023 release. We could be quite close using C3 on consoles again.

    github.com/MicrosoftEdge/WebView2Feedback/issues/2155

  • A milestone for you and us! I would expect alot more desktop games from C3 users in the near future... And who dares to say, maybe the start of consoles. Great work, Ashley!

  • It sounds very promising and finally we can export Desktop game without wasting users' space.

    It was a great challenge and I'm glad you made it work very well!

    Thank you for your effort! :)

    • [-] [+]
    • 1
    • Fib's avatar
    • Fib
    • 1 points
    • (0 children)

    I was very skeptical of the Windows 10/11 only requirement since I was thinking there was still wide usage of Windows 7/8. But looking at the Steam hardware survey made me feel much better about it because it says 95% of Steam users are using Windows 10 or 11. So I might actually use this WebView2 export in the near future for my next game I release on Steam.

  • Load more comments (14 replies)