Safari 16.4 rolled out last week, and for us it's been a nightmare. We make the browser-based game creation app Construct. Early versions of Safari 16.4 broke opening projects, previewing projects, and all existing content published with Construct, all in different ways. I wanted to share our experience so customers, developers, regulators, and Apple themselves can see what we go through with what is supposed to be a routine Safari release.
Most browsers provide pre-release versions for early testing. For example Chrome Canary and Firefox Nightly update daily, and there's also less frequent dev and beta releases. Apple provide Safari Technology Preview (STP), but it's only for macOS, and does not update to any public schedule. It seems to be about once every 2 weeks. Pre-release browsers are usually pretty rocky with obvious issues that get sorted out soon enough. However when things start making their way to beta, it's time to look more closely. So when Safari 16.4 beta 1 was announced on February 16th (also not to any public schedule), we started taking a closer look - and there were a lot of problems.
Breaking opening projects
Construct projects are based on zip files, and we use a popular library zip.js to read them, which in turn uses the Compression Streams API when supported. Safari 16.4 added support for the Compression Streams API, but it was somehow incompatible with zip.js, which meant opening projects in Construct usually failed. Potentially this also means many uses of zip.js across the web are broken too.
I duly reported the issue on February 17th. After some confusion about whether the issue was a duplicate of another (it wasn't), Apple engineers investigated, identified the problem, and said they had it fixed by February 27th. Not a bad response in the end, and good work by Apple employees.
Then Safari Tech Preview 165 came out on March 8th. It still had the bug. The Safari 16.4 release seemed close - but we didn't know for sure, as we don't know its schedule. What should we do? Did they ship the fix and there's still a problem? Or did they not ship the fix yet? But why would they have done that when it was over a week since the fix? Should we assume Safari will ship broken compression streams and issue an emergency patch to work around it? Surely they wouldn't risk breaking zip.js for everyone? Even if we do that, we don't know how long we have left to act. Should we instead do nothing and hope Safari 16.4 ships with a working fix? But then we won't know if we face disaster or not until it's already rolling out. It's a terrible dilemma.
I asked whether the fix would ship in Safari 16.4, and an engineer responded talking about STP. Unfortunately that doesn't answer the question: there is no particular guarantee that anything in STP will be in Safari 16.4.
In the end we decided to see what Safari 16.4 shipped with. Meanwhile, consider that during everything else described in this post, we weren't able to open most projects in Construct. Also consider that our only option to figure out what really ships is generally just to keep manually testing every Safari release, which means we have to waste time trying to find out something Apple surely already know. After an agonising couple of weeks wait, STP 166 came out on Thursday March 23rd, and then Safari 16.4 finally rolled out on Monday March 27th. There was just a single full working day in between. What did I do on that day? Nothing - I'd booked the Friday and the entire following week off for a holiday. There was no way I could have known when to book time off avoiding the Safari release as there is no public release schedule. So the soonest I was able to personally verify whether it worked or not was today, April 3rd, a full week after the worldwide release of Safari 16.4. I could not be totally sure our software was going to work in Safari until then. Thankfully, it worked. It also worked in STP 166, not that it would really have helped to know.
All this difficulty - and it could largely have been averted if Apple could just label an issue with the version it's fixed in, like all other browser makers do. A tiny bit of transparency would be a transformation in our planning.
Previewing projects broken
The next issue we discovered was that previewing projects just showed a blank screen in Construct. Oops! That's also a major issue for us. I also filed an issue. Apple engineers once again helped investigate and did a good job. I will skip over the details as they are very complicated and not really relevant, but Service Workers are important for previewing projects in Construct, and we accidentally relied on a Chrome bug that meant our Service Worker was broken in Safari 16.4. So in this case, the problem was really Google's fault (please fix it, Google), and it was our responsibility to fix it. Oh dear - but these things happen.
Once again we had the terrible dilemma, but worse: the Safari release seemed close. This time it's definitely on us to fix it. How long did we have to act? This is an important question. We have our own release schedule for Construct, with beta releases for testing, and every couple of months a stable release that we roll out to everyone. If we knew the Safari release date, we could compare schedules. If there's time, we could relax - we could spend some time to investigate, and our fix could reach our customers on our normal release schedule before the Safari update. But if Safari was due for release the next day, we have an emergency on our hands: Construct is broken and it's our responsibility to fix it by tomorrow to prevent disruption for customers! That's the difference between a routine bug and an emergency.
Of course we had no idea that Safari wasn't due for release the next day. Some Apple employees tried to drop hints about a schedule they obviously weren't allowed to talk about, but couldn't manage anything much more specific than "soon". So we were forced to treat it as an emergency and act immediately. Worse, Service Workers are difficult to develop with, having a variety of complications that make them harder to code. So we had to go through the disruption of dropping our other work, doing difficult work on an emergency schedule, and then rolling it out to all customers immediately, which is risky - what if we got something wrong and broke something else?
All of that disruption - and it could well have been averted if Apple merely published a release schedule, as every other browser maker does. A tiny amount of transparency would have saved us a lot of trouble. Fortunately our roll-out went well and didn't break anything else, although we had further headaches about Safari seemingly caching the old Service Worker script in a different way to other browsers. I never figured out why that happened, but it risked making the situation much worse.
In the end Safari 16.4 rolled out about a month later. We could have relaxed. The emergency response was an unnecessary disruption and a waste of time.
Breaking all content published in Construct
Amidst all this - being unable to open projects, and doing an emergency response - I missed an even bigger problem. It turned out that in Safari 16.4, all web games published by Construct for the past few years were broken.
The problem is Construct uses OffscreenCanvas when available to run the game in a Web Worker for better performance characteristics. Safari 16.4 adds support for OffscreenCanvas, but only the "2d" context - WebGL remains unsupported. Construct requires WebGL for rendering. So it was seeing OffscreenCanvas apparently supported, creating a worker, creating OffscreenCanvas, then getting null for a WebGL context, at which point it fails and the user is left with a blank screen. This is in fact the biggest problem of all. Reams of existing web content is broken by this. We issued a second emergency patch to fix it (going through all that disruption again), but since that content is published by our customers all over the web over a period of years, updating it all is basically infeasible. It spans thousands of games on sites like itch.io, Newgrounds, Poki, and our own arcade; training materials used by businesses; education materials used by teachers; interactive kiosks in museums... and more. It was still a potential disaster.
This really caught us out. We detected OffscreenCanvas merely by seeing if it was defined (i.e. typeof OffscreenCanvas !== "undefined"). I never expected a browser to ship it with some contexts but not others - I assumed it would have parity with the standard <canvas> element. Why wouldn't it? The MDN documentation mentioned nothing about inconsistent availability of contexts. Chrome shipped OffscreenCanvas with all contexts in 2018. Firefox shipped OffscreenCanvas with all contexts in 2022. This was the first time a browser had shipped inconsistent support for contexts.
Apple pointed out that this is apparently allowed by the spec, and that it was faulty feature detection on our part. Looking at the relevant spec, I still can't say, as a web developer rather than a browser maker, that it's obvious that it's allowed. But let's assume it's the case - after all, browser makers spend more time looking at specs than most web developers.
Firstly my understanding of the purpose of specs was to preserve web compatibility - indeed the HTML Design Principles say Support Existing Content. For example when the new Array flatten method name was found to break websites, the spec was changed to rename it to flat so it didn't break things. That demonstrates how the spec reflects the reality of the web, rather than being a justification to break it. So my preferred solution here would be to update the spec to state that HTML canvas and OffscreenCanvas should support the same contexts. It avoids the web compatibility problem we faced (and possibly faced by others), and also seems more consistent anyway. Safari should then delay shipping OffscreenCanvas until it supported WebGL, and then all the affected web content keeps working.
Secondly even if the spec is deemed correct and the authors decide against changing it, doesn't Apple care about web compatibility? Why not delay OffscreenCanvas anyway, regardless of the spec, as a pragmatic decision to improve web compatibility? They could postpone it a release and spend the time adding support for WebGL. I am sure most experienced software developers have done something similar in their careers: a problem with a new feature is found at a late stage, so you turn off the feature, postpone its launch to the next scheduled release, and use the extra time to get it right. Safari is shipping OffscreenCanvas 4 years and 6 months after Chrome shipped full support for it, but in a way that breaks lots of content; surely the better option is shipping 4 years and 9 months after Chrome with a full implementation that preserves web compatibility? Why choose the option that rushes it and breaks things? I tried my best to persuade Apple to delay it, but I only got still-fairly-vague wording around it being likely to ship as it was.
In my view, this is Apple using a technicality to avoid having to change their schedule. We faced disaster, but all Apple need to do to avert it is delay one of many new features in Safari 16.4. In the end, they added a special browser quirk that detects our engine and disables OffscreenCanvas. This does avoid the compatibility disaster for us. But tough luck to anyone else who made the same mistake - the engine-specific hack for us won't save you.
A personal note
I would not normally mention this, but I feel it's important to say it to emphasise the extent of the problem. This has personally been an extremely stressful few weeks for me. At times it's left me feeling sick in my stomach with worry, including in my personal life outside of work. It's the uncertainty that gets you: perhaps we face disaster, or perhaps everything will be fine. I don't know which, and I don't know when we'll find out either. Every day for weeks on end it will either be a normal day, or perhaps disaster will strike, and there's no way to tell which it'll be - or even which of multiple potential issues would be the cause of the disaster. While it's not usually this bad, past Safari releases have caused similar problems too.
We're all only human and these things take their toll. I would point out that having built this company up over a decade, from laptops in bedrooms to a successful business with 250,000 monthly active users, I've seen my fair share of disasters, controversy, angry customers, sudden unexpected server failures while on holiday, and so on. Those can be tough, but you get on with it as a professional. However dealing with this Safari release has been one of my worst and most stressful professional experiences. And most frustrating of all, it seems so easy to avert - there are such simple changes Apple could make to alleviate most of the stress - and yet they have steadfastly avoided making any such changes for years.
I want to be absolutely clear that this is not criticism aimed at any specific Apple employees. No person at Apple has individually caused this stress - in fact as I've noted many of them have done a good job, and Apple has no shortage of smart and hard-working people. The problem is Apple's policies for managing these releases. Things like a complete lack of transparency cause great uncertainty, and it is these corporate policies that I feel are mainly responsibile for the stress that I and presumably other web developers end up going through. My only goal in mentioning this is to highlight the reality of what some of us face when dealing with Safari releases, and to try to effect change to the policies at Apple that I believe contribute to this.
Apple's track record
Safari releases aren't usually this bad, but similar things have happened in the past, and also caused plenty of stress and disruption. It demonstrates that nothing has changed - in fact if anything, it's getting worse. Here's a brief summary of past issues we've had with Safari.
- iOS 11.2.2 broke WebAssembly. This broke parts of Wikipedia, all existing published Construct content, and probably more. Much of the difficulty could have been averted by just turning off WebAssembly, so everything reverted to fallbacks. Instead they left it enabled but broken until the iOS 11.3 release months later. In the mean time Apple provided no meaningful help or support whatsoever. We only got support from a helpful Wikipedia engineer who shared a workaround they discovered.
- Safari 11.1 broke MessageChannels in a way that broke Construct, and left us maintaining workarounds for years.
- Safari 14 shipped a broken replaceChildren() method, which caused glitches in Construct.
- Safari 14 broke localStorage and then broke IndexedDB
- Even issues that get fixed still can cause stress and uncertainty. An issue with audio in Safari 15 risked breaking audio playback in all Construct content. It was fixed before the full release of Safari 15, but Apple never confirmed that ahead of time. I was left to check by myself again.
- A WebGL bug in Safari 15.0-15.4 caused a blank screen to render for some Construct content. It was eventually fixed in Safari 15.5, but we were never told that, and only found out by manually checking every Safari release.
- For years we've wanted a single open audio file format that plays in all browsers. WebM Opus is almost there: it's supported on all browsers including Safari on macOS but isn't supported only in Safari on iOS and iPadOS. That is the sole hold-out for us being able to rely on that format in all browsers, removing a significant complication for audio support on the web. However it's unclear if Apple are even working on it, with no clear indication about their intentions since I filed that issue around a year ago.
- Safari 16 has an issue that broke audio playback in Construct in some circumstances. There does not appear to have been any meaningful response from Apple about it, despite being filed about 6 months ago. We are still maintaining a workaround that has also caused other complications.
- Sometimes issues disappear in to the void. This issue was filed about a year ago and there has been no meaningful response from Apple. Compare that to how a similar Firefox issue filed around the same time was dealt with.
Doubtless there have been more cases - this is not remotely an exhaustive list. In many of those linked issues you can see similar things happening: confusion about what's going on, unanswered questions about schedules, refusals from Apple to divulge details that developers need.
We only have such serious problems with Apple and Safari. It is rare we have any such issues with any other browser maker, and when we do, there is full transparency about what they are doing about it, so we can plan accordingly.
How to fix it
The short answer for how to fix this is "copy other browser makers". No other browser maker causes us such serious problems, and a big part of that is their far superior development processes for web developers like us. To be clear, this involves:
- Transparency: tell developers which releases bug fixes will come out in, and publish a release schedule. This alone would clear up an enormous amount of uncertainty.
- Update Safari independently of the OS: Safari is the last browser in the industry left tied to its OS. This is an enormous barrier to updating Safari. Even a small but critical update apparently must wait for the next full system update. This doubtless has increased the time it takes even for severe problems to get resolved, often taking months instead of weeks or days. If the Safari team controlled their own release cycle and managed it sensibly, I am sure it would significantly limit the impact of many of these issues.
- More pre-release testing options: an equivalent to Chrome Canary and Firefox Nightly, updating daily and independently of the OS, would help quickly iterate on issues and verify they are fixed. Both that and STP should also be available on iOS and iPadOS, as currently the only way to test pre-release iOS Safari is with an entire beta system update, which is slow and inconvenient.
- Communication: mistakes happen, and if disruption is caused by something broken in Safari, Apple should tell everyone what is going on, what they are doing about it, when they expect a resolution, and what developers can do in the mean time. Often the main response is silence, which gives off a strong impression of just not caring. I suspect employees at Apple really do care, but from the outside, it often looks bad.
All other browser makers do all of this very well in my experience. Only Apple fails on these four points. Shipping new features for Safari is generally welcome, as Apple appear to have focused on for Safari 16.4, but it does nothing to address these points.
Conclusion
So in the end, everything ended up working. So what's the big deal? I feel like I'm gaslighting myself by asking the question. Look at what a struggle it was merely to try to keep things working with a routine Safari release. We might go through this again with the next release, and the next after that, and it seems likely sooner or later someone's going to miss something and we'll face disaster again... and when I think about that I can feel that sick feeling in my stomach coming back.
I want to love Safari. It has great technology in it. The latest release ships lots of cool features that I'd normally be excited about. They clearly have no shortage of talented employees who are capable of quickly getting to the bottom of a technical issue, and I want to be clear, I have no personal issue with anyone at Apple. The sad thing is in the end I dread Safari releases. Apple could clearly do far better. The release process is badly broken and has been for years. In many cases the difficulty could largely be resolved with only minor actions by Apple. But they steadfastly stick to the same process and have never shown any interest in changing it. And developers like me go through development hell dealing with the problems that result: unnecessary disruption to our normal work, wasting time finding out things Apple already know but won't tell us, and facing the stress and uncertainty of looming disasters with no way of knowing what will happen or when. In my opinion this amounts to a policy of negligence towards developers.
I desperately want Apple to change. I want Safari to be a great browser. I want to develop amazing content that works brilliantly in Safari. I want it to provide strong browser competition for a healthy web. Frankly I want Apple to change for my mental health too. If Apple won't change I feel the only options are to start telling people to switch to Chrome or Firefox, and ask regulators to force them to change. Despite recent regulatory activity, this release appears to show problems with Safari releases getting worse, not better; meanwhile Apple seem to be more interested in quashing regulation than actually solving these problems. If we end up with a Chromium monoculture, that will be bad for the web, and will bring its own problems. But if that happens, I have to say that as it stands, I won't miss having to deal with Safari releases one bit.