Is there a method to access engine internals from JS blocks?

Post reply
1 favourites
From the Asset Store
Sheeky's Racing Sounds
$5.99 USD
75 vehicle sound effects, from real looping car engines to jet aircraft and sci-fi engines.
Mark Sheeky's avatar
Mark Sheeky

  • Ok thanks a lot for the clear answer : So it looks like preventing us to access undocumented Runtime features is in your plans. "the sooner, the better"

    To me this is very bad news, it would be a loss for everyone. It will put more expectations on you to fix everybody issues and to integrate the features every single advanced/complex/specific production needs and it makes the engine 10x less powerful for advanced users than it is today.

    Anyone who tried to do anything with the Editor API knows how impossible to do anything relevant with it because how limited it is. This is exactly the same with the documented part of the Runtime API. My experience is that asking Scirra to expose even the tiniest obvious obfuscated method in the Editor API requires an unbelievable amount of time and energy for a very unlikely chance to happen.

    One of the biggest drawback of construct is the lack of being able to customize our Editor experience (extra window/tools at edit-time). And instead of enhancing that aspect, the Runtime is about to fall into the same situation. Making the overall level of control over our own work 10x worse than it is today.

    i don't see how any C3 user could benefit from this, and i don't see how any C3 user wouldn't be impacted negatively by it in some ways

    Even people who don't use addons benefits from :

    • lowering the number of feature the community expect Scirra to develop
    • successful games and active community result in more chance for the engine to grow over the years (+ more tutorial, resources, word of mouth marketing)
    • a bunch of the most interesting features ever released officialy to c3 mainly exists because 3rd party devs like Skymen or RexRainbow made them first (Tweens, JSON, Touch fix/extra features, 3D Shape for the older ones, Dynamic Layers, Follower, CSV to name some of the most recent)
    • knowing community-made solutions to the most annoying limitations exists and are available to download (for free most of the time) at any time. A C3 user might not need a tweak/custom feature now, but what will happen if they're facing a wall at some point in their production ? they will be screwed, no addon dev will be able to help them anymore. I'm not sure anyone like to know any issue they might face would be unsolvable.

    Overall, it really hurts my trust/faith in Construct, and I'll think more carefully about my choice of game engine for upcoming projects. I know by experience that I need to use custom features very often to develop the game i want to make, especially as many vanilla C3 features are super opinionated or because achieving modularity / making logic reusable across project is otherwise impossible.

    But anyway, I appreciate your transparency !

  • I think you need to understand our stance on this Ashley while you are working on software architecture and striving to have something that will still work in 10 years, we are not.

    We are making games, and ask any game developer out there, from AAA to indie and they will all tell you that designing games like a software is itself a recipe for disaster.

    You are making a game engine, so I get that you need to have it work as long as you can, but at the end of the day you are making it for a crowd that actively depends on being able to break and hack stuff. You might think this decision is sensible for you, but it will be very hurtful for the bottomline of your userbase.

    If we reuse your car manufacturer analogy, that's like specialising in making cars used in rallies and complaining that people wanna mess with all the car parts. You can lock that down, but then the rally drivers will just look for another manufacturer. Sure you will be able to keep your parts secure but people actually competing will stop buying them.

  • But, isn't it a similar situation to any support request that Scirra receives: if a project has correctly-made third party addons and sends it to Scirra to fix, Scirra wouldn't touch these until the addons are removed?

    No, there is a distinction here: it is not Scirra's responsibility to provide support for third-party addons. If there is a bug in a third-party addon, then the third-party developer needs to deal with the report and fix the problem. However if the addon developer themselves identifies a bug in Construct's supported APIs, they can file an issue with us and we will fix it. (Note if the bug is in an unsupported API, we are not obligated to fix it, as per our warning about using unsupported features.)

    Overall, it really hurts my trust/faith in Construct, and I'll think more carefully about my choice of game engine for upcoming projects.

    I would point out nothing has changed. That warning has been there for a long time and it has always meant this. If this was not clear to you, I am not sure how to make it any clearer. Do you think the wording should be different? If so, what should it say?

    You can of course stick to the documented APIs and do anything you like, and we promise to support that indefinitely. I am pretty sure there is a lot more flexibility in using officially supported features than most people appreciate. As I said many engines make it difficult or impossible to hack the runtime code, especially at runtime, so this entire pattern of hacking the engine mainly via the weakness of JavaScript's encapsulation is not typically used in the industry. Everyone uses different approaches, like creative workarounds, or other implementation techniques. You can do all of that with Construct too, and have no risks or problems. As I said before, I think people are too quick to reach for "hack the engine" as their solution, rather than more thoughtful approaches that would be more reliable in the long run.

    I think you need to understand our stance on this Ashley while you are working on software architecture and striving to have something that will still work in 10 years, we are not.

    I realize game development may be different, but it doesn't change the fact we are in a different position having to maintain software for many years, and that imposes different and more stringent engineering requirements.

  • I would point out nothing has changed. That warning has been there for a long time and it has always meant this. If this was not clear to you, I am not sure how to make it any clearer. Do you think the wording should be different? If so, what should it say?

    This whole thread is about it, so it might be worth rereading everyone's comments ? There is a major shift with your recent statements.

    It went from "undocumented features could change or be removed at any time" which is perfectly fine and understandable to "we plan to make sure nobody could ever access and use an undocumented feature".

    There is a full world between an engine that let you customize/extend/create new features at your own risk (like any other game engine let you do) and an engine that don't let you the control over such things.

    Everything vanilla is using undocumented feature, so we can't do anything relevant using documented feature. There were not thought to be used, not battletested, and there is no cooperation with 3rd party devs to make things better.

  • It went from "undocumented features could change or be removed at any time" which is perfectly fine and understandable to "we plan to make sure nobody could ever access and use an undocumented feature".

    "We plan to make sure nobody could ever access and use an undocumented feature" is a case of "undocumented features could change or be removed at any time". It is precisely what the warning was warning you of.

    It is not removing engine customization completely. Engine customization remains, but with the provided APIs.

    As I said before though, if we do this, I promise we will do our best to approach it in a responsible fashion, and roll it out gradually and in phases and deal with compatibility issues where feasible.

  • To help clarify the point, I've updated the wording in the Addon SDK documentation to read:

    To be clear, it has always meant this - I'm just adding a clarification to emphasize the warning includes that possibility.

  • Up until now using undocumented feature meant we need to update addons if you make changes to the engine. I.e. you add RTL support to the text render and the animate text addon breaks, to fix that the addon dev needs to bring the addon back in line with the changes of the text renderer. (btw I don't think you got many, if any, people reporting that to you, but instead it was reported to Skymen).

    As you will always have some form of text rendering in the engine it's save to assume that while it might change, maybe even drastically, it is near impossible to be removed completely. This meant using undocumented features was a larger maintenance burden on the addon dev, with a small chance of being impossible to fix.

    Depending on the undocumented feature we could make an educated guess how risky it is to use. But now what you are saying means that you are purposefully intending to make all of this impossible permanently and you want to do it as soon as possible. Now we cannot weight the risk, because now there is Damocles sword over every undocumented feature.

    It's not clear to me what benefits this will bring, it sounds like a waste of time.

    I also think you underestimate how much value 3rd party addons bring to your product and it's a bit disheartening to hear that instead of trying to give more power to those developers you want to take power away. I also think you might not be aware how many developers and games moved away from Construct after hitting some limitation and then made highly successful games on other engines, these could have been hit games made with Construct.

    ... I could go on but I guess all these points have already been brought up by others

  • It is interesting to see other very successful engine companies philosophy about access to runtime internals. Obviously, some are many orders of magnitudes larger than Scirra, so going as far as they do, could be prohibitive in terms of support. Their general direction is interesting and informative and you can see they are focused on their users and game developers and shipping games.

    From UE:

    Accessing Unreal Engine source code on GitHub

    Unreal Engine includes full access to the complete C++ source code, so you can study, customize, extend, and debug the entire Unreal Engine, and complete your project without obstruction.

    Ref: unrealengine.com/en-US/ue-on-github

    The main difference I see from the discussions so far is their focus on the user and their game project/product. Ashley - can you, also put yourself in the position of C3 developers trying to ship ambitious games / products (some of your customers, certainly not all) and see how the above UE attitude / philosophy would be appreciated? Or do you consider this sub group of customers not very important in decisions about C3 roadmap (this is not a facetious question, I know Scirra has other focuses towards education, animation, casual hobby devs.)

  • Ashley

    Question 1:

    In light of what you have clarified,

    Can we find a solution? Do you want to support 3rd party development?

    Would it be possible to set up a way for folks to more easily get access to an expanded api? It's very difficult to be actively developing a game if you have to stop to go drum up popular support, pitch a suggestion that may or may not be responded to, then still have no idea if it will be made, as there is no road map!

    That is exactly a recipe for dev-hell.

    Question 2:

    If an official behavior uses an api call, why is it so hard to make those public and document them? Again, you needn't promise eternal stability, just a way to get the job done, and a promise for an alternative if the api changes. Which is why I mentioned perhaps a "experimental /dev" api in a previous post.

    Issue:

    You keep saying that you are pretty sure most things are possible without engine hacks, but you neither address the specific problems or describe the alternatives. You also have no projects or templates showcasing these scalable and well designed alternatives. As you say, using a private api is bad for a number of reasons, but If the alternative involves poor programing patterns, it isn't an alternative.

    In the specific case of creating a platformer like behavior...

    You told me you don't want anyone "duplicating" the platformer behavior (or any others for that matter). That statement alone indicates a certain level of arrogance... which leads to question 3.

    Question 3:

    Do actually believe that behavior alone addresses all needs within the genre in a scalable, and sensible way? Not only that behavior, but any of them?

    It's an objective fact that you can't possibly anticipate every need any game would ever have, which is why you need a robust way for users to extend construct's features. That is what this whole discussion is about.

    We want to be able to make OUR games, not yours!

    . C3 is supposed to be a gameenegine, not a prebuilt car package where all we do is change paint color.

    I don't want to duplicate your platform code. I want to make my own, because mine is better (for me)! The official one is fine, for those who can use it, and I use it all the time for prototypes, but never finished products. That isn't feature duplication, it is a wholly different behavior that is functionally different at all levels.

  • As you will always have some form of text rendering in the engine it's save to assume that while it might change, maybe even drastically, it is near impossible to be removed completely.

    No, this is an incorrect assumption, and text rendering is actually a good example of why. I have been monitoring the canvas formatted text proposal, which lets the browser do the text layout for formatted text when rendering to a canvas. Currently we have had to implement our own custom text layout engine in JavaScript, which is both extremely difficult, has lots of difficult bugs and edge cases, and is reimplementing what the browser already does. Ideally we could switch the implementation to a browser-based one with the canvas formatted text proposal and delete our entire custom implementation. It could then have more features, better performance, better consistency, broader internationalization support, and so on.

    However if we did that, all third-party addons which rely on hacking the internal text layout engine will be broken. All the internal text layout code would be deleted, so there is no more scope to modify it. It may then be impossible to achieve the same thing with the browser built-in feature, as from our point of view, we only need the new text engine to be compatible with the old official features. It is possible that third-party addon compatibility is then such a severe backwards compatibility problem that we have extreme difficulty implementing the new, better engine; or even we have to cancel the project to upgrade it. For us, that's game over: now we can't upgrade the engine, or it is far more difficult to do so, and so we end up stuck with inferior systems in the long term.

    This type of thing can happen with literally any part of the engine. The undocumented features warning is there because we want the flexibility to be able to upgrade the engine in the long term. So really the problem here is developers assuming it is fine, when it is not. It is meant to be a serious warning that is taken literally. If you ignore the warning and assume it will be fine, and then everything breaks, we reserve the right to say "sorry, we warned you about this" and proceed anyway, to avoid the outcome that we are unable to continue making improvements long-term.

    Do you want to support 3rd party development?

    Obviously yes - if we didn't, there would be no addon SDK at all. Exposing a controlled public API is the industry standard, and as I keep saying, the only reason people can bypass that in Construct is because of weak encapsulation features in the JavaScript language.

    Would it be possible to set up a way for folks to more easily get access to an expanded api?

    I think perhaps a good way to do this would be to put the addon SDK behind the scripting interfaces used when doing JavaScript coding in Construct. That is already a more comprehensive, documented API, which does have encapsulation, and it avoids us needing to implement and document APIs twice (once for the SDK, once for scripting) - by adding an API there it then becomes available to both addon SDK developers and JavaScript coding simultaneously.

    Again, you needn't promise eternal stability, just a way to get the job done

    Yes, we do need to promise indefinite stability. Otherwise if we change it say 5 years down the line, it breaks a popular third-party addon, thousands of projects are broken, and the original addon developer has left the community, then we have a huge compatibility mess which we are left to deal with. This has happened. Again, this is a case of people assuming things will be fine, but it is not. The only way to avoid this is a public, documented API that we promise to support indefinitely.

    Do actually believe that behavior alone addresses all needs within the genre in a scalable, and sensible way?

    Maybe the behaviors don't do everything. They are designed to be flexible. But beyond that, you can use custom logic in event sheets, or JavaScript coding within a project, to implement things like custom movements. You don't have to jump to hacking the engine internals. There are usually several ways to get things done.

  • Again that was the first point of my first message in this thread : we love to know that you can have the flexibility to change any part of the engine to make it better, without needing to support indefinitely undocumented methods, so we should use them with caution etc. Skymen often say he would love if any of his addons could be deprecated in favor of a vanilla solution.

    Totally preventing us from accessing those methods removes us the control to develop our own features and fix the issues/limitations we're facing, it just makes the engine worse. Recently, I find more value in the features built by advanced C3 users for advanced C3 users facing actual issues, and above all the features i built myself for my own private use than in the Vanilla updates. It's like if I was able to develop the engine very fast toward the specific direction i want it to be for my needs.

    Knowing that you plan to remove all this, is actually like as it already happened. It's not a weighted risk on each thing individually, it's a certitude that our experience with C3 will downgrade as whole, that all effort to achieve our game vision are vain and that every tiny thing we need will need to get through the feature request process (rewriting the same feature request every year, praying Scirra push it while it's almost certain it won't - for legitimate OR arbitrary reasons) while before it would just take 5 minutes for us to do it.

    I think perhaps a good way to do this would be to put the addon SDK behind the scripting interfaces used when doing JavaScript coding in Construct.

    The reason we use undocumented features is precisely because builtin features doesn't allow us to do what we want. Scripting API are often even more limited than Eventsheet features, and i know by experience requesting an obvious Scripting API corresponding to existing eventsheet feature can take more than a year to be pushed and even never happen.

    (Also we can already easily use Scripting API in Addons, so it's not as if we were losing something while winning something else... We're just losing possibilities. The opposite is also true, today we can use the full power of the Addon SDK in js scripting easily, soon, we'll only have access to the <1% exposed stuff with all its obvious holes)

    But anyway we already explain this, and as often, you don't seem to acknowledge the point we're raising, denying our experience with the engine and the hard work we're putting into making our games, as if vanilla features were fitting any gamedev need that could ever exist. This is exactly why we don't want to depend on you to fix every single issue we might face.

    This is a Apple vs Open Web situation here, I'm sure you can relate. Yes Apple is a huge company, but that is precisely why we can understand the logic behind their shady moves. Shareholders, dillution of responsibility ect. Here it just feel like a unipersonal decision at the expense of all paying subscribers and C3 users and shows again a lack of empathy and shared goal with the C3 community.

    On top of every drawbacks already mentionned, obfuscating the runtime would make the 3rd party porting solutions to console orders of magnitude harder (if not impossible) to develop. The best solutions are litteraly alternative runtime like Chowdren : construct.net/en/forum/construct-2/general-discussion-17/chowdren-fast-construct-134395

    It would be 100x harder or impossible to develop an alternative runtime without being able to take a look at the vanilla runtime. So it won't be possible to create Playstation or Nintendo Switch porting solutions

    So at the very least 3 of the main drawbacks of C3 are about to become worse

    - Porting to console is incredibly tedious

    - premade blackbox tools might create unsolvable limitations for our games

    - Not enough 3rd party devs empowering and providing value to the community

  • Try Construct 3

    Develop games in your browser. Powerful, performant & highly capable.

    Try Now Construct 3 users don't see these ads

  • If there is a change that clearly improves c3, i.e. an improved text layouter, the chance of breaking something like the animate text addon might be painful but it's a clear benefit that makes this ok. In the worst case people late in development can stick to the previous stable (which they probably are already doing, as updating late in development is risky and makes porting more difficult).

    I am personally in favor of breaking compat more often than c3 is doing, at least if there is a clear benefit and a transition period.

    Bringing the scripting interface and addonSDK more in line sounds like a good idea, I am not sure I fully understand what you are implying with the wording though.

  • Don't get hung up on whether or not we do it deliberately: the point is everything could break unintentionally, by accident, due to cleaning up our code, due to refactoring to reorganize things, due to performance optimizations, due to upgrading or replacing internal components, due to adding requested features, or for any other conceivable reason. Everything could break anyway. This is the risk everyone is running by using undocumented features, and this is why we have that warning. If that happens we reserve the right to say "sorry, you shouldn't have done that", otherwise we basically lose the ability to upgrade the engine.

    It sounds like stuff is already occasionally breaking and addon developers have to update their code. What happens when these addon developers decide to move on and stop maintaining their addons? We start facing backwards compatibility disasters and we are left to deal with the consequences.

    How do we avoid this risk? The industry-standard solution of using encapsulation to ensure people only use public, supported APIs and can't mess with the internals. It's nothing to do with Apple or the open web. The situation is JavaScript has weak encapsulation, so we've ended up in an especially bad situation for compatibility that other tools don't face, and we want to try to avoid the disasters that are inevitable if we carry on down this path.

  • Obfuscating the runtime is not an accidental or unintentional stuff to do. And yes it does matter that it could be avoided easily by not deliberately removing us our guarantee to be able to fix and create features ourself. It would be working against your users who pay you a subscription to enhance the engine, not to waste resources to lock down their own creations. Again, the whole point of this thread.

    You seem to put real effort to not understand or ignore our concerns. All this feels like a big waste of time. Betting on construct long term or even just for making professional projects with it feels like a risky gamble.

    Our experience reporting feedback to you is the same experience you describe reporting feedback to Apple / relying on them for your business.

    The analogy with Apple VS Open Web works well on different aspects. Example : Apple prevent other browser to exist on their platform that would allow healthy alternatives to exist for their 3rd party dev and users. It sounds familiar to me. No roadmap, half-baked features, everything locked on purpose, no action taken on the most requested stuff and deliberate moves to prevent 3rd party dev to create solutions for those themselves, everybody dependant on someone who obviously don't share their interest. Even the bad faith reason given by apple is similar: "security"

  • It sounds like stuff is already occasionally breaking and addon developers have to update their code. What happens when these addon developers decide to move on and stop maintaining their addons? We start facing backwards compatibility disasters and we are left to deal with the consequences.

    The community has created this Collective, besides funding new addons, it also mitigates that risk. So addons created under the collective can be maintained by other community members, including some monetary compensation.

    I also don't understand why you are left to deal with the consequences? We the community are. All you'd do is say: "your fault, told you so".

Post reply
Return to board index
Jump to:
Active Users
There are 1 visitors browsing this topic (0 users and 1 guests)