Stream level chunks to and from disk, Minecraft/Terraria style?

Given a tile-based procedurally generated world, in the vein of Minecraft or Terraria, is there a good way to save and load map data chunks to disk as needed?

That is if a player travels East, new chunks are procedurally created ahead of them, and old chunks far behind them are saved to disk and unloaded from memory. Then when the player turns back and travels West, the chunks ahead of them to the West are reloaded from disk as needed.

I know that the browser-based nature of C3 is likely to make this a bit tricky, because browsers don't have free reign to write files for security reasons.

Ideally I'd be looking for desktop style behavior, where the game stores map data essentially the same way Minecraft does.

e.g. Loosely speaking, Minecraft streams map chunck data to and from a local folder, which differs slightly by platform.

• Win: %appdata%\.minecraft
• Mac: ~/Library/Application Support/minecraft
• Linux: /home/USERNAME/.minecraft/

Another benefit of this desktop-style behavior, is that world data stored in the folder isn't vulnerable to erasure by clearing local browser data.

I'm not sure what the best approach would be, but my initial impression is that this would require exporting for NW.js in order to take advantage of the more desktop-like context NW.js operates in.

Would an NW.js exported application allow saving and loading of map data, without prompting the user for a save location for each chunk file? I figure that prompting multiple chunk file saves every few blocks would slightly encroach on the spirit of exploration. :)

I'm not familiar with the nuances of C3, but my experience with the prior incarnations of Construct so far, is that just about anything is possible, with enough insight.

If there's a better approach, or if it's genuinely not doable, or simply not something that C3 is a good fit for, any thoughts or suggestions are welcome. :)

I think you’d have to test the file reading/writing first to see if that works. I agree it would be a deal breaker if you had to get user permission to do it.

In general I think you’d have to keep a list in memory of all the chunks, then you could reference that to see if you should load a chunk or just use the generator.

Chunks could be identified by their position x&”,”&y. Then that list could just be a dictionary.

That said, if the file saving/loading doesn’t work well you probably could go pretty far by just storing all the chunks in memory. Instead of storing every tile in a chunk you could store just the changed ones, and/or compress/decompress it when saving/loading. You may be able to utilize the binary object if you want a tighter memory usage.

I’d imagine even in Minecraft the file streaming may have been added later.

It system itself can be done. The real question is, do you need to?

If you want to make a desktop game that takes advantage of local files for storage, do it properly and make a desktop game with nw.js. If you're making a web game, use localstorage or other online based database.

Either way, is your game actually running into memory limitations? In a 2d engine like Construct, you're more likely to run into texture memory bottlenecks before system memory, and chunk based loading isn't going to help you very much with that.

Also note that if you have a proper procedural generation design, you don't save entire chunks of data to storage, you only save changes and differences. Those are very small and normally can be kept in memory all the time, only writing to storage when saving. And then as mentioned above you'll want to write it to the proper storage media for your platform.

If you've got a river to cross, try a boat first, no need to learn how to build a railroad bridge and a train first just because someone else did. Systems like chunk based dynamic loading are often designed to solve specific problems, problems that you may or may not have in the first place. Basically I'd say the better approach is to build your game first, and then apply solutions to solve the problems you're actually having. For example, if you had your procedural generation system in place already, you'll gain a much better understanding of what exactly you'll need to save and load, and how to best go about it, if you need to at all.

Thanks for the replies, R0J0hound and Oosyrag, :)

What you both mentioned about a 2D world's data growing in memory at a much slower rate than a 3D volumetric one, and possibly fitting entirely in RAM for much longer is a really good point.

And your caution about getting stuck in the pre-optimization weeds before it's actually solving a concrete problem is good advice, Oosyrag.

Mainly, I just wanted to check with people more familiar with C3, that streaming to/from file wasn't ruled out by something I wasn't aware of. I figure at some point, an infinite world is going to have to go to disk, and even if it's a problem to solve much later on, I'm just at the engine scouting phase. So it's handy to have an idea of how much a given engine will be working for or against certain aspects of the project, before committing to one.

It sounds like NW.js would be the way to go for file read/write stuff, which has been my experience with C2. I made a C2 color theme editor quite a while back, and it relied on the NW.js file facilities, though I don't recall if I could invoke a save/load without a prompt using the built-in NW.js C2 object. I could always code a plugin though, if the functionality is available somewhere in NW.js features.

Anyway, thanks again for the info, R0J0hound and Oosyrag. :)