I don't think we ever published a tutorial on how to use the plugin. It's quite similar to the JavaScript DataView class but I couldn't find any good tutorials on that either.
When we first released it I did start writing a blog post about it, but got sidelined by other work. When I returned to it I wasn't completely happy with what I'd written. Too much fluff, not enough stuff. No one was asking for it at the time so I put it to the side and left it.
I'll post the incomplete draft here in case it helps anyone, if people are interested then I might take another look at it. Unfortunately it's not finished, or particularly edited. It's also missing most of the practical examples on how to use the plugin, but it does explain what it is and how it works.
A fresh Byte - introduction to bits, bytes and binary
A byte is a number, composed of 8 "bits".
Right... so what is a "bit"?
A bit is a single binary digit. A binary digit can be either 1 or 0 ( true or false ).
If it helps you can imagine a "bit" as a coin. Placed it on a table. The coin can be "heads" or "tails" depending on which side is up. Now place a second coin on the table. The first coin has 2 possible orientations, if you consider that the second coin also has 2 possible orientations as well you can see that in combination you will have 4 possible orientations. If your struggling it might help if you actually get some coins out and count the number of combinations! Add a third coin, and it will become 8 possible orientations. This gives us a neat pattern, for each coin we add we double the number of orientations we can have. So if we were to use a calculator we can calculate how many possible combinations you can get using the sum "2 ^ n" ("n" being the number of coins). Now our "byte" is made up of 8 coins ( bits ) so there exists 2x2x2x2x2x2x2x2 combinations. That 256 combinations, or values! If you have 2 bytes ( or 16 coins ) as individual bytes they can represent 256 values, or 512 values together. But if you make them into a single pile of coins they can represent 2 ^ 16 values ( 65,536 ). That is WAAAY more!
Storing data as bytes
Now while messing around with coins is all well and good, I haven't said why we need to know why we need to know about bytes and bits. For various reasons ( which I won't go into ) all computers are binary. This means at the most basic level modern computers only really deal with bits and bytes. So everything you deal with; numbers, text, images and sound have to be stored as as long lists of ones and zero's.
To convert a value to binary we "encode" it in a suitable format. Clever people have come up with ways to store different types of data in a binary form, and we are using these algorithms to encode the data.
Whole numbers ( "integers" or "ints" ) are the easiest, we can convert a decimal value into a binary value easily. This value is then chopped up into 8 bit chunks that can be put into bytes. That doesn't tell you if the number is positive or negative, so there's a second format called "signed integers" where one bit indicates if the number is positive or negative. The standard sizes for integers are:
8 bit unsigned integer (0 to 255)
16 bit unsigned integer (0 to 65,535)
32 bit unsigned integer (0 to 4,294,967,295)
8 bit signed integer (-128 to 127)
16 bit signed integer (-32,768 to 32767)
32 bit signed integer (-2147483648 to 2147483647)
Fractional numbers are more complicated. The format we use is called "floating point numbers" or "floats". As frational numbers have an effectively infinite number of combinations we instead hold an estimate of the value. The numbers of bytes that you use for the number determines how accurate that estimate is. When adding together lots of floats you get a small cumulative error, due to the fact the numbers are estimates. The plugin supports 2 sizes of float, and there is no unsigned variant.
32 bit float
64 bit float
Text encoding was originally quite simple when it was first conceived ( back when telegrams were still a thing ), but has become much more complicated over the years. Typically each character is assigned a number which is used to map between letter and number ( like 97 for a lowercase "a" ). Then we can store a character as a number. It becomes more complicated if you include other languages and emoji ( which the old methods did not ). You need a lot more numbers to map all those characters ( 137,929 are currently supported by the unicode standard ). To hold a number that could be any character would require at least 3 bytes. Given that standard English text can fit in less than a byte per character thats a lot of wasted memory. To get around this modern encodings break up that value into multiple chunks, then each chunk includes a little bit of extra information about how many blocks the letter needs as well as part of the characters ID. This way an encoded character only uses the number of bytes it needs. The BinaryData plugin supports the UTF-8 character encoding, which can be between 1 and 4 bytes per character. However, there is a downside to these clever encoding techniques. Without going over every letter in your piece of text you don't know how many letters your bytes hold, or how many bytes you need to hold a piece of text. This also makes it hard to pick an individual letter from a random place out of a chunk of bytes.
For more complex types of data, like images and sound, they are constructed out of the types above. A basic form of an image is to have 3 numbers per pixel (red, green, blue), with each number stored as an 8 bit ( or sometimes 16 ) unsigned integer. Audio is even more complicated, it has multiple sets of data for each speaker ( channels ). Each channel stores measurements of air pressure at a given frequency as a low precision floating point number. A typical frequency is 44,000 times per second so 1 second of stereo audio would be 88,000 bytes ( that A LOT of data ) and a 4 minute song is 22MB. You might be wondering why the songs on your PC aren't 22MB, and that's because audio, images and videos are generally compressed before they are stored. Compression is cool, and complicated... So we won't cover that. But I will say that media nearly always has to be uncompressed to be used.
What is the BinaryData plugin
The BinaryData plugin is quite simple really, it's a container for bytes. Through it's actions and expressions you can read and modify those bytes. Most of the time you won't be using the plugin by itself, but as a container for moving data from one place to another. At the moment we have expanded the NWjs, localstorage, websocket and AJAX plugins to allow them to interface with BinaryData.
The AJAX plugin in particular is very useful, as it allows you to fetch files straight into the BinaryData plugin. By using another plugin like the drawing canvas which exposes data as base64 or blob urls, you can "fetch" that data via the AJAX plugin in BinaryData. In the case of blob URLs this is usually very efficient, with very little overhead. Encoding and decoding base64 generally involves multiple copies of your data, so has more overhead.
Once you have that data you can pass it to another plugin, or you can modify it using the inbuilt methods. When reading and writing to the BinaryData plugin the tricky part is that you have to know what type of data is where. As the values are encoded you can read them out as whatever you like, but it might not make much sense. Reading a number as a piece of text will give you some weird letters for instance. You can even put copy the contents of one BinaryData plugin into another, which allows you to compose more complex data formats. It's possible to read complex file types or even create your own format for your game using these tools (Interchange File Format is a good starting place for this)