This post may help.
Sampling pixels is a relatively slow operation that adds up when doing it multiple times. In javascript you can copy a whole pixel area to an array. The result is very fast sampling of any of those values. The expression in the canvas object is .AsJSON and it returns a json string that then can be loaded in the Array object.
I added it as an experiment so it isn't very user friendly. For one the array size is [1, 1, width*height*4] and to access a particular pixel you have to do it with [0, 0, (width*y+x)*4+i] where i is 0 for red, 1 for green, 2 for blue, and 3 for alpha. Also all the pixel data is in the range of 0-255.
With regard to performance, one rgbaAt() is faster than one .AsJSON, but .ASJSON becomes faster when you want to sample many pixels. Also the bigger the canvas the slower it is, so when you only need to check a smaller area you should paste the canvas to a smaller one and use .ASJSON on that.
Also alternatively Another idea is to just copy the terrain to an array to read from and clear it manually when you clear parts of the canvas. Tedious, yes but gives good performance.