Have an idea that possibly could make it a lot faster. It's a pretty experimental approach to how we generally think of 3D. It would involve restructuring the point data, and have it indexed. I got the idea by looking at the way you get the point positions and the rgb value for those positions. Basically it would work like this...
It's pretty much creating a search algorithm, only getting relevant data from the point cloud, limited to number number of "pixels" in the tilemap.
Every cell of the tilemap just searches for a the color value it want's to show, and maybe we could get that directly from the point cloud if it was indexed. Translations and rotations would be done at Bounding box level for the object, so we don't need to reposition points individually.
For a quick indexing test. You mentioned the data i pretty normalzed already, so we create 50 steps between -1 and 1. If we take the point cloud data, populate a 50x50x50 array with the average color of every point in each "cell". so if there is any point with the value 0,0,0 we would place the rgb value of that point in the dead center of the array. if a cell has no value, it means transparent, if has a value, it would render to the tilemap.
If we can have some kind of bounding box representing the array data that we could rotate & transform A simple search from the each individual cell of the tile map looking at that bounding box, could get the indexed rgb rgb value of the cell it's looking at. For example. tilemap 25,25 is has the array cell # 26,25,12 (we can disregard all cells behind it as it is opaque)in it's view we would know what color to display for that individual pixel. If a tile map pixel is looking has multiple cells in it's view, we could get the average color of these, that way we could have something similar to anti-aliasing as well.
LOL. hopefully you can understand what I'm rambling about. I'm going to do some tests indexing the point cloud data, so we can minimize the data we need to get. Theoretically you could have massive amounts of points, if you can get them quick enough from files on disk with an index, you probably wouldn't even need to load them in to memory.
As long as each pixel know's what he's looking at, he could jump directly to the index point and get the color information from the correct "cell". the tile map cell is basically jumping straight in to "Column, 6, row 9, z 8, and getting the correct rgb value.