I was able to make it work somehow.
It still needs tuning on the point system and it can't swap the off-color even if the sprite it swapped with is a match.
But it's a start, it may help someone to either make one or improve, so here's the c3p of what I was able to do so far.
drive.google.com/file/d/1bncRnHUif3oLiFFYTbqAZ5wo0_-Bgvzj/view
Thanks Alextro, even though I didn't use your examples because I had already started with this one following the tutorial, I found yours really pretty and interesting, I might give it a go in the future!