I would check the 4 direct neighbors (up, down, left and right) and orderly put their values in a string like "1001", where 1 is land and 0 water, as an example.
Then you could have an array describing each possible combination along with a corresponding tile and its status (flipped/mirrored and/or rotated, whichever works best for you).
The more specific your tile set is, the more specific the source array should be. mirroring, flipping and rotating are mostly generic solutions for more minimalistic art or sample projects.