How about additionally checking the distance between the previous and the next checkpoint?
distance(player.X,player.Y,prevCheckpoint.X,prevCheckpoint.Y) / distance(prevCheckpoint.X,prevCheckpoint.Y,nextCheckpoint.X,nextCheckpoint.Y)
This gives a value between 0 and 1 as how far the player is between the checkpoints, e.g. 0.78 (= 78%)
You add the number you already have that gives +1 per checkpoint and add to this the number between the checkpoints. The player with the higher number is further along the track. Like player X has checkpoint 2 and is 55% along the track = 2.55 and player Y has checkpoint 2 and is 54% along the track = 2.54.
EDIT: You probably want quite a bunch of checkpoints for this to work properly, especially when there are many curves involved. Here is a checkpoint map from mario kart for reference.
EDIT2: You probably need to check for shortest distance rather than distance to the origin point especially at the finish line where it really matters to be exact... Not 100% sure how that works but I guess the player that passes it first wins should do the trick aswell.