Dont use lerp, set the color instantly. The lerp takes time.
Or
set the timer faster. Set it to 0.5 seconds, if you dont wanna wait 1 second.
But ...
Then the lerp is then ... lerp(dude.l,color.l, dude.Timer.CurrentTime("color") / (1/0.5)) <-- 0.5 stands for the timer time
Or
If you want it to only be instantly on the first touch.
Dont start the timer when previousTouched = -9
Set the values instantly when previousTouched = -9
Still, is using replace color, as i did in the first attempt, not a lot smoother ?