The problem is that you didn't reset your scal-variable back to 1 when you change from up to down scaling.
If scale is 1.8 after upscaling und you try to scale down. The first time you multiply with 1.6, then 1.4 and so on.
It needs a few click that scale is 1 or below.
One solution could be that you use two variables upScale=1.1 , downScale=0.9 for up and down scaling.