Whew! You say that your layout is smaller than your window, but thankfully in your project that is not the case.
Keep in mind that WindowWidth and WindowHeight are the dimensions of your view window, and (scrollx,scrolly) is the center of the view window over the layout.
So to keep from responding to touches outside the view window add the following conditions to each of the events that handles touch:
touch.X < scrollx-(WindowWidth/2)
touch.X > scrollx+(WindowWidth/2)
touch.Y < scrolly-(WindowHeight/2)
touch.Y > scrolly+(WindowHeight/2)
Here is the modified capx:
Pan_Fix.capx