Hm, I feel so silly now that I look at it with rested brains. I should just have taken the difference between ScrollX and MouseX then substract it from the ScrollX!
ScrollX-(ScrollX-MouseX)*0.03 (horizontal scrolling speed)
ScrollY-(ScrollY-MouseY)*0.04 (vertical scrolling speed)
Since I can't seem to edit my initial post, I'll just upload the updated cap here.
Edit: Oh, so that is what lerp does!