My advice would be to create an event driven behaviour for the camera object, so you control every aspect of its movement: acceleration, deceleration, time to pan to x/y, follow offsets etc. Then, if you change the target, the camera just moves like a physics object (as in it accelerates / decelerates, has a max pan speed etc) without any jerky effects. The lerp is ok to start with but I cannot think of an easy way to use it and do what you're asking here. There's also a camera plugin, IIRC, which is pretty excellent....