I figured it out... only took me 6 hours to wrap my head around it...
the solution was way more simple than I thought...
the formula is just: MapCenter + (Object.x/y - Player.x/y)
the MapCenter is in essence, the player.. and as the player moves, the objects move away from the player (even though the player is the one that is moving). You are basically just doing the opposite.
Hope this helps someone!