Again from that first pic, the red arrow is the direction the ship is moving. The ship has rotated to face upwards (but is still travelling to the right, along the red line). When the bullet fires it comes out of the ship and travels along the green line because it does not gain the same directional momentum as the ship.
Its like firing a gun on a platformer while falling - Normally The bullet would just go across the screen. I would like the bullet to "fall" with the platformer while still proceeding to travel across the screen.