I might have found a clue.
My project has these settings: 540, 960. My layout sizes are exactly the same settings.
Now, I also have Scale outer applied. The 'visible' area in my game is 800 x 1388. That's the total size of the background, I covered the 'outsides' of that area with solid colored sprites matching the background.
I do this because the objects spawn in under these sprites and move towards the centre of the screen, and I don't want them to be visible 'outside' the background.
My reason for extending the background width and height is to adapt to different screen sizes (mobile, tablet) and that works fine in combination with Scale outer.
Of course my browser on desktop doesn't 'see' any viewport since I covered it with sprites. So basically the horizontal viewport 'borders' don't exist. They extend way until the end of the browser window.
Probably why it does work on mobile devices is because, I made the project so it fits on almost every phone/tablet, and constructed it so that the objects spawn in nicely at the extremes of most possible device screen sizes.
I think it's working fine, it's just that I will have to adjust the coordinates somehow of left and right viewport based on how much the screen stretches or shrunk the game screen depending on the size of the monitor, and its resolution on desktop.
So measure a 'fake' viewport only for desktop, probably using the width of the background image, since that will also shrink/increase and its borders are basically what I would want the viewport to be.
Now this is a theory of course. What do you think?
Sorry for the long post!