First of all: Your assumption is right.
Now to your issue. Your description sounds like the webfont gets loaded from a server each time you actually use it? That's indeed strange as it should be cached once loaded. So, maybe the download wasn't completed.
Have you followed Ashleys advices exactly? From the manual:
"However, you can work around this by loading all web fonts on startup. For each web font you use in a project, add a text object to your first layout and use the Set web font action to load it. Make the text hidden so it's never seen - it's just loading the web font. Splash screens or title screens are ideal for this. If your splash or title screens use a web font, you might want to use an image there instead, or introduce a second or two delay to give the fonts a chance to load. Then, nobody should ever see the text before the fonts have loaded."