After a bit more testing and observation today, I'm almost certain that it's tied to autosave.
As soon as my project autosaves, I can expect a long manual save straight after, no matter what (while using project folders). Meanwhile, disabling autosave results in no longer getting a long save time intermittently.
Interesting guess, i will check it later! But yea, for me it definetly not the project files size, and also it's not tied to actions i make, it's just seems random to me, so it easily can be related to autosaves
UPD: oh, i saw Ashley answer on GitHub, seems like you were right! Im so happy this problem will be fixed, I was struggling with it for a year at least now!