You will not believe this. Solution was adding a line in some config file.

Pretty sure this used to be the fix for me:

https://forums.linuxmint.com/...

( scroll down to comment about default.pa and tsched=0 )

But I just checked my default.pa and it is stock values, so I am not sure anymore

When experimenting, setting cpu governor to performance also helped.