If this is always happening with BT, it's possible you have some issue with the quality settings in pulse audio.
Checking this blog out: https://kn100.me/improving-bluetooth-audio-linux/
Specifically, you want to run your audio at the _worst_ quality that is still audibly reasonable to listen to - this gives you the best range, least CPU load, most robustness to external noise/interference, etc.
And the reason you wouldn't have problems with calls is because phone network audio is highly compressed, so you're not going to eat up the data buffer.
Checking this blog out: https://kn100.me/improving-bluetooth-audio-linux/
Specifically, you want to run your audio at the _worst_ quality that is still audibly reasonable to listen to - this gives you the best range, least CPU load, most robustness to external noise/interference, etc.
And the reason you wouldn't have problems with calls is because phone network audio is highly compressed, so you're not going to eat up the data buffer.