Raspberry Pi in my experience (4b) is less than ideal for audio playback duties.

Pops and clicks from buffer underruns have proven quite difficult to 100% eliminate.

I think it's primarily due to some devices being pinned to CPU0. You can run irqbalance, but not everything can be moved around.