It's really annoying, and stops the music I'm listening to.

My phone is a Samsung Galaxy S23 Ultra, and it doesn't have a built in DAC (or at least it's disabled in the SoC firmware).

Addendum: it's not that it unplugs completely, but rather the USB DAC loses power and/or data connection.