I had a similar problem I could hear driver hitting the ball but silence when it landed ,sound when I putted but no sound when it went in the hole, also no commentary at all.
But when I connected my opitical audio cable to my turtle beach transmitter I had everything in my headset perfect.
So went into xb1 settings… display and sound. I noticed the optical audio was set at "stereo uncompressed" and HDMI was set at "5.1 uncompressed" when I changed HDMI to "stereo uncompressed" all sounds out of the TV were perfect