I would definitely want this to be optional. I personally prefer my very nice "home theater" speakers over the tiny, tinny DS4 speaker. IIRC, the original reason for that speaker in the DS4 was for player voice comms.
Generally the controllers' speaker is used for secondary audio, like in game comms, phone calls, audio logs, specific warnings chimes etc, which was likely always its' main intention.
Normally when someone wants to speak to other players they use a headset or even the single bud headphone and mic which came with the system (even as cheap as it looks and feels, it does work, though Bluetooth headsets/earpieces are generally better), though for when you don't want to connect a mic and wear a earbud/headset (or just don't feel like talking, but still want to hear other people talk) most multiplayer games, such as Battlefield games, Rainbow Six etc, play player audio though the main speakers.
For example, the PS4 version of GTAV, as mentioned, plays phone calls through the controller speaker by default, but it still plays all other sounds, including player audio if a mic isn't connected and in vehicle radio through the main speakers - really the speaker is normally used for more incidental/atmospheric sounds, in most games, rather than any other types of audio present
.
(Also off topic, but in reference to your signature - Tea in Elite's time, sells for more than aluminium and explosives, probably because it can only but grown on Earth for some reason and has to be shipped all over the Galaxy...
.)
