For mutually-exclusive buttons with the labels inside the buttons, there exists such a convention: Group the buttons together with no space between them. Sometimes, the corners of the buttons at the far edges are rounded but the lines between them are not.
You will often see these in word processors, for aligning text ( left | centred | or right ).
A related issue with radio buttons and check boxes is that in some GUIs, clicking the label next to the box/button also clicks the widget, but there was no visual indication that that is the case. Sometimes the clickable area even extends the whole width of the window. This carries the risk of users making a selection when all they wanted to do was to click the window in a seeminglyinactive area to make it come to front.
I think we could solve that issue and visually connect radio buttons by placing them onto a background rectangle with rounded corners — thus adapting a visual cue from the button group widget.
Then highlight the background portion that is clickable when the mouse hovers over it, and when the button/area gets clicked.
Further, you could choose to make the colours consistent with those you'd use for selecting items in lists.
The text alignment buttons work because you know you can't have both centered and left, otherwise relying on subtle spacing would mostly signal some connection, not mutual exclusivity (similar to how those | symbols group related buttons together)
One solution to selectable labels would be to add the indeed missing indicator, e.g., you could have the whole label have a border when it's active (though in this case maybe you don't need radio buttons, but just have multiple lines that look like distinct lines?)
Hard to access the background group, I know it doesn't work if it includes labels like in the screenshot in another comment, but maybe only for buttons it'd be more understandable
I thought there was an actual difference in presentation back in the days toolbar buttons actually had borders, but no, when you look at https://winworldpc.com/res/img/screenshots/6x-a08a659bdbb00f..., Word 6, you see it’s all just simple grouping of toolbar buttons, nothing to do with mutual exclusivity at all: New|Open|Save, Print|Preview|Spellcheck, Cut|Copy|Paste|PasteFormatting (if my memory serves me correctly?), …, AlignLeft|AlignCentre|AlignRight|Justify, Numbered|Bulleted|Dedent|Indent.
I thought it was true when compared alignment group with bold/italics, in the Word Ribbon UI they do seem to have less space separating them, but then in Wordpad the paragraph button next to alignment group seems to have the visually very similar space width and the icon is very similar, so there is no way such poorly perceptible convention helps even if it does exist
A related issue with radio buttons and check boxes is that in some GUIs, clicking the label next to the box/button also clicks the widget, but there was no visual indication that that is the case. Sometimes the clickable area even extends the whole width of the window. This carries the risk of users making a selection when all they wanted to do was to click the window in a seemingly inactive area to make it come to front.
I think we could solve that issue and visually connect radio buttons by placing them onto a background rectangle with rounded corners — thus adapting a visual cue from the button group widget. Then highlight the background portion that is clickable when the mouse hovers over it, and when the button/area gets clicked. Further, you could choose to make the colours consistent with those you'd use for selecting items in lists.