In many languages the distinction is quite blurred. In Japanese, most color words like 茶色 chairo (brown), 灰色 hairo (gray), 黄色 kiiro (yellow), 紅色 beniiro (crimson/red) incorporate the word "iro" (color), but while the first two are tea-color and ash-color, ki 黄 and beni 紅 now just mean "yellow" and "red" respectively.

It looks like 黄 and 紅 never had a distinctive non-color meaning (as characters) without 色, so maybe using them without 色 is more a recovery of their historic meanings rather than an invention of a new meaning?

(That's just a guess, I especially don't know the roles that written and spoken forms have played in the evolution of these meanings, or whether spoken "ki" and "beni" did or didn't have independent meanings.)

I can provide some non-native commentary on modern Mandarin Chinese. All color terms are most ordinarily used with 色 ("color"); it is unusual, though possible, to use a color term by itself.

It doesn't seem like a big stretch to imagine that this was also true when the terminology was borrowed into Japan from China, and that's why the 色 gets used in Japanese.

I can't claim to be an expert, but this is likely correct. The native Japanese words for color (shiro, kuro, aka, ao, midori etc) don't use the -色/iro suffix.

Japanese has the cool feature that you can clearly identify siro/kuro/aka/ao (a fairly typical four-way white/black/red-yellow/blue-green distinction) as the most basic colour words with relatively little diachronic analysis: they're the ones that stand alone in the rather limited class of 形容詞 ("i-adjective") stems.

You can almost trace the development of colour terms through the vocabulary strata...

Note that you still can use the the iro/color suffix for aka (red), ao (blue), midori (green), you just don't have to and they also function as standalone adjectives.

E.g. akai (赤い) -- red (adjective) vs akairo (赤色) -- the color red (noun)

compare: chaiiroi (茶色い) -- light brown (adjective) vs chaiiro (茶色) the color light brown (noun) -- literally "tea color"