Think about it in terms of hue. Warm (red/orange) vs cool (blue/green). Your neutrals (like beige, whites, and greys) should match the temperature of the main colors.