Check out DeepSeek v3 model paper. They changed the way they train experts (went...

		boroboro4 3 months ago \| parent \| context \| favorite \| on: Kimi K2 is a state-of-the-art mixture-of-experts (... Check out DeepSeek v3 model paper. They changed the way they train experts (went from aux loss to different kind expert separation training). It did improve experts domain specialization, they have neat graphics on it in the paper.