The primary argument for why it's not a violation is that the heuristic is (almost) free. LLM designed samplers are and will probably be better - but in order to start the recursive self-improvement engine a few free heuristics will be needed.
The bitter lesson critique is that the human designed heuristics were not free, and harmed the notion of "letting the computer figure it out" by slowing down training. High temp sampling is very important for half-way decent synthetic data generation and thus enabling "letting the computer figure it out" for natural language. Better sampling is the only way to make high temperature generations coherent.
The bitter lesson critique is that the human designed heuristics were not free, and harmed the notion of "letting the computer figure it out" by slowing down training. High temp sampling is very important for half-way decent synthetic data generation and thus enabling "letting the computer figure it out" for natural language. Better sampling is the only way to make high temperature generations coherent.