If you are evaluating how well people code without LLMs you are likely filtering for the wrong people and you are way behind the times.
For most companies, the better strategy would be to explicitly LET them use LLMs and see whether they can accomplish 10X what a coder 3 years ago could accomplish, in the same time. If they accomplish only 1X, that's a bad sign that they haven't learned anything in 3 years about how to work faster with new power tools.
A good analogy of 5 years ago would be forcing candidates to write in assembly instead of whatever higher level language you actually use in your work. Sure, interview for assembly if that's what you use, but 95% of companies don't need to touch assembly language.
> ... LET them use LLMs and see whether they can accomplish 10X what a coder 3 years ago could accomplish...
Do you seriously expect a 10x improvement with the use of LLMs vs no LLMs? Have you seen this personally, are you 1 10th the developer without an LLM? Or is the coding interview questions you ask or get asked, how to implement quicksort, or something?
Let's make it concrete, do you feel like you could implement a correct concurrent http server in 1/10th the time with an LLM than what you could do it without? Because if you jut let the LLM do the work I could probably find some issue in that code or alternatively completely stump you with an architectural question unless you are already familiar with it, and you should not be having an LLM implement something you couldn't have written yourself.
In that case, could you begin proving that point by having it write an http request parser. Let's make it easy and have it require content length header and no support for chunked encoding at first. Csn pick any language you like but since thats such critical infrastructure it must export a C API. Let's also restrict it to HTTP 1/1.1 for the sake of time.
Concidering this would probably at most take a days work to get at least a workable prototype done if not a full implementation, using an AI you should be able to do it in a lunch break.
For most companies, the better strategy would be to explicitly LET them use LLMs and see whether they can accomplish 10X what a coder 3 years ago could accomplish, in the same time. If they accomplish only 1X, that's a bad sign that they haven't learned anything in 3 years about how to work faster with new power tools.
A good analogy of 5 years ago would be forcing candidates to write in assembly instead of whatever higher level language you actually use in your work. Sure, interview for assembly if that's what you use, but 95% of companies don't need to touch assembly language.