Here's the part that really seems to matter the most.
This does create some danger of circular reasoning
(perhaps we're just carefully describing our own biases).
But we have to start somewhere, and basing our
evaluations on how people write actual code seems like a
good place.The really exciting point comes when we can
re-run all this analysis, basing it on actual job
performance, rather than interview results.
Absolutely. Results on the earlier screens and results on the later interview aren't exactly independent variables, and neither is the one that really seems to matter - subsequent on-the-job success. There are all sorts of biases and confounding factors likely to be shared between them, especially since there's no indication that the later interviews were even done blind w.r.t. the earlier screens. Until then, we're just measuring correlations between different interview techniques, and it should be no surprise that two different kinds of code-focused interviews show the highest correlation.
Author here. We do do the interviews blind to earlier screening results. That's clearly vital. But you're totally right that job performance is the real thing that matters
So, the jury is out until a followup. How to measure productivity without bias? Applying Demarco and Lister's "Coding War Games" approach may work between companies, but how to apply within a single company? What would be the standard error? A dozen new hires is a lot different than 300 interviewees.
What kind of skill set? My previous company contracted with Galye Laakmann McDowell to coach the engineers before an acquisition due diligence. I thought it was unnecessary because we rarely used O() notation in our day-to-day. Since then, I'm designing a machine learning product and find sections in McDowell's book extremely relevant. My takeaway is that it all depends: are you hiring an architect or a hiring a construction crew? How do you measure such different skill sets? Or, how to measure with a job that combines both?
Hi Ammon. I'm certain that we talked on both the 45-minute and 2-hour video conferences, so those are not blind. Do you mean that you and Harj didn't discuss applicants throughout the process?
