I think it shows how calcified standardized tests have become. We will have to revisit all of them, and change many things about how they work, or they will be increasingly useless.
I am struggling to imagine the frame of mind of someone who, when met with all this LLM progress in standardized test scores, infers that the tests are inadequate.
These tests (if not individually, at least in summation) represent some of society’s best gate-keeping measures for real positions of power.
This has been standard operating procedure in AI development forever: the instant it passes some test, move the goalposts and suddenly begin claiming it was a bad test all along.
There have been complaints about the SAT for how easy a test it is to game (get an SAT specific tutor who teaches you how to ace the test while not needing you to learn anything of actual value) for ages. No idea about the LSAT or the GRE though. Ultimately it’s a question of if you’re trying to test for pure problem solving ability, or someones willingness to spend ages studying the format of a specific test (with problem solving ability letting you shortcut some of the studying).
