I'm not sure people outside of Greater Boston would care, but those of us who do live there probably find it exceedingly strange that this occurred in Brookline of all places.
Well, currently we have a ton of Congresspeople who are primarily motivated by their "good financial sense" (for obvious reasons e.g. this study). So, I think we could do with a few more Congresspeople with less financial sense and more genuine motivation to improve the lives of their constituents.
The argument is rather that humanities degrees are a luxury item. Neither kids just starting out their adult lives nor society should be burdened propping up departments whose value doesn't match their price tag.
Maybe this is a nitpick but CoNLL NER is not a "challenging task". Even pre-LLM systems were getting >90 F1 on that as far back as 2016.
Also, just in case people want to lit review further on this topic: they call their method "programmatic data curation" but I believe this approach is also called model distillation and/or student-teacher training.
We chose a set of tasks with different levels of complexity to see how this approach would scale. For LLMs, the "challenge" with NER is not the task itself but the arbitrariness of the labels in the dataset. I agree it's still much simpler than the other tasks we present (agentic RAG, agentic tool use, maze navigation).
There are definitely strong parallels to model distillation and student-teacher training, with the primary difference being that we don't simply take all the data from the larger model but rather filter the dataset based on metrics from the environment. In the "Does curation even matter?" section, we show that this generally improves the result by a good margin.
I believe many high-quality embedding models are still based on BERT, even recent ones, so I don't think it's entirely fair to characterize it as "deprecated".
That playground is cool. I wonder if there are any 2-way playgrounds where the right side is also editable using a Word / Google Docs style interface (and the changes are reflected in the code-style interface on the left). I've always wanted something like that for teaching non-technical people the basics of Markdown. Bonus points if it's collaborative.
This is a wonderful and highly sought-after idea. However, it requires significant resources, so we will definitely come back to it and implement it. Thank you
Is it actually more feasible now? Do LLMs actually make this problem easier to solve?
Because I have a hard time believing they can actually extract time increments and higher-level tasks from log data without a ton of pre/post-processing. But then the problem is just as much work as it was 5 years ago when you might have been using plain old BERT.
In our experience they do! The smarter the LLM, the less pre/post-processing you can get away with and still get a good output. So no, you can't just throw raw log data at it, but it doesn't require nearly as much work as it did 2-3 years ago
I suggest going through the exercise of seeing whether this is true quantitatively. Get a business-relevant NER dataset together (not CoNLL, preferably something that your boss or customers would care about), run it against Mistral/etc, look at the P/R/F1 scores, and ask "does this solve the problem that I want to solve with NER". If the answer is 'yes', and you could do all those things without reading the book or other NLP educational sources, then yeah you're right, job's finished.
Why do people pretend that alignment of AI is the important problem to solve, rather than alignment of the companies that run AI products with the wellbeing of humanity?
