I agree this is not a great test. What's good about it is that it is a constraint satisfaction problem, and I would expect LLMs to be pretty bad at unknown problems of this kind. Simple reason, an LLM only has a a finite number of layers and it cannot do arbitrary long searches.
I almost made ChatGPT write a Python program that creates a monthly work schedule (for imaginary workers) based on specific constraints (e.g. there are 10 workers, 2 shifts (morning and night), must work 40 hours per week, must have at least one weekend in a month off, 2 minimum workers per shift, no more than 3 consecutive working days, and so forth).
I am not sure if I could make it give me a working solution, however, and I have not tried Claude, for example, and I have not tried to do it with other programming languages. Maybe.
The issue was that it messed up the constraints and there were no feasible solutions, that said, it did give me a working program for this that had fewer constraints.
I don't understand what you're saying - the idea is that we're asking the LLM to generate code to perform the search, rather than run an arbitrarily long search on its own, right? So why should the number of layers it has matter?
I meant in general like in for gaming uses, integrated gpus are limited by the memory bandwidth of the DDR memory bus compared to the high bandwidth memory on dgpus.
It also very much depends on the quality of your documents. E.g. if you have a set of very good recipes, rather than a random set coming for the Internet, then you can get very good answers with RAG
Riding is more fun with less resistance, it just "feels better". That alone is an argument to choose the correct tire (there are trade offs obviously). From my own experience (MTB) I can say that the difference in rolling resistance are very noticeable.
Humans also stumble with that as well. Problems being CSV not really being that well defined and it is not clear to people how quoting needs to be done. The training set might not contain enough complex examples (newlines in values?)
Even if you get it to work 100% of the time, it will only be 99.something%. That's just not what it's for I guess. I pushed a few million items through it for classification a while back and the creative ways it found to sometimes screwup, astounded me.
Yeah and that's why I'm skeptical of the idea that AI tools will just replace people, in toto. Someone has to ultimately be responsible for the data, and "the AI said it was true" isn't going to hold up as an excuse. They will minimize and replace certain types of work, though, like generic illustrations.
"The British Post Office scandal, also called the Horizon IT scandal, involved Post Office Limited pursuing thousands of innocent subpostmasters for shortfalls in their accounts, which had in fact been caused by faults in Horizon, accounting software developed and maintained by Fujitsu. Between 1999 and 2015, more than 900 subpostmasters were convicted of theft, fraud and false accounting based on faulty Horizon data, with about 700 of these prosecutions carried out by the Post Office. Other subpostmasters were prosecuted but not convicted, forced to cover Horizon shortfalls with their own money, or had their contracts terminated. The court cases, criminal convictions, imprisonments, loss of livelihoods and homes, debts and bankruptcies, took a heavy toll on the victims and their families, leading to stress, illness, family breakdown, and at least four suicides. In 2024, Prime Minister Rishi Sunak described the scandal as one of the greatest miscarriages of justice in British history.
Although many subpostmasters had reported problems with the new software, and Fujitsu was aware that Horizon contained software bugs as early as 1999, the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases.
[...]
challenge their convictions in the courts and, in 2020, led to the government establishing an independent inquiry into the scandal. This was upgraded into a statutory public inquiry the following year. As of May 2024, the public inquiry is ongoing and the Metropolitan Police are investigating executives from the Post Office and its software provider, Fujitsu.
Courts began to quash convictions from December 2020. By February 2024, 100 of the subpostmasters' convictions had been overturned. Those wrongfully convicted became eligible for compensation, as did more than 2,750 subpostmasters who had been affected by the scandal but had not been convicted."
https://eclipse.dev/mat/ can handle very large graphs of objects using a similar approach. It also does implement some kind of paging, such that you do not have to load the complete graph into memory when running some of the graph algorithms.
MAT is useful enough on its own to make me seriously consider Java almost everywhere. So many tricky memory leaks in other languages made completely trivial.
Yes, in many languages you can combine a few things plus a core dump and figure leaks out too... but average people actually use MAT because it's largely transparent, and it can operate on running processes. Very few languages can compete with that in practice, much less with reasonable performance.
Is there a tool for the jvm that can track data locality? TFA mentions three for Linux binaries but that won’t do the mapping to Java source if we can combine them with `java -jar` at all.
It is like others relying on tracking memory allocations. Which has it"s place, but to really analyze memory usage you would need to be able to analyze which "objects" are keeping other objects alive (dominator tree in MAT)
Especially since Google had always had the AOSP as a reasonable way of demonstrating their commitment to the industry as a whole disjoint from the places where they thought the Google ecosystem provided extra value (which they admittedly require you to take wholesale)
AOSP has effectively been dead for a long time. You need Google Play services for almost everything now and more and more creeps into the proprietary portion.
No you don't. Writing an app to use google services is a choice. There's MicroG for silly apps that require google services, and there's also life without using silly apps that require google services.
Please note how you dismiss apps requiring Google Play services as "silly," when most people who use those services use them as part of the hardware root of trust programs for work apps, or other device attestation for banking apps. Putting demeaning adjectives in front of the things you're arguing against isn't a compelling argument, I think you'll find, and "life without" banking and work apps leads to a lower quality of life for a lot of people.
