I do think it’s an interesting line of inquiry… but not robust enough.
E.g. this paper would be much more interesting if it measured the threshold at which the LLM starts to become good at X, and linked that threshold to the number and character of training examples of X. Then, maybe, we can begin to think about comparing the LLM to a human.
Alas, it requires access to the training data to do that study, and it requires a vast amount of compute to do it robustly.
Good point - I saw the FLAN anomaly and this didn’t occur to me!
A good follow up question would be: why didn’t the other models do better on the 2nd-order question? Especially BLOOM and davinci-003, which were middling on the 1st-order question.
I agree on your overall criticism of the experimental protocol, though.
Skimming through studies like this, it strikes me that LLM inquiry is in its infancy. I’m not sure that the typical tools & heuristics of quantitative science are powerful enough.
For instance, some questions on this particular study:
- Measurements and other quantities are cited here with anywhere between 2 and 5 significant figures. Is this enough? Can these say anything meaningful about a set of objects which differ by literally billions (if not trillions) of internal parameters?
- One of prompts in second set of experiments replaces the word “person” (from the first experiment) with the word “burglar”. This is a major change, and one that was unnecessary as far as I can tell. I don’t see any discussion of why that change was included. How should experiments control for things like this?
- We know that LLMs can generate fiction. How do we detect the “usage” of the capability and control for that in studies of deception?
A lot of my concerns are similar to those I have with studies in the “soft” sciences. (Psychology, sociology, etc.) However, because an LLM is a “thing” - an artifact that can be measured, copied, tweaked, poked and prodded without ethical concern - we could do more with them, scientifically and quantitatively. And because it’s a “thing”, casual readers might implicitly expect a higher level of certainty when they see these paper titles.
(I don’t give this level of attention to all papers I come across, and I don’t follow this area in general, so maybe I’ve missed relevant research that answers some of these questions.)
There are multiple senses of the word “creativity”, and this post focuses on one of them: divergent thinking. The other sense is that of constructive, goal-oriented creation, which ideas alone cannot achieve. It’s too bad we don’t have have more commonly used terms to make this distinction. I see a lot of comments here focusing on that distinction rather than the post’s central thesis.
I do have a comment on the thesis, which is:
> The purpose of this article is to challenge this assumption [that creativity is binary] and discuss aspects of ideation, i.e. the process of coming up with ideas.
I support/agree with this challenge and all of the article’s ideas. “And yet”, right?? “And yet” some people are perceived to “have something” which others do not.
Honestly, the explanation is rather simple, or at least, simply stated. It’s neurodivergence. I’d further claim that cognitive styles gravitate to certain “attractor points”. (That’s scientific lingo for: certain patterns which fit well within the environment and which reinforce themselves. Like the pattern of wheel-ruts which attract wheels, which makes them stronger. The “environment” in this case is all sorts of things, including both the brain’s biological details, and the body’s physical+social environment.)
The strongest of these attractor points, we give labels: ADHD, various species of autism, etc. And of course the “normal person” attractor - not a point, but a broad area with its little micro-attractors and, sometimes, niche wormholes leading to more divergent areas.
People tend to clump around the strongest attractor points, and sometimes get pulled into other more smaller ones. This easily explains the perception of binary other-ness, especially when you consider that deviation from the norm - in any of the many directions - is, itself, a strong, influential force in this dynamic. To the extent that we try to build society to work well enough for the majority, anyone who deviates will have different and novel experiences of those systems.
But look, people are complicated and dynamic. We sometimes work to push away from these pattern-ruts, and other times we let ourselves be pulled into them.
This article is saying: YES. You can do things that make you ideate more divergently. You can also do the work to explore your own cognitive-behavioral niche, and which pushes your idea output into more novel, “creative” realms. Play is a certain type of work, when you need to push yourself to do it.
The article also addresses this:
> Good ideas do not have to be completely novel
> A hallmark of creativity is the knowledge or intuition of picking ideas that make suitable combinations. [more worthwhile to pursue]
…which brings us back to the other sense of creativity: not just divergence, but convergence; pursuit of a vision or goal or “gut feeling” intuition. I think this is the better, fuller meaning of the word. The author describes interaction between convergence and divergence very well. In the best examples of “creative genius”, both of these forces are at play. (No pun intended but perhaps that’s revealing.) Fluid, progressive creativity is at the edge of these two forces, and a “creative” person steers the ship, aware of both convergent goals and overarching visions that can only be reached by leaving those same goals behind.
The general skill of steering is quite meta-learnable by, probably, nearly everyone with any ounce of cognitive control. It takes time and support. It’s easier in more specific contexts, more well-suited to one’s situation.
For what it’s worth, toddlers absolutely do exhibit this full version of creativity, when you consider that they are pursuing the instinctive, hard-wired goal of learning and adapting to the world.
I’ve had a variety of responses to this list over my programming life (~10 years hobby, ~10 years professional).
When I first encountered them as a hobbyist, they were surprising, as perhaps intended, due to the framing of classic vices as virtues. On some reflection though, it made sense, and shaped my understanding of programming as somehow _inherently different_ than other types of creation.
When I got started with a professional career, they functioned to soften the edge of anxiety. It meant that the community of programmers who came before me - which presumably included Larry Wall - would understand that these patterns in coder behavior were ultimately beneficial and that I would maybe fit in with a corporate programming environment. (Now I know that this isn’t always true; sometimes coworkers, both programmers and non-programmers, don’t always realize these unintuitive points, and in some special cases, those programmer instincts aren’t actually valuable.)
At some point, I disagreed with the framing. As others have pointed out, the patterns can be reframed as classic or functional virtues such as curiosity. Then I backpedaled and realized that the framing is important because these are unintuitive patterns and it makes us re-think habitual incentives that reward, e.g., work that is more productive but not more effective.
How did it hit me now? I realize that it’s also related to power dynamics. Programmers are assets to their employers but they’re also potential disrupters. The traditional “virtuous” framing of potentially-less-effective behavior, like patience, is related to the organizing and taming of a workforce.
It’s also related to the types of problems we encounter. When we work with computers, Larry’s list does usually lead to more effective outcomes. But when working with other humans, who have their own agency and idiosyncrasies, the traditional virtues are better-adapted behavior. This is also more true for the complex technical systems we deal with nowadays. So, as our careers transition from programming to system engineering and/or management, the traditional virtues become more relevant.
Anyhow, this is an evergreen and thought-provoking nugget of wisdom. Thanks to Larry Wall and those who have preserved it.
The main annoyance I have with Larry’s take is that efficiency is confused with laziness, which are entirely different things. Laziness, at its extreme, is to delay doing something until absolutely necessary, or even avoid entirely, efficiency is to optimize for the least possible effort, which is sometimes the same as being lazy
Wow - can you elaborate on your process to “debug” these different layers? What comprises your personal or team’s feedback loop, given that 6 months is short? (I suspect it’s a lot of sequencing?) Are there good comparisons to be made with getting acquainted with a new codebase/technical system? Are there any particular computational tools involved? Pardon all the questions, this stuff is fascinating!
To be clear this is/was a hobby project. Getting access to DNA files is required yes, but sequencing wasn't really the limiting factor. It started with simply poking around my DNA file and then some friends many of which were only done on 23andme. Most of the time was spent reading countless papers and making hypothesis, trying to invalidate them and iterating. Constantly seeking out new ways to look at the problem and going from there. Every time I got a new DNA file from someone I could see how their dna fit into the current hypothesis. Because every DNA was different it was a great way to test them. It anything it made my job easier. I wasn't looking for a single snp, but common patterns.
Sometimes this has involved using nebula and their whole genome sequencing dna test to get much more accurate data, but more often than not the cheap dna tests most people do were good enough.
As for actual programs, I did write some quick and dirty scripts to scan dna files for specific snps, but mostly I would just read them as the parts I needed were not that long.
There is a fair amount of phenotype data to start with. What ultimately started this was knowing 1) that there was a number of conditions that are seen in statistically weird numbers in the LGBT and 2) sex hormone levels in the LGBT are not exactly what you would expect.
My goto fun question when talking with someone in the LGBT is if they have hypermobility. A good percentage do. In one specific example those with classical like EDS will have 21-OHD and thus POTS and elevated 17-OHP, backdoor DHT production (aka PCOS for women) etc.
The real question I have been pondering is what exactly do I do with this as this is just a fun puzzle, not my job, I don't work for any school etc.
I don't get it. If you can solve for LGBT genes that easily, then why isn't this in the news? Surely academic scientists would have tried this if all you needed was some scripting to find patterns?
Been interviewed by a medical news journal and this has all been done in the public over the last year so there was never a “release date” or anything. Honestly at the start I was just another person with a guess.
There are some scary implications such as in some cases we have had sexuality and gender changes once we knew how to “inject into the system”
I guess when you get down to it, it wasn’t actually that note worthy by itself as most cases are simply minor versions of already well documented conditions. It is only when you combine them that they add up.
And lastly given that I have not paid to get it peer reviewed and published formally it isn’t news yet. Again no school affiliation. Just now mostly helping treat a bunch of those common conditions I mentioned.
There has been some research mostly into gay men. But honestly the transgender data set is much richer. At the end of the day it is a minority that is being politicized so not exactly being investigated, but once you figure it out it is like shooting fish in a barrel there is so much easy research. Before this it was (simplifying but not) brain scans for the most part. It was mostly unknown.
I know of Sapolsky saying what a decade ago that LGBT brain structures are, like, flipped wrt. heterosexual people. But I thought the scientific consensus has been that there is no easy way to find a gay gene(s), so your claim of finding such low-hanging fruit seems to fly in the face of that. I'm already imagining that academic scientists would be ready to dismiss your work outright.
Consensus is that there is no single "flip this and you're gay" gene, but we've known for decades there is a genetic component because of twin studies. Fits right up with what this person says they've found.
Why didn't the medical news journal hook you up with a professor? They could've taken a look at your work. Like, how do you know your scripts isn't just doing pseudoscience and based on a superficial understanding of all those papers, especially if this is just a hobby project. There could be blind spots.
I am already working with a doctor and have talked with those in academia. They find it neat, but are not going to jump projects, they already have their area of study that they are working to publishing something on, not this.
There absolutely could be blind spots and been iterating on it all year each time the tweaks are smaller, but the core idea has not changed, but simply accumulated more and more evidence.
Like genes, the following collections of information are plans, as other entities (workers, compilers, cells, etc) can reliably use them to produce larger, more complex objects:
- Blueprint
- Instruction manual
- Recipe
- Code
Unlike genes, they’re all human-designed. The top-down forcing function - the “back” in the feedback loop which shapes them - is human artifice.
I’ll leave it as an “exercise to the reader” to consider the differences in how their environment and execution apparatuses affect the resulting objects.
Genes are like blueprints, but obviously not the same. For one, they haven’t passed the county permitting process! And living organisms are like buildings because you can point to the plan behind them. But I’ll be darned if a house has ever had to struggle for survival.
Summary: Scott Alexander recounts his gullibility to various well-reasoned crackpot arguments on a topic, and describes how he to decided to trust experts instead of investing time into learning enough to assess the topic for himself. Then he reflects on the nature of argument-accepting and its relation to rationality.
I don’t think the term “learned helplessness” fits well here. It suggests a lack of agency, whereas he exercised much of it, employing his skill of critical thinking to arrive at the right epistemic stance.
A better term might be “bounded agency”, to pair with the concept of “bounded rationality”. We recognize that we cannot know everything, and we choose how to invest the capability and resources that we do have. This is far from any type of “helplessness”.
He talks about the pitfalls of pure rationality. There can be competing explanatory frameworks for the same thing, and they often contradict each other. Rational arguments may seem rigorous like math, but are in practice standing on shifting sands.
It ultimately comes down to what you decide to believe in. This is where traditional values and religion come at play.
Yes, It's not "gullibility", it's believing things in terms of the mechanism of standard argumentation.
The basic thing is that arguments involve mustering a series of plausible explanation for all the visible pieces of evidence, casting doubt on alternatives, etc. Before Galileo, philosophy had a huge series of very plausible explanations for natural phenomena, many if not all of which turned out to be wrong. But Galilean science didn't discover more by getting more effective arguments but by looking at the world, judging models by their simplicity and ability to make quantitative predictions and so-on.
Mathematics is pretty much the only place where air-tight arguments involving "for all" claims actually work. Science shows that reality corresponds to mathematical models but corresponds only approximately and so given a model-based claim can't be extended with an unlimited number of deductive steps.
A further thought that is too much for an edit… one of Alexander’s final conclusions is:
> I’m glad that some people never develop epistemic learned helplessness, or develop only a limited amount of it, or only in certain domains. It seems to me that […] they’re also the only people who can figure out if something basic and unquestionable is wrong, and make this possibility well-known enough that normal people start becoming willing to consider it.
I think there’s better framing here as well: he is glad that a few people direct their own bounded resources towards what I’d call high-risk epistemic investments.
I’m also thankful for this. As species, we seem to be pretty good at this epistemic risk/reward balancing act - so far, at least.
E.g. this paper would be much more interesting if it measured the threshold at which the LLM starts to become good at X, and linked that threshold to the number and character of training examples of X. Then, maybe, we can begin to think about comparing the LLM to a human.
Alas, it requires access to the training data to do that study, and it requires a vast amount of compute to do it robustly.