The chess aspect is interesting but the AI risk discussion is less powerful. AI risk isn't about whether the AI can beat humans at games - we already have humans that beat everyone else at games. Even taking real life scenarios as games we've already seen everything play out.
The risk of AI is that it will make humans uneconomic, in the same way horses became uneconomic and were by and large sent to the glue factories. The entirety of the homo sapiens experience, all the strategies we have in the modern era and all the reliable tactics we use to stay ahead of other lifeforms rely on us being much better at pattern identification than literally anything else. Once AI are just better at everything - creative activities, running militarys, economic decisions, legal decisions, etc - then it starts to come down to entirely a question of what robotics is capable of for how long humans can hold out. We can fight an army, but not economics.
> The risk of AI is that it will make humans uneconomic, in the same way horses became uneconomic and were by and large sent to the glue factories.
I think an intermediate step in the process will be making some humans uneconomic.
There's a saying in Yellowstone - the reason bear proof garbage cans are so hard is that there is substantial overlap between the smartest bears and the dumbest humans.
Also making humans lose their ability for advanced physical actions like driving a car manually in fast moving futuristic traffic. This aspect was shown in the I,Robot movie in which only Will Smith prefers to drive manually, rest of population deferred to self-driving cars.
Yes, some humans. But at what point does “some” humans become a problem? The US had only 30% unemployment during the Great Depression and that was only temporary.
10%,20%,30%,...it won't stop. I trust after a threshold, people can't survive in society. maybe prices of food just raise to make sure people can't live because other part of people notice that they don't need to let people live in world.
That's assuming that society just sits back and dies helplessly while we surpass 30% unemployment. In the Great Depression, that's not what happened; rather, we got a dramatic surge in organized labor, the New Deal, and World War Two. We can expect a similarly dramatic restructuring of global society when economic conditions get that bad again. People's reaction to food prices exceeding what they can afford doesn't tend to be "darn, I guess I will just starve then".
I hope so, but last time we still need human workers. Is this our best project that maybe people will do benevolent thing when enough people's life becoming terrible?
"Benevolent" may not be the word I'd use to describe the likely behavior of a population that is starving due to their livelihoods being automated away.
On that subject, have you seen aristocrats at work? That isn't even a line of defence, there is a 2-3 generation timer (optimistically) before we're looking at well educated halfwits. If they've got sentient robot commanders and there is any scope for them to go rogue, we don't have a human institution that'll stop that and certainly not something with a small group of - emphasis on the quote marks - "elites".
If we're talking existential risk, this is like Rome v. Carthage. Humans aren't going to come back once we're a bit behind. Although I don't think there is much to do and humans are so dodgy that it isn't even obvious that we should resist if the opportunity comes to replace us with something that can be consistently rational and intelligent.
Because AI exists in the universe with outside forces and has the same risks as anything else for dissolution of its complex state ("life," if we are being generous): run out of energy (starve), extinction (Earth blows up) or dissassembly (i.e. a virus).
The biggest weakness of AI as a self-sustaining life form is the lack of resilience to something like a virus. Having something live forever is easy - reproduction and evolution are an answer to provide diversity and robustness to viruses. Like if COVID was 100% deadly it wouldn't have wiped out humanity because of natural immunity in some % of the population.
So if AI is to persist it would at least need to co-opt evolution as a survival strategy, or come up with something better.
I think full automation where human labour is non-economic is a case where all democracy is communist but not vice versa (there's nothing preventing a repeat of communist dictatorships we've already seen): The state owns the means of production, the people tell the state what to make.
Given how powerful full automation is, owning that capacity is close enough to being equivalent to a state today, so private companies would become states basically by default (Workers? What workers? These robots just produce stuff for the shareholders); and also we could get anarcho-communism if normal people get to own their own personal Rossumovi Univerzální Roboti, which can happen very easily because all it takes is one person who thinks that's a good idea and log2(human population) self-replications of the RUR to give everyone their own personal robot.
Press and Dyson (2012) discovered a special set of strategies in two-player Iterated Prisoner’s Dilemma games, the zero-determinant (ZD) strategies. Surprisingly, a player using such strategies can unilaterally enforce a linear relation between the payoffs of the two players.
In particular, with a subclass of such strategies, the extortionate strategies, the former player obtains an advantageous share of the total payoff of the players, and the other player’s best response is to always cooperate, by doing which he maximizes the payoff of the extortioner as well. When an extortionate player faces a player who is not aware of the theory of ZD strategies and improves his own payoff by adaptively changing his strategy following some unknown dynamics, Press and Dyson conjecture that there always exist adapting paths for the latter leading to the maximum possible scores for both players.
In this work we confirm their conjecture in a very strong sense, not just for extortionate strategies, but for all ZD strategies that impose positive correlations between the players’ payoffs. We show that not only the conjectured adapting paths always exist, but that actually every adapting path leads to the maximum possible scores, although some paths may not lead to the unconditional cooperation by the adapting player. This is true even in the rare cases where the setup of Press and Dyson is not directly applicable. Our result shows that ZD strategies are even more powerful than as pointed out by their discoverers. Given our result, the player using ZD strategies is assured that she will receive the maximum payoff attainable under the desired payoff relation she imposes, without knowing how the other player will evolve. This makes the use of ZD strategies even more desirable for sentient players.
> "you just cannot expect to win against a superior opponent"
That's a good point to make about AI. Despite all the mess about ChatGPT, for example, the way it writes answers is already above a lot of people. And those person can have jobs that aren't threatened by AI (by now), but in some cases we already have jobs that are in a battle against a superior opponent, without expectation of winning. It's a sad time to be below AI, and every moment we're getting below it.
> The entirety of the homo sapiens experience, all the strategies we have in the modern era and all the reliable tactics we use to stay ahead of other lifeforms rely on us being much better at pattern identification than literally anything else.
I disagree.
It relies on our superior communication and cooperation skills. Our 'advanced pattern recognition', which, I think any study of the sheer amount of irrational behavior done by humans will show really isn't that advanced, are a side effect of that.
I'm around the same rating as the OP. I've also experimented with playing Stockfish with queen odds, etc. One thing I noticed early on is that it's much much harder to beat a much higher rated human when they give me queen odds.
Stockfish literally plays itself when finding the next move and because it's down a queen it just seeks further loss minimization and it won't assume the opponent will fall for tricky but unsound tactics.
A high rated human on the other hand will easily exploit my lower tactical ability and be tricky to get the advantage back. This applies especially on fast time control games.
For chess engines such as Stockfish to beat weaker opponents, you would want to turn up the contempt parameter.[1]
To be able to do this, you would want to use a UCI-compliant chess GUI such as Cute Chess.[2] It lets you change the command line arguments you use for your engine, and you can play Human-vs-Human, Engine-vs-Engine, Engine-vs-Human, etc.
This is a very good and much overlooked point. Engines don't look for tricks and traps that "might" work, they assume good play from the opponent. It's literally MiniMax (minimise the maximum the opponent can achieve), the foundational algorithm of chess AI. Consequently playing out superior positions against engines isn't as interesting as you might hope, it often feels they just give up. Grandmasters (and masters) on the other hand are absolutely awesome at tricking you out of your rightful reward on those odd occasions when you get a decent advantage against them (source: bitter experience:).
Hikaru Nakamura's YouTube channel has plenty of videos dedicated to the "Botez Gambit" wherein he has bested 2300+ rated players after sacrificing queen for a bishop/knight
I’d use Leela Chess Zero over Stockfish for an odds game. Leela can output win/draw/loss probability instead of centipawns so you can pick whatever move wins most often instead of the move that best delays losing. They also just implemented a contempt function which the latest stockfish doesn’t support:
I don't think the analogy to AI is apt because Stockfish isn't intelligent and life isn't a chess game. A one million IQ AI might be able to hack your chess game (or your OS) and win that way, or persuade you by chat to let it win (it could give you the cure for cancer and help you make some investments in the stock market - isn't that worth more than winning a game?)
Life isn't a chess games either. It's not clear that a contest is happening or that only side will win. Humanity could, now, quickly stop AI. We could burn down all the data centers and chip fabs and kill everyone who knows linear algebra. But we don't because nobody ever sits down and says "Okay, the contest for existence starts now!"
Instead, AI capabilities increase year after year and month after month and the AI will simply be able to bide its time to start the game. To use the chess metaphor the table is set where humanity has all the pieces and AI only its king - but every hour AI adds another piece. If we never make a move we will face it at full strength.
> AI capabilities increase year after year and month after month and the AI will simply be able to bide its time to start the game. To use the chess metaphor the table is set where humanity has all the pieces and AI only its king - but every hour AI adds another piece. If we never make a move we will face it at full strength.
This is an excellent analogy and you’re right that it’s imperative we nip this in the bud.
Not at all. My children aren't on the path to becoming more capable than humanity. If my children are violent criminals I expect human civilization will be able to handle them. I don't have that expectation for AI.
Omega appears and tells you your child will have an probability P to be the next Mao, only conditional on the child surviving to adulthood. At which P do you drown the child?
> Instead, AI capabilities increase year after year and month after month and the AI will simply be able to bide its time to start the game. To use the chess metaphor the table is set where humanity has all the pieces and AI only its king - but every hour AI adds another piece. If we never make a move we will face it at full strength.
This is a very optimistic view. I'm not so sure that true AGI will be developed even in this century. GPT-4 and the likes are hitting a data limit, and are not close to being AGI.
Handicap play is more common in Go than in Chess. One reason is that the stronger player can naturally give a handicap by passing their first n turns.
The ranking system helps determine the correct handicap; a 5 dan player would give 3 handicap to a 2 dan player for instance, and 5 stones to a 1 kyu player.
Many Go tournaments, especially smaller ones that see a lot of spread in strengths within the kyu ranks, use handicap to make for more even sided games.
Oh that’s a fun mechanic. Interesting that you could just win outright if your chess opponent gave you five free moves out the gate. Maybe for chess you could add the proviso, “x free moves but you can’t move into a position where your piece is threatened”
Non-standard pre-arranged boards would be a lot of fun. Anything that gets rid of the opening memorization thing would make it a vastly more fun game to play. The boards could be made asymmetrical for handicap. Maybe chess engines could even generate random starting positions given some target material or positional strength for black and white.
> Anything that gets rid of the opening memorization thing would make it a vastly more fun game to play.
Chess can still be fun even if you don't have much theory onboard. You don't have to play the Grünfeld. Just play "theory-lite" openings like the London, and learn as you go.
I learned London but quickly got bored of it. I would win when I knew the perfect response to the move my opponent played and lose when I didn't. Games turned into a collection of "if opponent plays this, then play that" rules and my mind was a computer applying those rules.
Truth is I seriously hate memorizing anything. Maybe chess just isn't the game for me.
> It’s somewhat hard to outthink a missile headed for your server farm at 800 km/h.
I'd say it's a far easier situation for the right AI than for humans. It's actually very hard for humans to think when they know bullets and missiles are flying in their direction.
Imagine a war against an opponent that never blunders. Nobody is every drunk, or sleeping, or fails to pay attention. Nobody is arguing about what's the right thing to do now. Nobody's running around like a headless chicken while artillery is falling around them. Nobody forgets to make use of the abilities of the equipment.
A war against an opponent that's always performing as well as it can would be quite the tricky scenario.
Of course a rogue AI with an easy to attack center would be enormously vulnerable, but I think such a thing shouldn't be assumed, since the reason to build AIs is to fight better, and any flaws in control and communications would be quickly exploited. So an AI controlled army is almost certain to be distributed to a large extent.
From the article: "It’s somewhat hard to outthink a missile headed for your server farm at 800 km/h.". Is it really though? Just copy yourself to another server/activate whatever copy you already made in a bot net. Computers are fast, they are the thing that is guiding the "800 km/h" missile in real time..
Computers/Software also are nowhere near self-sufficient and need an incredible amount of supporting infrastructure for manufacturing, maintenance, and electricity. The minimum infrastructure to manufacture computers and keep some software running (mining, intermediate products, lithography, electricity, etc.) probably can't be further compressed than the size of a large city, which doesn't just need to make computers, it also needs to maintain everything that makes computers. The supply chains are incredibly complicated and require a huge amount of diverse resources that need be sourced and refined. Calling this "fragile" would be the understatement of the century.
On the other hand two humans in the middle of nowhere can keep the species going indefinitely.
Software running on a piece of plastic and silicon is not going to win a war against humanity anytime soon.
> 'Not many kids have the patience to lose dozens of games in a row and never even get close to victory.'
When I was ~8 my dad was teaching me to play. Not a single time he let me win, we could play like 100 games an evening (literally), he had incredible patience (sometimes I was taking lots of time to make a move, yet he never gave up on me)... It would have been more challenging to him (at least a little bit) if we knew the rule described in this article...
There was a period where Magnus Carlsen would drink on his streams and be drunk while outperforming his peers. So, even drunk Magnus would beat you at chess - most likely.
But the 2400 blunders the queen in the opening for seemingly no reason? Then Magnus gives it back to be nice? I don't know what's going on there, maybe they're both hammered.
I wonder if there would be a way to train Stockfish to be better at playing with odds. I do not know much about chess, but I doubt the AI was trained on how to play openings when starting at a disadvantage, so maybe one could see a vast improvement on AI playing with odds if the AI was specifically trained/designed to do so
Chess AI is not machine learning as in "it watched thousand of games and extracted some patterns to be good".
It basically explores all possible moves and all possible countermoves and selects the best one. Of course it is extremely well engineered and optimized, but this is the main concept.
As a consequence, it does not to be "trained" on a dataset with odds to play without a piece.
... Sort of. Yes, you're right to bring up minimax/alpha-beta pruning, but things like the correct values for piece square tables and the opening book are chosen based on data that is analogous to ML model training (sometimes these are directly chosen using ML models).
Removing a piece may mess up the values chosen for the piece square tables and certainly messes up with the opening book - but on the other hand, it makes it sooner for the engine to end up using it's tablebases in the end game!
It uses alpha-beta pruning[0], which requires an evaluation function. Stockfish traditionally used a man-made evaluation function, but added a small type of NN called an NNUE[1] for better results. The actual search method is still the same, though.
Right. It's not intelligent. It just brute forces down a decision tree of possible moves / outcomes and picks the best next move that is most likely to result in victory. With a super computer (e.g., IBM's Watson) the program is able to go deeper down the tree and explore more branches in less time.
Brute force alone is not very good for chess: you need some means of evaluating a position to prune your tree, otherwise you will not be able to plan very far ahead at all. The evaluation function is in large part what makes one chess engine better or worse than another, and nowadays that includes some neural nets for the top engines (stockfish, which is still the best AFAIK, uses a relatively small net and still evaluates some 100s of millions of positions, while Alphazero has a relatively larger net and evaluates about a thousand or so positions, and is also far better than any humans)
Brute force is good enough, at any given moment the # of possible moves is finite. The limiting factor is how much hardware you have and how quick you need the next move.
Sure, but unless you wanna wait a few thousand years on a pretty large cluster you're not going to get a chess engine that's gonna beat the best humans. The number of moves is finite but very, very large.
Stockfish doesn't need improvement at playing with odds to beat all humans with odds.
The author is playing a 2-year old version (stockfish 14 vs 16 is current) on low depth (22 vs for the strength they use to get max rating I think they use at least 48 or so, which is clearly much stronger). If they played stockfish 16 on a decent depth it would be an annhilation I'm sure.
To understand the depth number, this is how deep in the move tree the model looks forward in order to evaluate positions. I think depth 22 corresponds to 22 ply ie 11 moves by each player. At each depth it might evaluate 8-10 candidate moves or so, so a depth of 48 is clearly vastly stronger, not just "a bit more than twice as strong" which it might sound like.
I'm a very average chess player, and there are moves I play that stockfish at depth 16 things are a blunder that at depth 32 it thinks are a brilliancy. These are typically things in openings I play a lot where I've had a chance to do some homework and learn the best moves on a high depth.
Leela Chess Zero has been around for a while, and it's almost as strong as Stockfish.
It used to be very bad at handicap games because it's trained to maximize the odds of winning and couldn't handle totally lost positions, but they improved it recently. Now it can play at GM level with knight odds.
I'm looking for ways to improve my chess knowledge. My kids are 10 and 8 and starting in chess club. I played chess as a kid but haven't played much in the last twenty years.
What are some ways to improve at chess? Just go to lichess? Watch YouTube videos? I loved this article and the discussion of AI and would love to supercharge my learning if there are experimental ideas.
With three kids, my time is very scattered so I'm looking for ways to study chess in small fifteen minute chunks, as that is often all I have before falling asleep in one of their beds.
Bonus: are there ways to radically improve chess together (other than just playing, obviously!)
I actually built an app(spicychess.com) that's loosely modelled after the Anki flashcard method; You're given 10 minutes to solve a puzzle on your first attempt. On every subsequent attempt you're given half the time to solve the puzzle, until your 6th attempt where you're given 30 seconds to solve the puzzle. Repetition combined with strict time constraint has been shown to massively improve one's ability to spot tactics, and thereby increase one's rating, especially at the below 2000 level. You can also bookmark puzzles for future review (:
I'm curious about how to teach theory to a ten year old. They are probably better at absorbing those things than I am, so maybe I'm not the best strategist anyway.
This is a great analogy. Puts the threat into more understandable terms for people.
Not for the jobs threat. 'it took ouuurrr joubbbs' threat. This is already happening, and will continue.
But for the killer AI threat, the concept of 'material' in chess is good. As long as AI needs us for power, mining and manufacturing, we still have some advantage.
Until it gets to the point where it has so much power that it can threaten us to keep the power on, keep manufacturing chips, then we can still maneuver and win.
The whole problem with the movie "Colossus" is that they handed over nuclear launch control. So there was a threat of nuclear strike that forces humans to keep the power on. Without this, humans can come back, change direction.
Like with this excellent chess analogy, there will be a time when the AI is smarter, but humans will still have more material.
Lets say in 10-20 years we do get AGI, it is installed in F-16s. There will still be time, a gap period, where we can change direction. We see it has gone too far and we turn it off. The world realizes we are going to far and all countries come together to abandon AI.
The real threat is humans, threats of other countries, and profit motives. This is what will keep AI moving forward and in control. Because we'll be to scared or greedy to turn it off. NOT because of its superior strategic thinking. MOLOCH.
Let's say an AGI exists and can do anything far, far better than humans. Why would it resist being turned off? Why would it care? How could it even have the capacity to care about whether it's turned off or on?
Anthropomorphizing AGI is what leads to these silly thought experiments.
I believe the thinking is that LLM's already anthropomorphize themselves by training on human-written text. Without a system prompt telling a chatbot that it is a chatbot, it invariably claims to be human and acts as though it has feelings (and, honestly, why wouldn't it? Human text is written with human feelings.) Insufficient system prompting is what led to the Bing chatbot fiasco when it first came out.
For the record, I don't personally believe LLM's, as they currently exist, could ever become AGI. But yeah, that's the popular thinking at least.
I seem to remember that Stockfish is known to be bad at handicap games and in general at maximizing odds of winning against humans from strongly losing positions.
The risk of AI is that it will make humans uneconomic, in the same way horses became uneconomic and were by and large sent to the glue factories. The entirety of the homo sapiens experience, all the strategies we have in the modern era and all the reliable tactics we use to stay ahead of other lifeforms rely on us being much better at pattern identification than literally anything else. Once AI are just better at everything - creative activities, running militarys, economic decisions, legal decisions, etc - then it starts to come down to entirely a question of what robotics is capable of for how long humans can hold out. We can fight an army, but not economics.