That's basically why I said I'd have put the threshold number much higher than a "mere" 100k.
But even then I would say that the point is still important: disregard singularity, foom, and paperclipping, and AI is still often described as being akin to a faster rerun of the industrial revolution. Just having all the turmoil and unexpected consequences — health, social, environmental, economic, military, political — of the industrial revolution squeezed down from "economy doubling every 30 years" (which is what we saw) to "economy doubling every 18 months" (Moore's Law), is easily going to cause catastrophic consequences all by itself, even with no malice.
"Health and safety" rules had to be created because too many bosses thought it was just common sense to not put your hands into dangerous equipment while in motion, while paying people to do jobs that could only be accomplished by doing exactly that; how can we do more than guess what the equivalent to that is for AI before we see it?
What is to AI, that which global warming is to coal power?
What's the necessary consequence of everyone using AI, which is analogous to the necessity of building a sewage system in response to widespread installation of flushing toilets in cities?
We organised ourselves to build the latter, but we still haven't globally dealt with the former.
The only reason we’re still using coal is due to run away safetyism. That’s like the number one example of how being overly cautious or irrationally stigmatizing something when you should have just kept improving the technology can result in massive negative consequences like climate change.
Also, you’re wrong about flushing toilets necessitating sewers. They were still manually emptying toilets in London when they built its sewage system. So sewage systems became popular before flushing toilets were widely adopted.
> The only reason we’re still using coal is due to run away safetyism.
China building new coal plants (while also building new solar, wind, and nuclear) suggests that is not the only reason.
And it's not like any of the complaints about PV elsewhere are safety related.
> So sewage systems became popular before flushing toilets were widely adopted
What existed was not, however, sufficient. That is what necessitated building a massive new system:
"""The Great Stink was an event in Central London during July and August 1858 in which the hot weather exacerbated the smell of untreated human waste and industrial effluent that was present on the banks of the River Thames. The problem had been mounting for some years, with an ageing and inadequate sewer system that emptied directly into the Thames."""
"""During the early 19th century improvements had been undertaken in the supply of water to Londoners, and by 1858 many of the city's medieval wooden water pipes were being replaced with iron ones. This, combined with the introduction of flushing toilets and the rising of the city's population from just under one million to three million,[b] led to more water being flushed into the sewers, along with the associated effluent."""
"""The Building Act 1844 had ensured that all new buildings had to be connected to a sewer, not a cesspool, and the commission set about connecting cesspools to sewers, or removing them altogether."""
Just because a thing existed, doesn't mean they had previously done it at scale: the Ancient Greeks had electricity, railways, and steam; battery cars predate gasoline; we've today got the tech for a fully global power grid; none of this is at the scale relevant to the problems, and the same can be assumed for at least one possible thing that AI might force us to consider.
My guess would be spam and filters. Already exists, but likely to become so much more severe the old solutions will no longer hold.
The brand new flushing toilets that belonged to London’s upper class weren’t the primary cause of the great stink. The much bigger contributor was the other 99.9% of the population which had doubled in only 50 years preceding the stink. They had no flushing toilets, they just emptied their chamber pots into the street and it eventually found its way into the Thames after enough rain. The solution to the problem was to both upgrade the system and redirect more waste into the system instead of the surface gutters so it could be routed to an appropriate location. Its right there in the text you quoted!
The BBC podcast I recently listened to that was specifically on this topic doesn't agree with you, and I clearly don't read the same meaning as you do for that text.
Not that it really matters to my core point: what's the AI equivalent.
