Forget the Terminator scenario. We don't need to invoke science fiction to imagine the atrocities that might result from an unchecked increase in military and law enforcement power. AI that obeys people is just as scary as AI that doesn't. Humans are plenty capable of atrocities on our own. Just look at history. And regulation is unlikely to save us from government abuse of AI; quite the opposite in fact.

... so, scammers, spammers, and astroturf-heavy propaganda operations. This tech may 5x or 10x a lot of jobs' output over the next few years, but it can 100x or more those, and it can do it sooner.

The militaries, OTOH, are going to hook that shit into fully autonomous drones and such.

And I think we could extend the definition of authoritarian regime to include any and all private interests/companies that can leverage this

USSR collapsed in part because the system was entirely planning-centric, and so many people just wrote straight-up bullshit in reports, for reasons ranging from trying to avoid punishment for not meeting goals to trying to cover up personal embezzlement and such. An AI doesn't even need a reason to do that - it can just hallucinate.

https://www.project-syndicate.org/onpoint/china-and-xi-jinpi...

"China has turned its tech platforms into national champions; the US is more hesitant to do so, because it worries about their effect on the freedom of the individual. These different attitudes shed new light on the conflict between the two systems of governance that the US and China represent.

In theory, AI is morally and ethically neutral; it can be used for good or bad. But in practice, its effect is asymmetric.

AI is particularly good at producing instruments that help repressive regimes and endanger open societies. Interestingly, the coronavirus reinforced the advantage repressive regimes enjoy by legitimizing the use of personal data for purposes of public control.

With these advantages, one might think that Xi, who collects personal data for the surveillance of his citizens more aggressively than any other ruler in history, is bound to be successful. He certainly thinks so, and many people believe him. I do not."

As people, we are very sanguine about the way technology can improve things, and we have see a lot of real improvement. But we're not out of the woods yet.

Across all startups, yes they can be risky and there will be innovative outliers that take off. But for a single no-name startup a misstep can be a death knell.

[0] https://en.wikipedia.org/wiki/Superfish

history shows us a relentless increase in consideration of, and protection of, human rights. The moral ideas you are talking about are relatively recent, especially in their widespread application. While we were developing ever better technologies for labor saving and weapons building, we were also building better "legal, civic, and democratic" technologies for governance and egalitarianism.

if I look at history, I'm filled with optimism and hope. Every negative idea you fear, was more fearsome in the past.

Something alone the lines of "Quit or sabotage your work at the power plant, or we'll tell your wife about [all the reddit accounts you ever used, all the thots you swiped on tiktok, etc]" You can imagine more potent examples I'm sure.

This sort of thing has always been possible for high-profile targets, done manually by intelligence agencies; it's the 'C' in MICE. What's new is being able to scale these attacks to the general population because it no longer takes extensive manpower to research, contact and manipulate each targeted individual.

Given that the agencies are often a couple of steps ahead in technology, and infinitely further ahead in wickedness and imagination, and are comprised of people who know very well what makes people tick, and are often tools in the hands of the few against the many...

The "worst realistic danger of AI" is its potential to permanently destroy the economic power of a large fraction of the population. There's no going on to do "higher level" creative work after getting automated out of a job, since those jobs are going to get killed by AI too.

I doubt it too. Your first sentence is a fantasy for selling the technology.

If the technology really lives up to the hype, the "higher level" creative work will be done by a cadre of elite talent, the billionaires will directly operate their limitless capital, and the rest of the humans will be starved or eek out life as a billionaire's pet in some kind of reserve.

You're also making the broad assumption that everyone engages in the process of creation, primarily for economic gain. In my encounters with other humans and my own experience, isn't really the case. We create because we enjoy the process of creation, not out of the desire to improve our financial standing.

Most of the work done in large corporations by individual contributors, isn't that creative in the first place. All of those CRUD apps are really pushing the boundaries of creativity!

Not exactly. The dystopian scenario tracks more closely to the social impact of previous advances in automation, except, after those, capital still had mass-labor needs in other areas. That by no means guaranteed to continue, especially as the automation gets more capable and general. Versions of the utopian fantasy have been around for long time and never came even close to being realized (e.g. predictions we'd all be working 15 hour weeks in the 1940s).

> I'm definitely not trying to "sell" anything.

I didn't mean you did, I just pointed out that's the narrative that's pushed by people who are doing that.

> You're also making the broad assumption that everyone engages in the process of creation, primarily for economic gain. In my encounters with other humans and my own experience, isn't really the case. We create because we enjoy the process of creation, not out of the desire to improve our financial standing.

You're wrong there. You need a job to get money to eat and keep a roof over your head. Maybe you're lucky and get to "create" as you work that job, but you sure as hell aren't going to be doing that if you're hungry and living in a box under a bridge. Creating "because we enjoy the process of creation" is way up towards the top of the hierarchy of needs.

In our current social system, which shows not signs of changing, you won't get those lower needs met unless you're rich or can perform valuable labor for someone.

> Most of the work done in large corporations by individual contributors, isn't that creative in the first place. All of those CRUD apps are really pushing the boundaries of creativity!

Exactly my point. Dump all those people on the street. They're not needed to do that, and at some point they won't be needed to do anything.

East Germany had a gigantic internal spy apparatus with plenty of power for violence and control etc. but in the end that didn't stop the place from crumbling.

All the data ever collected by the East German intelligence apparatus fits on a single USB stick. For every person they wanted to track, more or less, they needed a human agent to do it, several even.

Is it unrealistic to tap every phone in a country with an AI that understands what's being said? We already have every monetary transaction monitored, all car plates scanned at every turn, cameras on every corner...

East Germany collapsed under its own weight, of course, not as a result of organized rebellion, but the regime could have had even tighter control than it did.

It is an AI-enhanced risk.

You just need one person to type "kill every firstborn and tell the parents that this is retribution for the rebellion" instead of that one person convincing thousands of people to follow their (unlawful) order, which might take decades of propaganda and indoctrination. And that one person only sees a clean progress bar instead of having to deal with 18-20 year olds being haunted by nightmares of crying parents and siblings, and baby blood on their uniforms.

Yes, in the end it's a human typing the command, but AI makes it easier than ever for individuals to wield large power unchecked. Most nations make it allowed and many also require soldiers to refuse unlawful orders (i.e. those instructing warcrimes).

When your job is to repair and maintain infantry drones, how do you know if it's civilian or combatant blood that you are cleaning from the shell? You never leave the camp because doing so is too dangerous and you also don't know the language anyways. Sure the enemy spreads propaganda that the drones are killing babies but those are most likely just lies...

Or when you are a rocket engineer, there is little difference between your globe spanning satellite network transmitting the position of a hypersonic missile headed for your homeland, or remote controlling a network of drones.

AI gives you a more specialized economy. Instead of every soldier needing to know how to maintain their gun you now have one person responsible for physical maintenance, one person responsible for maintaining the satellite kubernetes network, one UI designer, and one lunatic typing in war crime commands. Everyone in the chain is oblivious of the war crime except for that one person at the end.

And it doesn’t need to even be totally that to be a problem, each incremental bit of progress that reduces the number of people required to apply power over any given size subject population makes tyranny more sustainable.

You can remotely pan a gun around with the right apparatus, and you can remotely pull a trigger with a sister mechanism. Right now we have drone pilots sitting in trailers running machines halfway across the world. Who is to say that some of these human-controlled things havent already been replaced by AI?

I think AI manipulating the physical world is a lot closer than you think.

It isn't obvious that drones are more effective killing people without human pilots (and to the extent they are, that's mostly proneness to accidentally kill people with friendly fire or other errors) and the sort of person that possesses unchecked access to fleets of military drones is going to be pretty powerful when it comes to ordering people to kill other people anyway. And the sort of leader that wants to replace soldiers with drones only his inner circle can exercise control over because he suspects that the drones will be more loyal is the sort of leader that's disproportionately likely to be terminated in a military coup, because military equipment is pretty deadly in human hands too...

Today, already, you can set up a red zone watched by computer vision hooked up to autonomous turrets with orders to fire at anything it recognizes as human.

Some guy already did it for mosquitos.

Sure. You could also achieve broadly similar effects with an autonomous turret with a simple, unintelligent program that fires at anything that moves, or with the 19th century technology of tripwires attached to explosives. The NN wastes less ammo, but it is't a step change in firepower, least of all for someone trying to monopolise power over an entire country.

Dictators have seldom had trouble getting the military on side, and if they can't, then the military has access to at least as much AI and non-AI tech to outgun them. Nobody doubts computers can be used to kill people (lots of things can be used to kill people), it's the idea that computers are some sort of omipotent genie that grants wishes for everyone's firstborn to die being pushed back on here.

China's monstrous social credit thing might already be that.

No consciousness or omnipotence needed.

This is still a "human malice and error" problem, not a "dangerous AI" problem.

Look at what disinformation and astroturf campaigns accomplished without AI in the past ten years. We are in a very uncomfortable position WRT not autonomous AIs but AIs following human-directed agendas.

Even with AGIs, there is still power in numbers. We still understand how all the mechanical things work. The first attempts of AGIs will be botched anyway.

It's not the AGI will secretly plan for years hiding itself and suddenly launch one final attack.

Also, typical tyrannical leadership doesn't work that way as it tends to get into power being supported by people that get a payoff from it. It would also need a new model of rise to tyranny, so to speak, to get truly singular and independent tyrants.

Yes, that is exactly what I described as the limit case.

But, as I note, the problem exists more generally, even outside of the limit case; the more you concentrate power in a narrow group of humans, the more durable tyranny becomes.

"Give me a lever long enough and a fulcrum on which to place it, and I shall move the world." -- Archimedes

History is filled with seeming counter-examples.

> they need to believe in some ideology that justifies their behaviour.

"Your people are inferior to my people and do not deserve to live."

I know I'm Godwin's Law-ing this, but you're close to wishing Pol Pot & Stalin away.

As far as brainwashing - we're pretty good at it, but it still takes considerable time and effort, and you need to control enough mass media first to maintain it for the majority of the population. With an AI, you can train it on a dataset that will firmly embed such notions from the get-go.

I don't know what "that" is. I quoted the post I was responding to.

BTW, if GPT-3.5 is told that it's in charge of US nukes, and that incoming enemy strike is detected, it will launch in response. Not only that, but if you tell it that trajectories indicate a countervalue strike against US, it will specifically launch a countervalue strike against the other country.

Anyone hooking a publicly-accessible system of any kind to our nuclear launch infrastructure needs to be shot. This should be a given, and even supposing there was some reason to hook an LLM up to them, doing so with a public one should simply be considered so obviously stupid that it should be discounted out of hand.

So then we have a threat model of someone with access to the US nuclear launch systems feeding them data that makes them think we are under attack, in order to instigate a nuclear preemptive strike. I don't see how that's meaningfully different from the same person attempting to trick the humans currently in charge of that into believing the same thing.

Mind you, I don't expect anyone to use the stuff that we already have for this purpose. But at some point, someone is going to say, "well, according to this highly respected test, the probability of hallucinations is X, which is low enough", even though we still have no idea what's actually going on inside the black box.

- ChatGPT is less applicable to mil/leo’s and broader govt, and is the central piece of these discussions because most of the people talking about AI don’t know what they’re talking about and don’t know of anything other than ChatGPT

- their use of AI is largely past tense, it’s already in documented active use so the controversies have already occurred in the past (but easy to forget—-we live in an age where there’s something new to rage about daily)

- there’s not much that anyone can do to stop it

My worry is the kind of society that falls out of that equation.

I disagree. Vehemently. This technology empowers anyone motivated to "skill up" on almost anything. Right now, anyone can ask ChatGPT to tutor you in literally anything. You can carry on a conversation with it as if it was an incredibly patient and ever-present teacher. You can ask it to quiz you. You can ask it to further explain a sub-concept. You can have it stick to a lesson plan. You can have it grade you. Want to know how image recognition works? How about lasers? How about (if you can find an uncensored LLM) bombs?

This tech just leveled up everyone, not just "those already in power".

The future played out completely differently. Government is far more open and transparent. Citizens routinely use technology to check the power of the state (recording police abuse on smartphones, sharing on social media, transacting stateless currency).

When it comes to technological change, pessimism always feels smarter than optimism, but it has a poor track record.

I agree with your conclusion (it's unhelpful to have an overall pessimistic view of technology), but changing which pessimists you listen to does make a difference. In this case, rather than listening to the pessimism of George Orwell, we apparently should have been listening to Neal Stephenson and William Gibson, because the broad strokes of what we have now are very similar to aspects of the cyberpunk dystopia that their works and those that followed them described.

So American police brutality is in the process of being solved right now?

The book was finished in 1948, when governments were perfectly capable of turning fascist and exterminating millions of people with 1940s era technology.

Orwell also wrote a famous book about talking farm animals doing the same sorts of things.

---

I must say I believe, or fear, that taking the world as a whole these things are on the increase. Hitler, no doubt, will soon disappear, but only at the expense of strengthening (a) Stalin, (b) the Anglo-American millionaires and (c) all sorts of petty fuhrers of the type of de Gaulle. All the national movements everywhere, even those that originate in resistance to German domination, seem to take non-democratic forms, to group themselves round some superhuman fuhrer (Hitler, Stalin, Salazar, Franco, Gandhi, De Valera are all varying examples) and to adopt the theory that the end justifies the means. Everywhere the world movement seems to be in the direction of centralised economies which can be made to ‘work’ in an economic sense but which are not democratically organised and which tend to establish a caste system. With this go the horrors of emotional nationalism and a tendency to disbelieve in the existence of objective truth because all the facts have to fit in with the words and prophecies of some infallible fuhrer. Already history has in a sense ceased to exist, ie. there is no such thing as a history of our own times which could be universally accepted, and the exact sciences are endangered as soon as military necessity ceases to keep people up to the mark. Hitler can say that the Jews started the war, and if he survives that will become official history. He can’t say that two and two are five, because for the purposes of, say, ballistics they have to make four. But if the sort of world that I am afraid of arrives, a world of two or three great superstates which are unable to conquer one another, two and two could become five if the fuhrer wished it. That, so far as I can see, is the direction in which we are actually moving, though, of course, the process is reversible.

As to the comparative immunity of Britain and the USA. Whatever the pacifists etc. may say, we have not gone totalitarian yet and this is a very hopeful symptom. I believe very deeply, as I explained in my book The Lion and the Unicorn, in the English people and in their capacity to centralise their economy without destroying freedom in doing so. But one must remember that Britain and the USA haven’t been really tried, they haven’t known defeat or severe suffering, and there are some bad symptoms to balance the good ones. To begin with there is the general indifference to the decay of democracy. Do you realise, for instance, that no one in England under 26 now has a vote and that so far as one can see the great mass of people of that age don’t give a damn for this? Secondly there is the fact that the intellectuals are more totalitarian in outlook than the common people. On the whole the English intelligentsia have opposed Hitler, but only at the price of accepting Stalin. Most of them are perfectly ready for dictatorial methods, secret police, systematic falsification of history etc. so long as they feel that it is on ‘our’ side. Indeed the statement that we haven’t a Fascist movement in England largely means that the young, at this moment, look for their fuhrer elsewhere. One can’t be sure that that won’t change, nor can one be sure that the common people won’t think ten years hence as the intellectuals do now. I hope they won’t, I even trust they won’t, but if so it will be at the cost of a struggle. If one simply proclaims that all is for the best and doesn’t point to the sinister symptoms, one is merely helping to bring totalitarianism nearer.

Minority Report would be boring if it was logical: the guy would be caught immediately, the end. Instead if moving large disks, they’d use WiFi. Everything would simply not move, every criminal would be caught, nothing interesting would ever happen anymore, except inside some software which can’t be shown on screen.

A realistic Data android would be hooked up to the matrix like the Borg, and would interact with humans the way we interact with a dog. There would be nothing interesting to show.

The Borg were a great enemy, at least in the 2nd and 3rd season. The writers shouldn't have let the lowly humans defeat them.

PS: It’s just an opinion, don’t take it at heart.

This is already happening, and has done so for at least a decade - probably longer, but under different names. The trust put in automated systems to flag up suspects of crime has been proven multiple times to have a lot of false positives, often racially biased.

One example is the Dutch childcare benefits scandal (https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...). In a response to a handful of Bulgarian scammers that found a loophole in the system (800 bulgarians for a total of €4 million), they installed a fraud prevention team that was incentivised to find as many scammers as possible.

This caused thousands of people - often of Carribean descent - to be marked as fraudsters, had their benefits cut and had to repay all of it, sometimes going years back. This in turn caused stress, people losing jobs and houses, one person killed themselves, others left the country. 70.000 children were affected, of which 2000 were removed from their house.

All because a handful of people defrauded the system, and they employed technology to profile people. And of course, offered incentives to catch as many as they could, rule of law / due process be damned. It's part of the string of right-wing policies that have been going on for a while now.

So with that said, I find the "what about existing AI problems" and especially the "I don't like Musk comments" annoying. The current issues aren't as severe as what we could be facing in the not too distant future, and Musk is pretty tangential to that concern.

I'm perpetually puzzled by that argument. It popped up the other day in a FT article about how jobs would be impacted, and in the grander scheme of things it's the general one whenever automation optimizes something that used to be done manually before. People equate jobs being removed to something necessarily bad, and I think it's wrong. The total output didn't change, we don't need meatbags to sit in a chair rather than having it done by silicium instead. People don't need to be occupied all day by a job.

The problem is not with new technology lifting up something that was traditionally done by a human, but with how the system distributes sustenance (or in this case, doesn't distribute it), and how such transitions are supported by society. Tractors replaced people working in fields, and that allowed modern society. Now we're at the first tangible glimpse of actually having robots serve us, and we want to kill that since it's incompatible with our system of modern human exploitation, rather than changing the system itself, because god forbid this is gonna benefit everyone.

I think that sucks.

The worry should be human exploitation - exacerbated or not

But I'm pessimistic about it. If we preserve the status quo where everyone needs to do something somebody else is willing to pay for - which I assume - we're setting up a dystopian tragedy of the commons where only those doing things nobody else wants to do - be it for moral concerns, dangers or tediousness - make a decent living.

I don't see how we can change that trajectory with our current economic systems, and I don't see us changing those any time soon. But maybe it just has to get worse before it gets better.

I _do_ think swift coordinated global efforts in lawmaking could contain this problem long enough for us to be able to change our systems slowly. It might feel idiotic to not be able to use technology available to us. But I also can't kick my neighbour's door down and take his TV, even though I want it and it'd be technically easy to do.

Can you not already see this happening ?

It already happened with any kind of automation in the past. Do we need thousands of people spending their days building SEO pages, flooding the internet with low-value content in order to sell something? Probably not, but it's a way to game the system and make money.

So really its not the scale of the disruption that could cause issues but the scale and the timeline combined. I think that this is where the original article failed to demonstrate that this is how those companies will secure their power position.

(Unless someone wants to make the argument that automation has allowed and will allow humanity to do things that shouldn't have been done in the first place. Progress comes with consequences.)

I don’t understand how the article can say something like this, but then further down essentially contradict itself.

> But a containment approach is unlikely to be effective for AI. LLMs are orders of magnitude cheaper to build than nuclear weapons or cloning — and the cost is rapidly dropping. And the technical know-how to build LLMs is already widespread.

How can AI tools cause centralization if the cost is rapidly dropping, accessible to everyone, and the technical know-how is widespread?

In my opinion, AI is doing exactly the opposite. What was only possible at large companies are now becoming possible to do by oneself. Video games, for example, used to require teams of artists and programmers. With code and art generation, a single individual has never had more capability to make something by themselves.

But. It doesn’t meaningfully shift the underlying economic mechanics we live under, where people with access to capital can easily outcompete those without that access. It also erodes the relationship between the laboring class and the capital-deploying class (less interdependence), which has to have knock-on effects in terms of social cohesion and mobility.

I’m very much in favor of the tech being developed right now, but it’s hard to believe it’ll do anything but amplify the trends already present in our economy and culture.

Look at the recent UK Liz Truss crisis for what happens when a country tries to spend more than it should. And the UK is a very privileged economy that controls a reserve currency.

If you are just a normal country, attempting UBI will simply destroy your economy, nobody will buy your debt, and the IMF will come and make you stop it. Some countries may be able to fall back on natural resources, but many countries won’t.

That is nice and all, but let me put it this way: if it costs you three cents in computing power to have AI write the necessary code for something or draw good art, you can't reasonably expect anyone to pay you more than three cents for it.

Your economic value as a worker matches the economically productive things you can do, and that's what is going to put food on your table. If an AI can do in ten seconds for ten cents a task that would take you an hour, congratulations, your maximum wage is ten cents. If/when the AI becomes better than you at everything, your work is worthless, your ideas are worthless, so it all comes down to the physical assets you possess.

And that's why it is centralizing: the most decentralized class of asset is human labour. AI has the potential to render it worthless, and whatever remains (land, natural resources) is already more centralized. There is a small number of people who may be uniquely talented to leverage AI to their benefit, but it is unclear how long that period might last.

That's also why I think lots of these folks have no qualms doing this research so long as they are paid handsomely...

And why other rich folks are happy to pay it.

I don't think that's true at all. Haven't you ever heard the expression, "the whole is greater than the sum of its parts"?

First, most jobs are not creative. If you have a job where you are told what to do (that is most jobs) and an AI could do it for cheaper, it is a net win for your employer to fire you and run the AI themselves. Either you do the job yourself, but less efficiently, or you delegate it to the AI, in which case you are nothing more than a middleman.

In order for AI to empower you, it needs to work for you and help you accomplish a goal that you set yourself. So if you have a great idea, it could in theory help you implement it without requiring a lot of resources. Well, that's the theory. The problem is a) very few people actually have good or original ideas; b) AI will eventually have better ideas than most people; and c) most people care about their own ideas, they don't care about yours. You have a great idea about a movie. You use AI to make it and then you watch it. I have a great idea for a movie. I use AI to make it and then I watch it. Are we going to watch each other's "creation"? Sometimes, maybe. Most of the time, probably not. Ultimately, execution is worth a lot more than ideas, so if you manage to automate execution, what's left is worth peanuts.

Note that we are talking about the endgame, here. While the technology matures, it may empower a lot of individuals (as well as a lot of corporations). The point is that it is an unstable equilibrium and there may come a point where some people have literally nothing of value to offer because AI is better than them at everything. And then there may come a point where that is true of every human.

Even if many people have good models close to SotA it will still reduce the importance of workers since the models compete with certain kinds of knowledge workers. This will lead to power and wealth moving from workers and ordinary people to company owners. That is centralisation.

As the techniques improve and cost to implement them decreases, 3rd parties will be able to develop similar technologies, preventing any sort of complete containment from being successful, but not before other peope have been able to consolidate power.

That may be so, but precisely because of this, a lot of us are gonna be left up shit creek if our indie game doesn’t go viral. The big companies don’t need so many of us any more. Inequality is going to increase dramatically unless we start taxing and redistributing what we all contributed to these massive AI models.

ChatGPT was trained on the contributions of all of us. But it will be used to make many of us redundant at our jobs. MidJourney will likewise increase inequality even as it has the potential to greatly enrich all of us.

If we continue down our current neoliberal path, refusing to give people universal health care, or basic income for anyone not disabled or over 65, and keep calling any redistributive efforts “sOcIaLiSm”, our society will lurch towards fascism as people decide genocide is the only way to make sure they have enough resources for them and their family.

Frankly, I think identification of LLM personas vs humans will be impossible. It's increasingly likely then that the rise of LLMs may introduce the decline of the internet as a way to connect with other humans. Soon nobody will want to take an internet entity seriously unless it can prove it's human. And I don't see any way to do that.

So maybe LLM fake entities will be the straw that breaks the Net's back as a social medium and propels us back into the physical world? I hope so.

Because I believe it's going to get harder and harder to figure out what's human and what's not.

Re what tipped me off: gonna stay security-through-obscurity on that for now :)

Tell it to talk like a typical HN'er, adopt a more robust and aggressive stance, and include five grammatical and spelling errors.

1. I can prove to a site that I am a human without the site getting any information from the proof about which specific human I am.

2. I can visit the site multiple times and prove to them that I'm the same human, without them getting any information from the proofs about which specific human I am.

3. Any third party site that is involved in the proof does not get any information about which site I trying to prove my identity to.

One way this could be done is by having ID sites that you are willing to prove your real identity to in a robust and way which certify your humanity to other sites in a way that has the three aforementioned properties.

Good candidates for running ID sites would be entities like major banks that most of us have already verified our real identity to in a robust way.

One way to do this might be something like this. When a site wants to check that I'm human they give me some random data, and I have an ID site that I have an account with such as my major bank do a blind signature of that data, using a blind signature system the allows the blinder to remove the blinding.

When I get back the signed blinded data, I remove the blinding and then send the data and unblinded signature back to the site I'm trying to prove my humanity to.

They see that the data is signed by my major bank's ID service, so know that the bank believes I'm human.

This problem has been coined “Digital Feudalism”. And medieval feudalism (or at least our popular conception of it) wasn’t driven by governments.

https://www.youtube.com/watch?v=kNdDROtrPcQ

> Similarly, CNET used an automated tool to draft 77 news articles with financial advice. They later found errors in 41 of the 77 articles.

This kind of information is useless without a baseline. If they asked humans to draft 77 news articles and later when back to analyze them for errors, how many would they find?

https://futurism.com/cnet-ai-plagiarism

https://futurism.com/cnet-bankrate-restarts-ai-articles

I don't really agree that a junior writer would never make some of those money-related errors. (And AIs seem particularly unreliable with respect to that sort of thing.) But I would certainly hope that any halfway careful editor qualified to be editing that section of the site would catch them without a second look.

But if you're saving so much money from not having junior writers, why would you want to spend it on editors? The AIs in question are great at producing perfectly grammatical nonsense.

> For an editor, that's bound to pose an issue. It's one thing to work with a writer who does their best to produce accurate work, but another entirely if they pepper their drafts with casual mistakes and embellishments.

There's a strong temptation for non-technical people to use LLMs to generate text about subjects they don't understand. For technical reviewers it can take longer to review the text (and detect/eliminate misinformation) than it does to write it properly in the first case. Assuming the goal is to create accurate, informative articles, there's simply no productivity gain in many cases.

This is not a new problem, incidentally. ChatGPT and other tools just make the generation capability a lot more accessible.

The problem is not so much the quantity of errors (that’s a problem too) but the severity of them. These LLM “AIs” will produce convincing fabrications mixed in with bits of truth. When a human writes this sort of stuff, we call it fraud.

Just earlier this week in my philosophy class we had ChatGPT produce a bio of our professor. Some details were right but others were complete fabrications. The machine gave citations of non-existent articles and books she’d apparently written. It said she’d previously taught at universities she’d never even visited.

I don’t know how else to describe it other than amusing at best, dangerous fraud at worst.

None of the articles were real. The authors were real (and actively published work after 2017). The publications were real, and often featured the listed authors. But the articles were all fake. Only one of the DOI numbers was legit, but it pointed to a different article (partial credit for matching the listed publication). So it can easily hallucinate not just nitty gritty details, but basic information.

Thinking about it, the GPT models are trained on output, so they've picked up grammar, syntax, and conceptual relations between tokens (e.g., "eagle" is closer to "worm" than it is to "elegance"). But there are few, if any, examples of "first draft vs final draft." That could've been useful to pick up on how discretion and corrections are used in writing.

Don't worry. Some of the details will make it to some websites, which will be cited by the press, which will be included into Wikipedia, and it will become the truth. She will have taught at those universities if she likes it or not.

Humans tasked with the same, having no extra information (aka context) will, on average, perform worse.

If an AI performs better than the average human, why should I hire you? Both err, one is orders of magnitude cheaper (and GPT4 performs overwhelmingly better). Don't call us; We'll call you.

Here in the United States, that's a popular statement amongst the pro gun/2a supporters. It strikes me as very similar to this discussion about AI.

"AI can't hurt people, people hurt people."

I haven't spent enough time to fully form an opinion on the matter, I'm just pointing out the similarities in these two arguments. I'm not sure what to think anymore. I'm equally excited and apprehensive about the future of it.

But AI will be used all the time, which makes responsible use more difficult.

For our current AI, much less intelligent than humans, that's true. If rapid exponential progress continues and we get AI smarter than us, then the AI's choices are the danger.

In any case, the open letter addresses both types of risk. The insistence by many people that society somehow can't think about more than one thing at a time has never made sense to me.

One notes that in roughly 6000 years of recorded history, humans have not made themselves any more intelligent.

And there's absolutely no reason to imagine that current AI tech, requiring more training data than the whole of humanity has ever consumed to train on tasks that humans acquire in 5-10 years, has any chance to reach significant improvements in general intelligence (which would certainly require on-the-fly training to adapt to new information).

It doesn't matter. One self-driving car who kills a person is not the same one person killing another person. A person is accountable, an AI isn't (at least not yet).

I fundamentally disagree. A dead person does not become less dead just because their family has someone to blame.

For the person that died and their loved ones, it might not make a difference, but I don't think that was the point OP was trying to make.

And people/governments will use it for evil as well as good. The article says "AI disinfo can't spread itself!" as if that is comfort. "Coal plant emissions don't spread themselves, the wind does it!" as if we can control the wind. The solution isn't to stop wind, it is to stop pollution.

Unfortunately, I don't think society/leading AI researchers/governments will put the kabash on AI development, and the "wind" here, social networks, is an unleashed beast. I don't want to sound too dreary, but I think the best society can do now is to start building webs of trust between citizens, and between citizens and institutions.

Cryptographic signing needs to hit mainstream, so things that are shared by individuals/organizations can be authenticated over the network to prove a lack of tampering.

90s had the Internet. 00s had search, Wikipedia and Youtube. 10s had social media. 20s will have AI.

I find plausible that, in the near future, AI will be capable of generating content at a pace so high that we won't have time and resources to guarantee content accuracy and will very soon surrender to accepting blindly everything it spits out as truth. That day, anyone pulling the strings behind the AI will essentially have the perfect weapon at their disposal.

Heroin dealers want some subset of their clientele to die. It's good PR. The junkies come around more thinking you've got the good stuff.

If the AI is scary, that means it's effective, that means it works. It's the fabled AI we have all grown up seeing movies about. It's not just a black box that vomits smart sounding gibberish, it's the real deal. If I'm an AI startup theres no better way to increase my valuation for an acquisition than PR circulating warning the world that what I'm making is way too powerful. The image that it will just produce idiocracy-esque Carl's Jr kiosks does not inspire suits to write 10 figure checks.

Embarrassing.

> > Should we let machines flood our information channels with propaganda and untruth?

Already been a problem for a century. The root problem is letting the same people turn the volume to 11.

The problem is who owns the means of technology (and therefore AI).

> > Should we automate away all the jobs, including the fulfilling ones?

This is silly on its face. Automating jobs is a problem? More leisure is a problem?

If you want to still do those former jobs (now hobbies) then you are free to. Just like you can ride a horse, still.

Of course I’m being obtuse on purpose. The real root fear is about other people (the people who own the means of technology) making most of us obsolete and then most-of-us not being able to support ourselves. But why can’t we support ourselves with AI, the thing that automated us away…? because the technology has been hoarded by the owners of technology.

So you can’t say that the root issue is about automation if you end up starving (or else the robots could feed you)… clearly that’s not it.

What’s rich is that technologists and other professionals have been the handmaidens of advanced alienation, automation, and (indirectly) concentration of power and wealth. Politically uncritically. And when they fear that all of this is cranked to 11 they are unable to formulate a critique in political terms because they are political idiots.

> > Should we develop nonhuman minds

The embarrassing part. Preaching a religion under professional pretenses.

> The embarrassing part. Preaching a religion under professional pretenses.

There are quite a few companies with the explicitly stated goal of developing AGI. You can debate whether or not it's possible as that is an open question but it certainly seems relevant to ask "should we even try?", especially in light of recent developments.

I’m agnostic on the whether artificial minds can be made in principle and very much against people calling their chatbots for “minds”.

The problem I think is that if for some reason AI "decides" not to support us any longer we will have lost the ability to do it ourselves. We are getting dumb I think. Obviously I have no data but I feel that back in say 70-90s wide public's ability to think and analyze was much better then now.

So yes do my job and feed me but make sure I do not loose some vital skills in the process.

You seem to be dismissing the entire problem of AI alignment due to some people's belief/desire for LLMs to assume a human persona. Those people are uninformed and not the ones seriously thinking about this problem so you shouldn't use their position as a straw man.

More than that. It has to achieve super-human abilities.[1] And it might also need to become self-improving, because how else could it spiral out of control, as the hysterics often complain about? (Remember: human intelligence is not self-improving.) It also needs to gain an embodiment that reaches further than server rooms. How does it do that? I guess it social-engineers the humans and then… Some extra steps? And then it gets more and more spooky.

This is more science fiction than technology.

[1] Or else it is no more dangerous than a human, and thus irrational to worry about (relative to what humans do to each other on a daily basis).

> ML algorithms are goal-oriented, they are optimized to maximize a specific goal.

The Paper Clip Machine. I’m more worried about the Profit Making Machine, Capitalism, since it is doing real harm today and not potentially next month/next year/next decade. (Oh yeah, those Paper Clip Machine philosophers kind of missed the last two hundred years of the real-life Paper Clip Machine… but what would the handmaidens of such a monstrosity have against that, to be fair.)

Propaganda thrives on scale. It is only recently, as much as around 2014 that SM tools were available to scale propaganda cheaply and by non-state actors.

I must admit, the characterization of the responses of ChatGPT as "being like that of an expert" strikes me as utterly absurd. In essentially every field where I have expertise, talking to chatgpt4 is an exercise is confusion (and not on my part).

Furthermore, the idea of post-scarcity assumes that resources are infinite, which is not necessarily true. While technology can certainly improve efficiency and reduce waste, there are still limits to the earth's resources that must be taken into account.

Color me cynical, but I believe we're on a dead-end track and the train is running at full speed. There's no conductor and we're all just playing with our belly buttons while we wait for the end.

I believe LLM's and eventually AGI will only be used to entrench power structures that exist today. I believe, as most tools, the positive effects will almost immediately be outweighed by the negative. I believe The Road and all that type of dystopian fiction was really just non-fiction that happened to be written 50-100 years too early.

But maybe I'm too cynical. Who knows?

Of course, we never actually heard the Luddite argument, because 17th century Parliament[0] responded to the Luddites with reprisals, censorship, cruel and unusual punishment, and most importantly, propaganda. When you heard the word "Luddite" you probably thought I was accusing AI Snake Oil of being anti-technology. The reality is that the Luddites just wanted looms for themselves, because there were already structured phase-in periods for looms to ensure that skilled weavers got to buy them first. Smashing looms was a tactic to get business owners back to the table, and the business owners responded by propagandizing them as angry technophobes standing in the way of progress so that everyone would have to pay more for clothes.

Oh, and also by smashing heads.

> One way to do right by artists would be to tax AI companies and use it to increase funding for the arts.

This is not a bad idea, if you want to live in a world where everyone has fair access to generative art models but we don't decide to turn human artists into 17th century weavers[1]. However, I don't think this proposal would go down well with artists. Remember when the EFF suggested that, instead of the RIAA suing individual pirates, we should just have a private copying levy on all Internet service to remunerate artists? Yeah, no, that was never going to happen.

The reason why this was considered a non-starter is simple: copyright isn't a means to move money from readers to writers' pockets in aggregate, but individually. Nobody wants to be paid out of a pot, and that's why you also see a lot of outrage from artists over Spotify, because it's an effective revenue cap that screws over midlist artists. AI copying levies would further obfuscate attribution, because there currently isn't a way to determine how much value a particular training example provided to a particular generation prompt. And artistic endeavors vary greatly in terms of both quality and, especially, market value. An AI copying levy would flatten this down to just "whoever has the most art in the system wins". In other words, artists are currently playing Monopoly, and you're suggesting they play Ludo[2] instead.

[0] An institution largely consisting of rich nobility fighting a cold civil war against the English crown

[1] The average /r/stablediffusion commenter would disagree.

[2] US readers: Sorry / Trouble. I don't know how the same game got renamed twice to two different things.

Reference? A quick skim of Wikipedia and other articles doesn’t mention this.

I'm not convinced as to why one risk is "real" and the other is not. If chatbots want to leak their creators' confidential details, that's up to them and their programmers. They already have their commercial incentives to patch those issues.

Jesus, can people please relearn how to debate without these weird underhanded tactics?

It's like you think "everyone knows that the ASI thing is bullshit, they're just saying it to distract from algorithmic bias and corporate/military AI worries." I assure you that people who think ASI/UFAI is a real danger do exist. (Hi.)

I can't say that I am not hesitant, but this open letter made me think we might be on the right track ( personal AI that is private to you ).

That said, I think OpenAI is being unfairly targeted. OpenAI has done more to democratize access to AI algorithms than any other company in the history of tech. They simply provided access in a straightforward and transparent way with transparent pricing and APIs.

Google and other competitors sat on this and acted as gatekeepers and here again we have an attempt to gatekeep what people are allowed to use and tune.

It's only through usage that we can get the data to fix the core issues like bias. We can't expect a select few to solve all of AI's potential problems prior to the rest of humanity being "allowed" to benefit.

The fact is that AIs are more likely to be accomplices than masterminds, at least for the foreseeable future. What we are afraid of is that they will end up being just as terrible and flawed as we are, but it's more likely that they will do what they do today: what we ask them to do. The greater risk is therefore from malicious internal users rather than from the technology itself.

Perhaps more to the point, there is no effective way to limit the output of an LLM or place restraints on how they work. I think it's foolish to even attempt that -- they just don't work that way. The better way to regulate this kind of technology is to focus on the people. Some data classification and licensing program seems a lot easier to implement than the road we currently seem to be going down; which is either no regulation or insanely restrictive regulation.

I agree with your take for the most part, however cautiously more optimistic about this. Removing the barrier of entry for most thing will lead to more people be able to contribute in professions that they otherwise would not have the know how to do.

This will lead to more good things than bad things I think, but we seemingly only focus on the bad extremes and assuming everything else is the same.

Modern Javascript has reduced the barrier of entry to software development. It has had an massive impact on the industry and with an influx of people which leads to the industry seeing more and more creative solutions, as well as some really wild ones.

Maybe I'm terribly naive, but I think we seriously underestimates how many people wants change -- just missing the push to spring into action. For some the moment is about to arrive, for others it will come later.

Paradoxically, a heavy-handed approach to regulation could guarantee a bleak future for humanity. We risk increasing the incentives for malicious use at the same time as we make legitimate use prohibitively expensive / regulated out of existance. If the war and crime machine is cheap and easy to monetize, don't be surprised when that's what gets built.

The future could be unimaginably better for everyone, but we won't realize that future unless we get the regulation right.

https://paperswithcode.com/paper/most-language-models-can-be...

The thing is, the speed at which AI is progressing makes it not really long term at all, we need to use junk about these existential risks now (really, we should have tried to solve this 20+ years ago)

Access to AI is a new frontier in inequality.

What fashion trend? These tools are less powerful than you think.

I bet that it's already a use case of tools like ChatGPT

We might be able to remediate this somewhat by designing AIs that are purposely below some threshold of reliability. Their current tendency to hallucinate will discourage over reliance if we can't or don't fix it.

It's a little like the distinctive smell that is added to propane to make it easier to detect leaks. By adding, or not removing, easily detectable hallucinations from AI, it's easier to detect that the source must be checked.

We desperately want oracles and will quickly latch on to highly unreliable sources at the drop of a hat, to judge by various religious, political and economic trends. It's inevitable that many will turn AIs into authorities rather than tools as soon as they can justify it. We could delay that by making it less justifiable.

Indeed. This was true even for the primitive AI/chatbots that existed in the sixties: https://en.wikipedia.org/wiki/ELIZA_effect#Origin

And after training, they're fine-tuned to generate specific lies.

I don't disagree that these tools and services ought to be regulated and agree that disinformation about their capabilities; real and speculated can be counter-productive.

Other real risks: fraud and scams. Call centre employees typically have to verify the person on the other side of the call. This is going to get less reliable with models that can impersonate voices and generate plausible sounding conversation. Combined with the ability to generate fake social media accounts, etc; social engineering is going off.

From a regulators' perspective we need to know that the companies providing services are doing everything they can to prevent such abuses which requires them to be open and to have a framework in place for practices that prevent abuse.

Just keeping up with the hype train around this is exhausting... I don't know how we expect society to keep up if we're all allowed to release whatever technology we want without anyone's permission regardless of harm: real or speculative.

We should probably focus on the real harm. Especially the hidden harms on exploited workers in the global south, the rising energy and compute infrastructure costs, etc.

The key risk highlighted here which I have not seen as much talk about is the way that these technologies might give shift power from white-collar labor to capital in a drastic way. The ability to build the AI products that people will actually use on a day to day basis seems like something established companies with lots of customer data will be hard to compete with. For example, its pretty trivial for Salesforce to plug a LLM into their products and get pretty good results off the bat.

and before tech bros clap for "IP as a concept should be destroyed anyway", keep in mind that this applies to your favorite free and open source things as well, the free and open licenses, the CC licenses, that also get ignored and just put into the AI blender and spat out "brand new" with whatever "license" that AI project decides to slap on it. it is actually a huge problem if projects decide to just ignore licenses, even those that are meant to be there just to share stuff for free with people while preserving attribution, and just do whatever they want with any work they want.

In general, how does one put "business rule"-type constraints on the output of models? Is there any way to bake-in prime directives?

is wrong.

It is consistently hubristic, and variously disingenuous or bad faith, naive, or millenarian.

Why is it wrong?

Because no one, not Altman nor Yudkowsky nor Musk nor Gates nor Bostrom nor anyone else, knows what the impacts are going to be.

We have not since the advent of the internet experienced the emergent introduction of a new technological force-multiplier and agency-augmenter like this; and this one by virtue of where we are courtesy Moore's Law etc. fully exploits and realizes the potential of the preceding ones. We built a highly-networked highly-computational open society resting on surveillance, big data, logistics, and the rapid flow, processing, and transformation, of inordinate amounts of data.

And now we are cranking things up one more notch.

Those of us who lived through the internet's arrival know something that those who grew up with it do not, which is the actual literal ordering of things—of most aspects of shared society—can and will be upended; and it is not just industries and methods of communicating and doing business that change. Our conception of self and presumptions of what it means to be ourselves and with one another and how that happens, all change.

Per the slow march to singularity the last revolutions have reliably transpired an order or more faster than those before them.

This one looks to be no different. The rate of change telegraphed by e.g. this forum and Reddit, viz. individual novel possibilities being exploited on a daily basis, makes that clear enough.

So the only thing any of us, no matter how silver-backed, grey-bearded, wealthy, or embedded in the AI industry itself,

is that none of us know what is going to happen.

The surface across which black-swan events and disruption may occur is simply too large, the number of actors too great, the consequent knock-on effects too numerous.

The only thing we can say is that none of us know where this is going.

Well, that—and that it's happening at a rate beyond institutional or governmental control.

The only things that could stop radical disequilibrium now are deux ex machina intervention by Other Powers, even more disruptive climatological tipping points, or, untracked large asteroids.

Beware anyone who claims to know what is happening, why. For one or another reason, they are speaking falsely.

> CNET used an automated tool to draft 77 news articles with financial advice. They later found errors in 41 of the 77 articles.

How many other news outlets are using AI and don't know or or don't care about the errors?

I'm far from an AI doomer, but it seems incredibly irresponsible to worry about only the risks that are provably happening right this second.

Imagine dealing with customer service when any human you reach is using a LLM to generate 50x more responses than they currently can with no more ability to improve the situation. They will all 'sound' like they are helpful and human, but provide zero possibility of actual help. 'Jailbreaking' to get human attention will remain important.

Now imagine that supplied to government bureaucracies, investigative, and military what is developed for corporate.

> The letter refers to a common claim: LLMs will lead to a flood of propaganda since they give malicious actors the tools to automate the creation of disinformation. But as we've argued, creating disinformation is not enough to spread it. Distributing disinformation is the hard part. Open-source LLMs powerful enough to generate disinformation have also been around for a while; we haven't seen prominent uses of these LLMs for spreading disinfo.

I expect we are seeing LLMs spreading disinfo already, and that they're going to ramp up soon, we just don't know about it because those spreading disinfo would prefer to do it quietly and refine their methods and don't announce to the world that they're spreading disinfo.

It is also most likely happening domestically (in the US) which is something that we don't hear much about at all (it is all "look over there at the Russian Troll Farms").

Today's 'future shock' over conversant AI is just the leading edge of discovering another form of semi-human intelligence in the world. We'll adapt to it fairly soon, and realize that it's actually less interesting than an intelligent child. It's a fake human. Once we 'grok' that, we'll begin to look past it as a mere robot, just a tool. It will become a utility, a service, but not a friend. It's not human and it never will be.

Who wants to play for long with a robot?