Agreed. If we cannot get an AGI takeoff that can get 99% "extrapolated buy-in" ("would consider acceptable if they fully understood the outcome presented"), we should not do it at all. (Why 99%? Some fraction of humanity just has interests that are fundamentally at odds with everybody else's flourishing. Ie. for instance, the Singularity will in at least some way be a bad thing for a person who only cares about inflicting pain on the unwilling. I don't care about them though.)

In my personal opinion, there are moral systems that nearly all of humanity can truly get on board with. For instance, I believe Eliezer has raised the idea of a guardian: an ASI that does nothing but forcibly prevent the ascension of other ASI that do not have broad and legitimate approval. Almost no human genuinely wants all humans to die.

While I understand the risks (extinction among them) I also think these discussions ignore the fact that some kind of utopian, starfaring civilization is equally within reach if you accept the premise that takeoff is so risky. Personally, I’m very worried about the possibility of stagnation arising from our caution, because we don’t live in a very nice world with very nice lives. Humans suffer and scrape by only to die after a few decades. If we have a, say, 5% chance of going extinct or suffering some other horrible outcome, and a 95% chance of the utopia, I don’t mind us gambling to try to achieve better lives. To be fair, we dont even have the capacity to guess at the odds yet, which we probably need to have an idea of before we build an agi.

Gambling on the odds we all die for the chance at a "utopian starfaring civilization" seems liek the sort of thing that everyone should get a say in, and not just OpenAI or techies.

People shouldn't be able to block others developing useful technologies just based on some scifi movie fears.

Just like people shouldn't be able to vote to lock up or kill someone just because - people have rights and others can't just vote the rights away because they feel so.

> People shouldn't be able to block others developing useful technologies just based on some scifi movie fears.

The GP was suggesting we have to develop AI because of scifi movie visions of spacefaring utopia, which if anything is more ludicrous.

I personally don't believe in AI "takeoff", or the singularity, or whatever. But if you do, AI is not a "useful technology." It's something that radically impacts every single life on Earth and takes our "rights" and our fate totally out of everyone's hands. The argument is about whether anyone has the right to remove all our rights by developing AGI.

Both are unlikely but only one of the sides is arguing for limits/regulations of actually useful technology because they saw Terminator.

It seems strange we're allowed to argue for a technology because we read Culture and not against it because we saw Terminator.

Nevertheless, the goal of OpenAI and other organizations is to develop AGI and to deliberately cause the Singularity. You don't have to have watched Terminator to think (assuming it is possible) introducing a superpowered alien intellect to the world is a extremely risky idea. It's prima facie so.

I am against all regulation of LLMs. "AI safety" for what we currently call "AI" is just a power grab to consolidate and solidify the position of existing players via government regulation. At any rate nobody seems to be arguing this because they saw Terminator, but that they don't like the idea of people who aren't like them being able to use these tools. The "danger" they always discuss is stuff like "those people could more easily produce propaganda."

As a doomer who is pro-LLM regulation, let me note that the "people could produce propaganda" folk don't speak for me and that I am actually serious about LLMs posing a danger in the "break out of the datacenter and start making paperclips" way, and that I find it depressing that those folks have become the face of safety. Yes I am serious, yes I know how LLMs work, no I don't agree that means they can't be agentic, no I don't think GPT-4 is dangerous but GPT-5 might be if you give it just the right prompt.

(And that's why we should rename it to "AI notkilleveryoneism"...)

I get this point, but I just don't see us anywhere near technology that warrants this level of concern. The most advanced technology can't write 30 lines of coherent Go for me using billions of dollars in hardware. Sure, more compute will help it write more bullshit faster, and possibly tell better lies, but it's not going to make it sentient. There's a fundamental technological problem that differentiates what we have and intelligence. And until there's some solution for that I'm not really worried. To me it looks like a bunch of hype and marketing over a neat card trick.

I'm really confused about this. I've been using GPT-4 for coding for months now and it's immensely useful. Sure it makes mistakes; I also make mistakes. Its mistakes are different from my mistakes. It just feels like it's very very close to being able to close the loop and self-correct incrementally, and once that happens we're dancing on the edge of takeoff.

It seems like we're in a situation of "it has the skills but it cannot learn how to reliably invoke them." I just don't think that's a safe place to stand.

I'll try to clarify and maybe you'll see what I'm getting at. If the code you're writing was written and posted a million times online then ChatGPT is great at regurgitating that and even applying it to more specific applications. No argument. That's pretty cool. It can seem very surreal and give a lot of magic to the card trick.

But try this; take a sample of code from the Gio package for Go. I like this example because there's not a lot of published examples for this. The machine would actually have to "think" to accomplish basic work and obviously that's not what it does.

Take some example code from their tutorial and just ask ChatGPT 4 to do something simple like change the background black.

In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors.

The two attempts that did compile; one was a blank white canvas with nothing and the other didn't change anything, it looked exactly like the tutorial example.

Something else you can try; tell it to generate a logo and give it some big words for the company name, like "corporation." Watch how it can't even spell. Eventually it will admit that it's not able to do it, which I can only guess was a manual patch to save on operating costs so that you don't keep trying.

In short, this is not "intelligence" technology. That's a marketing term. It doesn't do anything remotely close to that and there is no clear path from this technology to that technology. It's just not in the same realm.

Maybe the doomsayers have seen some tech that I'm not familiar with, but I am not persuaded by ChatGPT in its current state. I think it will be another tech revolution or maybe two or three before AI leaves the realm of Sci Fi and enters the realm of even theoretical possibility.

This tech is machine learning, which is a creative way of saying "near real time statistics." That is really cool in and of itself. But it has nothing to do with "intelligence."

Hm. I've not had any problems using GPT4 with code that it has never seen. But sure, I'll try that...

Edit: I took the GIO Hello World sample. I fed it into the GPT-4 API.

And I swear to God, first try, first click:

https://gist.github.com/FeepingCreature/bde583aa8a2e8e5e1c56... Near-perfect success. One extra import, sure, I'll give it that one, LLMs can't correct when they notice they don't actually need an import in the code. But then it works.

Gist link cause I use OpenRouter against the GPT-4 API, maybe that's why? Maybe the "stochastic parrot" vs "oncoming apocalypse" perspective genuinely represents that ..... GPT just doesn't work for some people?? Did they fuck the ChatGPT4 web interface enough somehow so that it's just inept, but somehow not affect the API?

Edit:

> In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors.

Important note: there's a theory that if you've got it making mistakes a few times, it thinks that it's playing "the sort of AI that fucks up a lot" and starts making more mistakes, reinforcing its role. Start over fresh if it seems strangely incapable. Like, it is absolutely possible to use GPT-4 in a way that results in it systematically being incapable of the most basic tasks. These things are not reliably competent; but they're occasionally competent! In my experience, even with completely novel tasks in completely novel environments, the competence is "in there" and can often be elicited with dedicated poking. That's why I think LLMs are enough for a takeoff with enough scale (and basic online learning probably): in my opinion, it's not a matter of attaining the skills but removing the roadblocks.

It's like the joke about seeing an old man playing chess with a labrador, and the labrador just carefully picking up figures and moving them around the board, every time in a legal move, and you say "that's amazing, a chess-playing dog!" And the old man scoffs and says, "Nonsense. His endgame is rubbish." Three years ago we didn't think a dog could understand pawn promotion at all.

edit: Hey, share your repros? I'm genuinely curious what's going on here now.

edit:

> Watch how it can't even spell.

This is a specific issue with the current generation of LLMs and should be thought of more as a disorder than a fundamental inability. LLMs don't actually ever see letters, they see BPEs, which consist of one up to n letters. For instance, the word "corporation" looks to the AI more like "<27768>". So like- yes. It can not spell. It fundamentally, architecturally, cannot perceive letters in the words you give it; its ability to manually split words into letters is based on chance memorization. Instead, try getting it to split the word into letters in code.

I see. This is no longer a good example because of the additional functionality and documentation that's been added to the Gio package since my experiment took place. So, in this case, I can't show you behind the curtain without a lot more effort.

But even given your example, you've essentially told me about a parrot that says "four!" when someone says "two plus two." Even if the same parrot can recite the entire US constitution backwards and forwards, it doesn't actually understand what it's saying.

I just spent a few minutes on YouTube watching parrots. They can be very convincing with the right training. For me it's the same thing.

It's uncanny, but it's not mutiny. It's barely even in the direction of intelligence.

This just seems a bizarre argument to me. I've written entire (fully novel!) programs with these things. If this is a parrot, we should hire parrots for software development and worry about them being the next dominant species.

Like, sure it doesn't know things that have zero documentation. I'm not saying it's human-level yet! That's never been the argument!

Look, you pulled this example out. You said it doesn't even work 37/38 times. Do you have another example? I just really don't understand what the basis for this argument is. I've found the thing a capable programmer within its limits!

You got my curiosity going so I went for the experiment again. It turns out my example is still very much valid. It's still not able to do basic work with the Gio package despite great documentation from the Gio folks, if you ask me.

In this case, the LLM is struggling to version match.

https://chat.openai.com/share/d5e39e52-9140-4f74-9fef-04fc34...

I cut this one short because I'm not that interested to go 40 questions again. I have better things to do and I have already seen the outcome.

It's clear the LLM is working as intended, which is neither for programming, nor for problem solving. It's giving the next most probable response. That's truly great for repeating and reintegrating existing solutions. But repeating is not what's needed for the "kill all humans" scenario. That scenario requires outreasoning humans. Processing power is not remotely the issue. I've never seen a theoretical solution that purports to give a machine the ability to actually reason, and because of that, I don't believe that ChatGPT, or any other LLM, even begins to approach "intelligence."

If you are having success with it, I think it's because you're helping it more than you'd like to admit, or your solutions might not be as novel as you think. Probably the prior, but I must admit, I've wrongly thought I was working on new stuff many times before. I think that's more common than we think. Or at least I'd like to think that so I can feel good about myself.

In any case, I won't be hiring ChatGPT or parrots; and for the same reason. I need developers that can follow directions, solve problems in better ways than what is readily available, and do it faster than my competitors. And I'm not even in some cutting edge business, I'm just building integrations, specifically in the manufacturing and distribution industry. Even in something so mundane, I've hardly found ChatGPT to be truly useful for much more than a GitHub search tool, and to suggest cocktails when my liquor selection runs scarce.

Worth $20? Sure. But it's certainly not scaring me.

Again, nobody is saying that GPT-4 is scary or that GPT-4 can kill all humans. I'm saying it contains the skills necessary to write programs in novel domains.

Just today, I asked it to translate a demo to OpenGL4. It did this using a matrix lib it had probably not seen before, since I just wrote it a few years ago and there's like one file using it on Github, but its code mostly worked fine after I gave it a copy of the documentation to read. (I had to do some debugging, but its issues were in hindsight very understandable. Stuff like not knowing that a matrix is row-major, which is an admittedly unusual layout.)

And I had to ask for some specific corrections after, sure, but its responses were perfectly correct. ("This shader doesn't seem to be handling alpha blending with discard." "Oh yeah sure here's how you do that.")

Yes, if you keep asking it for corrections its performance will continue to degrade. This is a known problem. Yes, it isn't good at incremental problem-solving. Again, known issue. And yes - it's not yet human level. Nobody's saying it's a replacement for a human programmer, as it stands.

It seems like you're saying "it can't think as good as a human, therefore it can't think." There are skill levels below human but above parrot! And I'd argue this thing as it stands is much closer to human than parrot already.

That said, looking at your challenge:

> create a graphical application in Go using the Gio package. It should have two buttons down the left column and a main view area where graphics, text and input can be received from the user.

I couldn't do that. What the hell does "a main view area where graphics, text and input can be received from the user" mean?? Do you want a text field? With mouse paint support? What input is there beyond graphics and text? Like, GPT-4 is trained to just do what you tell it to, so it can't incrementally gather additional requirements, so you do actually have to be really specific in what you tell it to do. If you offered me that as a spec, I wouldn't sign it. If my company had already signed it, I'd be desperately trying to phone your designer to ask what they were smoking. This is just not a good requirements document. Garbage in...

Look, prompting a LLM is a skill. In this year 2024, we are not yet at the point where you can just throw vague requirements at a LLM and get constructive dialogue out eventually culminating in working code. But I just think it's silly to say "this model doesn't have the complete skillset of a fullstack developer and designer, therefore it has no skill at all."

You're right. I had a typo.

"a main view area with graphics, text and where input can be received from the user."

But notice how you can look at that and try to reason through what I meant, realize it's not clear, and respond to that.

The above is reasoning.

"The capacity for logical, rational, and analytic thought; intelligence."

ChatGPT can't do that and I don't mean that it can't do it good yet. I mean it's literally not doing that. There is nothing to advance or to expect advancement in. The "AI" doesn't exist. LLMs are simply not AIs and they are not in the path to AI, although they likely will play a major role if a theory towards AI were to be conceived.

It can do it, it just isn't trained for it. This one is actually kind of depressing. It's genuinely that ... as far as I'm able to tell, this sort of reasoning just doesn't appear in the finetuning set that they feed it to make instruct models. The way we train these things is honestly terrible in so many ways. Anyway, that's why I think the current mode of "well, I'll give it one attempt by charging blindly ahead, ignoring any mistakes I made and then blindly claiming success" is purely temporary.

In my experience, if you feed it a metric ton of explicit guidance you can occasionally get it into a mode where it reasons incrementally and actually notices when things are unclear. It is in there, it's just not foregrounded. They're trained not to ask questions, you see.

It's an interesting take. I don't see that it's in the technology. If you have a way to make this tech do that I'd say you're in a class of your own and you've got something no one else has. You should press it forward.

I don't know, I don't see these people you're talking about. It's always someone talking about world-ending AGI runaway that will take over your AWS instance, then AWS itself and then convert the solar system to a DC, or something.

To be fair, as somebody who thinks that, it's not like that's the plot of any particular movie. (Terminator went completely differently, for one.)

I think many AI safety advocates, me included, would readily take these odds.

We just think it currently looks more the other way around.