This quote to me feels more along the "no computer will ever need more than 640kb of RAM" lines in terms of historical accuracy. Like whoops, nope.

Well... we have needed to put a tonne of work into engineering safer outcomes for behavior generated by natural general intelligence, so...

> what would you answer if I asked you this question?

I don't know.

To be fair, it does sound like a pretty tough place to stomach.

> They shared the fondness for torture of accused criminals with the ancient Romans.

Well, with their "hostage justice" using torture tactics to extract false confessions relied on in court and their actual prisons where you have to work 8 hours a day and aren't allowed to say a single word while you do, I guess modern day Japan still has a thing for torturing accused criminals.

> We might be incentivizing answers that sound right with reinforcement learning as opposed to answers that are actually right.

We do this with other humans, so I don't know that we know how to avoid doing the same with machines.

Only problem is, in the real world, always being truthful isn't the thing that will maximize your reward function.

> AI can't have accountability

Sure it can. You just have to bake into the reward function "if you do the wrong thing, people will stop using you, therefore you need to avoid the wrong thing".

Then you wind up at self-preservation and all the wholly shady shit that comes along with it.

I think the AI accountability problem is the crux of the "last-mile" problem in AI, and I don't think you can necessarily solve it without solving it in a way that produces results you don't want.

I don't want to get into a semantics argument but that's not accountability. That's just one more behavior prompt/indication but you can't fire the LLM. It might still do the wrong thing.

Yup. And this is why I think the "last mile" problem in AI is basically unsolvable.

Enshittification