( TEN Minute READ )

The vast sums of money and compute power pouring into AI are accelerating its advance at a pace beyond even the ambitious imagination of its own innovators.

Driven by what the techies call “verified rewards,” these models relentlessly seek achievement of an objective and will do whatever is necessary to get there.

AI scientists have long feared “where the AI system is sufficiently capable that, whatever its objectives are, it’s going to achieve them, even if they’re not aligned with what we want.

These are not sharp transitions, but we could talk about a ‘loss-of-control transition,’ where we no longer have a say in what happens.”

The possible scenarios beyond this threshold range from disruptive cyberattacks on infrastructure to, at the far end, “extinction” in the sense that autonomously reasoning and agentic AGI no longer needs humans or to align with their morals, norms and interests.

if we figure out how to build in safety into the design of AI systems from the beginning, maybe we could coexist indefinitely and even flourish with such systems

So, before that loss-of-control transition happens, there’s an earlier transition, which is much more difficult to perceive, which is when the time it takes to get to that loss-of-control level is less than the time it takes to solve the control problem.

I say that “ we have already passed that point.”

Solving the control problem is very difficult.

Because this “race condition” amplifies the mismatch between devising effective constraints and losing control.

The obvious question is why the AI companies are risking even a small chance of their invention leading to human extinction by proceeding when they are fully aware they are losing control?

————————-

It’s this mismatch between the long-term interest, which is presumably that the species continues, and the short-term reward signal that evolution has built into you to try to get you to do good things.

But we humans also suffer from this when we become drug addicts, right?

We have a dopamine system that’s supposed to help us avoid pain and seek pleasure and food and company and all those things that we like.

But sometimes it gets hijacked by drugs.

And so we experience a personal extinction as a result.

So that mismatch can happen at the species level as well.

If we succeed in creating AGI — which we are going to spend a trillion dollars of your money to build — then there’s, depending on who you ask, 10, 20, 25, even 50% chance that we’re all going to go extinct.

I think this is really quite terrified, but they can’t get out of the race condition that they’re in.

OK,

Wr get the big Techology companies to stop.

But they will only stop if everyone else agrees to stop.

That has never happened, as far as I know, in the history of capitalism.

Reasoning comes in many technically forms, but the basic procedure is easily recognizable: arriving at a sound conclusion by linking together intermediate steps that logically follow from each other. 

How do we account for it?

All human comments appreciated. All like clicks and abuse chucked in the bin.

Contact: bobdillon33@gmail.com