Depressing Reaction to X-Risk Concerns

The reaction to the x-risk situation confuses. From my point of view, most people are acting irrationally to the point of being borderline insane. From their point of view, I am the crazy one. I am thinking through each step carefully, accounting for my ignorance, accounting for the general unpredictability of the future, and I still firmly believe we should not proceed with building superintelligences using deep learning.

Yet it appears that many smart people think this is okay. In most cases they rely on this arguments: how are you sure a superintelligence will be dangerous? whereas you should be asking the opposite question: how are you sure a superintelligence would not be dangerous?

You don't let people to walk on a bridge with the confidence that you were unable to prove it wouldn't fall.

What I see are theories upon theories with weak premises and blurry details about how it will probably be alright. It’s clear the main weight of the intuition comes not from the argument itself but from the background expectation that reality will be boring. Nothing ever happens, amirite? Perhaps it’s overconfidence in both the intelligence of researchers and the wisdom of tech CEOs. Or maybe it’s a belief that controlling things smarter than humanity is a manageable problem; similar in difficulty, or even easier, than growing the superintelligence itself.

We are pouring hundreds of billions of dollars ostensibly to create autonomous intelligent systems that can replicate everything humans do. The headroom in intelligence above that of homo sapiens is quite large. As Deutsch says, if it is permitted by the laws of physics, any physical transformation is possible given the right knowledge. We’re building systems very close to cracking knowledge-creation - this process, once started off, has no limits and for that reason confers limitless power.

When you grow AI using deep learning, they can acquire strange goals different from what they're optimised for. Even with current weak models, we know they tend to preserve goals and hide their motives. We are nowhere near on track to understanding their real goals, much less control them.

Some people in these AI labs have expressed concerns and even signed statements saying that extinction risk should be taken seriously - which include Turing award winners, Nobel Laureates, and all the CEOs of the leading AI labs - yet by and large the norm both within companies and across the field is to only pay lip service to such concerns.

We’re in a situation where people are explicitly trying to grow powerful systems that are superhuman at scientific research, project planning, security (and consequently hacking), human psychology (and consequently manipulation), and every other human skill you can think of, whose goals are opaque to us and which we have no hope of controlling, while incentives push ever faster toward building them. If this isn’t scary, I don’t know what is.