Could we understand what a Superintelligence is up to?

Obviously no, it's a super intelligence, of course we can’t understand it!

Yet it’s also the case that we humans are fully general; there’s a sense in which we can understand anything that could be understood. David Deutsch calls us universal explainers. Universal implies there’s nothing higher, it’s not a spectrum. 

This proposition is at least intuitive to me. Daniel Dennett once said, and I paraphrase poorly here, that if this wasn’t the case “then something weird must be true, as though there’s no series of books which can bring us to an appreciation of the answer”. He was responding to the claim that certain answers are forever out of our cognitive horizons.

It may be that we can come to an appreciation of Superintelligent thoughts. But serious understanding, like that of a rocket engineer who’s sure that the rocket will not blow up, is likely impossible. It will be like trying to understand how a computer works. With enough time and patient learning, we can perhaps come to an appreciation of it, but even after the entire study we might struggle to reason through the cause of a firmware bug, we might miss an unforeseen implication, or a cascading failure condition. 

Except in the case of a Superintelligence mind, the matter is so much worse.  Artifacts of that order will be created so frequently and at such volume, that as we make halfway progress on a sub-module within a sub-module twenty levels deep, there’s been a tenfold increase in the amount of material to make sense of.

Even if time wasn’t an issue, there’s no guarantee that it can be logically partitioned into a hierarchy of abstractions, with each layer being simple enough to fit into our puny little universal brains. Engineering, by humans, is only possible because our systems can be sliced into piecemeals parts. If it’s a big tangled mess of highly coupled parts, then comprehension will become a working-memory-bound problem, with our minds being completely outmatched by the scale. This, incidentally, poses no problem for a Superintelligence, which can fit everything in its context window of fifty terabytes without losing track of all the causal chains. 

All of this still assumes that it’s writing in tokens that are meaningful to us, which, of course, is on a timer. At some point it will go from a convoluted thick web of interconnections to pure gibberish. 

Worst of all, even if it writes in English or C, with neat human-friendly abstractions, it could design a machine that exploits a new law of physics that’s unknown to us, so that, even if we’re given the full blueprint of this machine and can inspect every inch of it, when you switch it on and it starts opening up a wormhole, we will be at a complete loss for how it works.

It’s like sending the instructions for assembling a refrigerator back a thousand years, and seeing that even the smartest minds of that era will notice the cold air coming out and think it’s magic. Smart as they were, they just didn’t know that compressed air, when expanded, cools down. 

In the presence of a Superintelligence, it means close to nothing that we’re universal explainers that can understand anything.