Below is the text of my bet on Long Bets. Go there to vote.
"By 2050 no synthetic computer nor machine intelligence will have become truly self-aware (ie. will become conscious)."
(This summary includes my argument, a method for judging the outcome of this bet and some other thoughts on how to measure awareness...)
A. MY PERSPECTIVE...
Even if a computer passes the Turing Test it will not really be aware that it has passed the Turing Test. Even if a computer seems to be intelligent and can answer most questions as well as an intelligent, self-aware, human being, it will not really have a continuum of awareness, it will not really be aware of what it seems to "think" or "know," it will not have any experience of it's own reality or being. It will be nothing more than a fancy inanimate object, a clever machine, it will not be a truly sentient being.
Self-awareness is not the same thing as merely answering questions intelligently. Therefore even if you ask a computer if it is self-aware and it answers that it is self-aware and that it has passed the Turing Test, it will not really be self-aware or really know that it has passed the Turing Test.
As John Searle and others have pointed out, the Turing Test does not actually measure awareness, it just measures information processing---particularly the ability to follow rules or at least imitate a particular style of communication. In particular it measures the ability of a computer program to imitate humanlike dialogue, which is different than measuring awareness itself. Thus even if we succeed in creating good AI, we won't necessarily succeed in creating AA ("Artificial Awareness").
But why does this matter? Because ultimately, real awareness may be necessary to making an AI that is as intelligent as a human sentient being. However, since AA is theoretically impossible in my opinion, truly self-aware AI will never be created and thus no AI will ever be as intelligent as a human sentient being even if it manages to fool someone into thinking it is (and thus passing the Turing Test).
In my opinion, awareness is not an information process at all and will never be simulated or synthesized by any information process. Awareness cannot be measured by an information processing system, it can only be measured by awareness itself---something no formal information processing system can ever simulate or synthesize.
One might ask how it is that a human has awareness then? My answer is that awareness does not arise from the body or the brain, nor does it arise from any physical cause. Awareness is not in the body or the brain, but rather the body and the brain are in awareness. The situation is analagous to a dream, a simulation or virtual reality, such as that portrayed in the popular film "The Matrix."
We exist in the ultimate virtual reality. The medium of this virtual reality is awareness. That is to say that whatever appears to be happening "out there" or "within the mind" is happening within a unified, nondualistic field of awareness: both the "subject" and the "object" exist equally within this field and neither is the source of awareness.
This is similar to the case where we project ourselves as dream protagonists in our own dreams---even though our dream bodies appear to be different than other dream-images they are really equally dream appearances, they are no more fundamental than dream-objects. We identify with our dream-bodies out of habit and because it's practical because the stories that take place appear from the perspective of particular bodies. But just because this virtual reality is structured as if awareness is coming from within our heads, it does not mean that is actually the case. In fact, quite the opposite is taking place.
Awareness is not actually "in" the VR, the VR is "in" awareness. Things are exactly the opposite of how they appear. Of course this is just an analogy---for example, unlike the Matrix, the virtual reality we live in is not running on some giant computer somewhere and there is no other hidden force controlling it from behind the scenes. Awareness is the fabric of reality and there is nothing deeper, nothing creating it, it is not running on some cosmic computer, it comes out of of nowhere yet everything else comes out of it.
If we look for awareness we can't find anything to grasp, it is empty yet not a mere nothingness, it is an emptiness that is awake, creative, alert, radiant, self-realizing.
Awareness is empty and fundamental like space, but it goes beyond space for it is also lucid. If we look for space we don't find anything there. Nobody has ever touched or grasped space directly! But unlike space, awareness can at least be measured directly--it can measure itself, it knows its own nature.
Awareness is simply fundamental, a given, the underlying meta-reality in which everything appears. How did it come to be? That is unanswerable. What is it? That is unanswerable as well. But there is no doubt that awareness is taking place. Each sentient being has a direct and intimate experience of their own self-awareness.
Each of us experiences a virtual reality in which we and our world are projections. That which both projects these projections and experiences them is awareness. This is like saying that the VR inherently knows its own content. But in my opinion this knowing comes from outside the system, not from some construct that we can create inside it. So any awareness that arises comes from the transcendental nature of reality itself, not from our bodies, minds, or any physical system within a particular reality.
So is there one cosmic awareness out there that we are all a part of? Not exactly, there is not one awareness nor are there many awarenesses because awareness is not a physical thing and cannot be limited by such logical materialist extremes. After all if it is not graspable how can we say it is one or many or any other logical combination of one or many? All we can say is that we are it, whatever it is, and that we cannot explain it further. In being awareness, we are all equal, but we are clearly not the same. We are different projections and on a relative level we are each unique, even though on an ultimate level perhaps we are also unified by being projections within the same underlying continuum. Yet this continuum is fundamentally empty, impossible to locate or limit, and infinitely beyond the confines of any formal system or universe, so it cannot really be called a "thing" and thus we are not "many" or "one" in actuality, what we really are is totally beyond such dualistic distinctions.
Awareness is like space or reality, something so fundamental, so axiomatic, that it is impossible to prove, grasp or describe from "inside" the system using the formal logical tools of the system. Since nothing is beyond awareness, there is no outside, no way to ever gain a perspective on awareness that is not mediated by awareness itself.
Therefore there is no way to reduce awareness to anything deeper; there is no way to find anything more fundamental than awareness. But despite this awareness can be directly experienced, at least by itself.
That which is aware is self-aware. Self-awareness is the very nature of awareness. The self-awareness of awareness does not come from something else, it is inherent to awareness itself. Only awareness is capable of awareness. Nothing that is not aware can ever become aware.
This means awareness is truly fundamental, it has always been present everywhere. Awareness is inherent in the universe as the very basis of everything, it is not something anyone can synthesize and we cannot build a machine that can suddenly experience awareness.
Only beings who are aware already can ever experience awareness. The fact that we are aware now means that we were always aware, even before we were born! Otherwise we never could have become aware in the first place!
Each of us "is" awareness. The experience of being aware is unique and undeniable. It has its own particular nature, but this cannot be expressed it can only be known directly. There is no sentient being that is not aware. Furthermore, it would be a logical contradiction to claim that "I am not aware that I am aware" or "that I am aware that I am not aware" and thus if anyone claims they are not aware or have ever experienced, or can even imagine, there not being awareness they are lying. There is nobody who does not experience their own awareness, even if they don't recognize or admit that they experience it.
The experience of being self-aware is the unique experience of "being" --- an experience so basic that it is indescribable in terms of anything else --- something that no synthetic computer will ever have.
Eventually, it will be proved that no formal information processing system is capable of self-awareness and that thus formal computers cannot be self-aware in principle. This proof will use the abstract self-referential structure of self-awareness to establish that no formal computer can ever be self-aware.
Simply put, computers and computer programs cannot be truly self-referential: they always must refer to something else---there must at least be a set of fixed meta-rules that are not self-referential for a computer or program to work. Awareness is not like this however, awareness is perfectly self-referential without referring to anything else.
The question will then arise as to what self-awareness is and how it is possible. We will eventually conclude that systems that are self-aware are not formal systems and that awareness must be at least as fundamental as, or more fundamental than, space, time and energy.
Currently most scientists and non-scientists consider the physical world to be outside of awareness and independent of it. But considering that nobody has or will ever experience anything without awareness it is illogical to assume that anything is really outside of awareness. It is actually far more rational to assume that whatever arises or is experienced is inside awareness, and that nothing is outside of awareness. This assumption of everything being within awareness would actually be a more scientific, observation-based conclusion than the opposite assumption which is entirely unfounded on anything we have ever or will ever be able to observe. After all, we have never observed anything apart from awareness have we? Thus contrary to current beliefs, the onus is on scientists to prove that anything is outside of awareness, not the other way around!
Awareness is quite simply the ultimate primordial basic nature of reality itself---without awareness there could be no "objective reality" at all and no "subjective beings" to experience it. Awareness is completely transcendental, beyond all limitations and boundaries, outside of all possible systems. What hubris to think we can simply manufacture, or evolve, awareness with a pile of electrified silicon hardware and some software rules.
No matter how powerful the computer, no matter what it is made of, and no matter how sophisticated or emergent the software is, it will still never be aware or evolve awareness. No computer or machine intelligence will ever be aware. Even a quantum computer---if it is equivalent to a finite non-quantum computer at least---will not be capable of awareness, and even if it is a transinfinite computer I still have my doubts that it could ever be aware. Awareness is simply not an information process.
B. METHOD OF JUDGING THIS BET...
So the question ultimately is, how do we measure awareness or at least determine whether a computer is or is not aware? How can we judge the outcome of this bet?
I propose a method here: we let the bettors mutually agree on a judge. If the judge is a computer, fine. If the judge is a human, fine. But both bettors must agree on the judge. If both bettors accept that party as the judge then the result will be deemed final and reliable. If a computer is chosen by both parties to judge this, then I will concede defeat---but it would take a lot for any computer to convince me that it is aware and thus qualified to judge this competition. On the other hand, my opponent in this debate may accept a human judge---but obviously since they believe that computers can be aware if they accept a human judge they would be contradicting their own assertion---if a computer is really intelligent and aware why would they choose a human judge over a computer judge?
This "recursive" judge-selection judging approach appeals to our inherent direct human experience of awareness and the fact that we trust another aware sentient being more than an inaminate machine to judge whether or not something is aware. This may be the only practical solution to this problem: If both parties agree that a computer can judge and the computer says the other computer is aware, then so be it! If both parties agree that a human can judge and the human says that the computer is not aware, so be it! May the best judge win!
Now, as long as we're on the subject, how do we know that other humans, such as our potential human judge(s), are actually aware? I actually believe that self-awareness is detectable by other beings that are also aware, but not detectable by computers that are not aware.
C. A REVERSE TURING TEST FOR DETECTING AWARENESS IN A COMPUTER...
I propose a reversal of the Turing test for determining whether a computer is aware (and forgive me in advance if anyone else has already proposed this somewhere, I would be happy to give them credit).
Here is the test: Something is aware if whenever it is presented with a case where a human being and a synthetic machine intelligence are equally intelligent and capable of expression and interaction BUT not equally aware (the human is aware and the machine is not actually aware), then it can reliably and accurately figure out that the human being is really aware and the machine is not really aware.
I believe that only systems that are actually aware can correctly differentiate between two equally intelligent entities where one is sentient and the other just a simulation of sentience, given enough time and experience with those systems.
How can such a differentiation be made? Assuming the human and computer candidates are equally intelligent and interactive, what is the signature of awareness or lack of awareness? What difference is there that can be measured? In my opinion there is a particular, yet indescribable mutual recognition that takes place when I encounter another sentient being. I recognize their self-awareness with my own self-awareness. Think of it as the equivalent of a "network handshake" that occurs at a fundamental level between entities that are actually aware.
How is this recognition possible? Perhaps it is due to the fact that awareness, being inherently self-aware, is also inherently capable of recognizing awareness when it encounters it.
On another front, I actually have my doubts that any AI will ever be equally intelligent and interactive as a human sentient being. In particular I think this is not merely a matter of the difficulty of building such a complex computer, but rather it is a fundamental difference between machine cognition and the congition of a sentient being.
A human sentient being's mind transcends computation. Sentient cognition transcends the limits of formal computation, it is not equivalent to Turing Machine, it is much more powerful than that. We humans are not formal systems, we are not Turing Machines. Humans can think in a way that no computer will ever be able to match let alone imitate convincingly. We are able to transcend our own logics, our own belief systems, our own programs, we are able to enter and break out of loops at will, we are able to know inifinities, to do completely irrational, spontaneous and creative things. We are much closer to infinity than any finite state automaton can ever be. We are simply not computers, although we can sometimes think like them they cannot really think like us.
In any case, this may be "faith" but for now at least I am quite certain that I am aware and that other humans and animals are also aware but that machines, plants and other inanimate objects are not aware. I am certain that my awareness vastly transcends any machine intelligence that exists or ever will exist. I am certain that your awareness is just as transcendent as mine. Although I cannot prove that I am aware or that you are aware to you I am willing to state such on the basis of my own direct experience and I know that if you take a moment to meditate on your own self-awareness you will agree.
After all, we cannot prove the existence of space or time either---these are just ideas and even physics has not explained their origins nor can anyone even detect them directly, yet we both believe they exist, don't we?
Now if I claimed that a suitably complex computer simulation would someday suddenly contain real physical space and time that was indistinguishable in any way from the physical space and time outside the simulation---you would probably disagree. You would say that the only "real" space-time is actually not in the computer but containing the computer, and any space-time that appears within the computer simulation is but a mere lower-order imitation and nothing like the real space-time that contains the computer.
No simulation can ever be exactly the same as what it simulates, even if it is functionally similar or equivalent, for several reasons. On a purely information basis, it should be obvious that if simulation B is within something else called A, then for B to be exactly the same as A it must contain A and B and so on infinitely. At least if there is a finite amount of space and time to work with we simply cannot build anything like this, we cannot build a simulation that contains an exact simulation of itself without getting into an infinite regression. Beyond this, there is a difference in medium: In the case of machine intelligence the medium is physical space, time and energy---that is what machine intelligence is made of. In the case of human awareness the medium is awareness itself, something at least as fundamental as space-time-energy if not more fundamental. Although human sentience can perform intelligent cognition, using a brain for example, it is not a computer and it is not made of space-time-energy. Human sentience goes beyond the limits of space-time-energy and therefore beyond computers.
If someone builds a Turing Machine that simulates a Turing Machine simulating a Turing Machine, the simulation will never even start, let alone be useable! As the saying goes, it's Turtles All The Way Down! If you have a finite space and time, but an infinite initial condition, it takes forever to simply set up the simulation let alone to compute it.
This is the case with self-awareness as well: It is truly self-referential. No finite formal system can complete an infinitely self-referential process in finite time. We sentient beings can do this however. Whenever we realize our own awareness direclty---that is whenever we ARE aware (as opposed to just representing this fact as a thought) we are being infinitely self-referential in finite time. That must mean we are either able to do an infinite amount of computing in a finite amount of time, or we are not computing at all. Perhaps self-awareness just happens instantly and inherently rather than iteratively.
On a practical level as well we can see that there is a differnece between a simulated experience within a simulation and the actual reality it attempts to simulate that exists outside the simulation. For example, suppose I make a computer simulation of chocolate and a simulated person who can eat the chocolate. Even though that simulated person tastes the simulated chocolate, they do not really taste chocolate at all---they have no actual experience of what chocolate really tastes like to beings in reality (beings outside the simulation).
Even if there are an infinite number of levels of simulation above the virtual reality we are in now, awareness is always ultimately beyond them all---it is the ultimate highest-level of reality, there is nothing beyond it.
Thus even an infinitely high-end computer simulation of awareness will be nothing like actual awareness and will not convince a truly aware being that it is actually aware.