Metacognition, known unknowns, and emergence of reflective identity
Paul Watson asks:
Will decent AI gain a sense of identity, e.g., by realizing what it knows and does not know? And, perhaps valuing the former, and maybe (optimistically?) developing a sense of wonder in connection with the latter; such wonder could lead to intrinsic desire to preserve conditions enabling continued learning? Anyway, answer is Yes, I think, as I tried arguing last night.
Paul recommends this article about macaque metacognition may be relevant.
Relatedly, Paul says
- A search for knowledge cannot proceed without a sense of what is known and unknown by the “self.” Must reach outside self for most new knowledge. Can create new knowledge internally too, once you have a rich model of reality, but good to know here too that you are creating new associations inside yourself, and question whether new outside knowledge should be sought to test tentative internally-generated conclusions.
- Self / Other is perhaps the most basic ontological category. Bacteria have it. Anything with a semipermeable membrane around it — a “filter.” Cannot seek knowledge without having at least an implicit sense that one is searching for information outside oneself. In highly intelligent being, how long would that sense remain merely implicit.
What do you think?