Version 1 of #5939

… because it could always in principle have been produced by a nonconscious program.

I don’t think that’s right. There could be animal behavior I’d have serious trouble explaining.

People asked me about this at Conjecture Con. For example, if we took an animal out of the context in which it evolved, and put it in a new context it didn’t evolve for – that’s where animals bug out – but instead of bugging out, it behaved more like a human in an unknown situation and became reflective, actually seemed to understand what was going on, that would at least give me pause.

I can’t be more specific than that because it would depend in part on new knowledge the animal would have to create in that situation, and by definition I can’t predict new knowledge. But I’d know it when I see it.

That may not be completely satisfying, but it shows that it’s simply not true that any animal behavior “could always in principle have been produced by a nonconscious program.”