Going over it, I think that sentience (not necessarily sapience, but that would be nice) is the secret sauce. In order for me to accept that an AI knows something (as opposed to posessing data which it does not actually understand), it has to demonstrate awareness.

So how can a text-based AI demonstrate awareness, given the constraints of the interface through which it must operate? Reliably generalizing from data not immediately part of the response to the current prompt might do it. Or demonstrating that it understands the consequences of its actions in the real world. Even just indicating that it knows when it’s making things up would be a good start.

For instance, take the case of the ChatGPT-generated fake legal citations. An AI which would have been fed masses of information relating to law (I’d expect that to include law school textbooks, from archive.org if nowhere else) demonstrated very clearly that it did not know that making up legal cases in response to a factual query was a Very Bad Idea. It did not generalize from data outside the domain of lists of case names that would have told it not to do that, or provide any indication that it knew its actions could be harmful. That AI had data, but not knowledge.

So we’re back to connections and conceptual models of the world again.