An AI which would have been fed masses of information relating to law (I’d expect that to include law school textbooks, from archive.org if nowhere else) demonstrated very clearly that it did not know that making up legal cases in response to a factual query was a Very Bad Idea. It did not generalize from data outside the domain of lists of case names that would have told it not to do that, or provide any indication that it knew its actions could be harmful.

I mean, was it a bad idea? For the lawyer sure, but ChatGPT was not penalised by it’s own cost function. It may well have known in some way that is was just guessing, and that generally a legal document is serious business, but it doesn’t have any reason to care unless we build one in. Alignment is a whole other dimension to intelligence.

Reliably generalizing from data not immediately part of the response to the current prompt might do it. Or demonstrating that it understands the consequences of its actions in the real world.

It sounds like the biggest models do this reasonably well. Commonsense reasoning would count, right?