Call it a parasocial relationship, like what some people develop watching Streamers. Harry knows Moody somewhat, because Barty Crouch Junior apparently acted so well in this role that nobody noticed any difference.

The weirdness is acknowledged at the start of OotP:

‘Professor Moody?’ he said uncertainly.
‘I don’t know so much about “Professor”,’ growled the voice, ‘never got round to much teaching, did I? Get down here, we want to see you properly.’
Harry lowered his wand slightly but did not relax his grip on it, nor did he move. He had very good reason to be suspicious. He had recently spent nine months in what he had thought was Mad-Eye Moody’s company only to find out that it wasn’t Moody at all, but an impostor; an impostor, moreover, who had tried to kill Harry before being unmasked.

Mad-Eye Moody was now sitting at the kitchen table swigging from a hip flask, his magical eye spinning in all directions, taking in the Dursleys’ many labour-saving appliances.
‘This is Alastor Moody, Harry,’ Lupin continued, pointing towards Moody.
‘Yeah, I know,’ said Harry uncomfortably. It felt odd to be introduced to somebody he’d thought he’d known for a year.