It’s signal strength and signal clarity, both on a scale of 1 to 5.

“you’re 5-by-5” thus means " you’re loud and clear."

Edit: There is also a famous line in Aliens where a pilot says “we’re in the pipe, five-by-five”. I think it’s also said in buffy.