You don't need NaN (or Inf) to have unexpected behaviour with floats and naive loops in lua, using 1e52 is enough. Conflating floating point and integers (independent of encoding) is always a bit of a trap.
⇒ I expect this choice got careful consideration. (Edit: in the end, the reasoning still could have been “unfortunately, existing implementations do it this way; we’ll begrudgingly follow them”)
Indeed, and makes you realise that NaN is not "not a number" but rather "not a particular number", if one cathetus is infinite then the hypotenuse must be infinite whatever the other cathetus is ...
⇒ I expect this choice got careful consideration. (Edit: in the end, the reasoning still could have been “unfortunately, existing implementations do it this way; we’ll begrudgingly follow them”)