I know that the task was rejudged.
I added the condition for new test, where A, B and C collinear: cos(ACB) - 1.0 < eps, where eps = 1e-8, then answer = AB, but I've still get WA 48. Give me a hint, please.
Thanks in advance.

Now already accepted, I have found stupid bug in my code.
Ignore first message.

And what bug did you found? Just like you I've added a condition for test:
2 4 0
3 6 0
0 0 0
3
The answer is not NaN now, but I still have WA48.

Try cases when all points belongs to one axis only:

Case 1:
6 0 0
2 0 0
4 0 0
1

Case 2:
0 6 0
0 2 0
0 4 0
1

Case 3:
0 0 6
0 0 2
0 0 4
1

For all cases correct answer will be 4.51

Good luck!