I'm not into it, true. But my logic stems from my opinion that if a boxer can't beat the heavyweight, he's not the best boxer ever. If you're not the best boxer ever, how can you be the best athlete ever?
For example, I don't see how Tyson or Ali would lose to someone in lower weight categories, when they were all in their prime.
It's all a matter of opinion though, I know my "logic" might be a bit unfair. If you regard skill and pound for pound as more important, then sure. It's sort of like why a right back will never be the best player of all time. It's always an attacker, like Maradona, Ronaldo, Pele, etc...