Men in average are just 10-12cm taller than women in average, so around 7% difference on mean. Of course that a lot of women are taller than a lot of men, in pretty much every setting, so not sure what is your argument.
In NBA/WNBA to be fair, the difference is 6 inches, which is higher than in the average population (again, this makes sense, if you shift the mean, the outliers will shift more).
And we accept that fact because it's visible, on average men are significantly taller than women, 10 to 12 cm is significant. Now how many women are actually out of the norm when it comes to their biology, what if it is the same proportion than with height? We accept that a lot of women secretes abnormal amounts of growth hormones, why the same couldn't be true for other hormones?