Yeah, but Nadal was only great on clay from 2005-08 and won his first non Clay Slam only in 2008. So Federer was unopposed during 2003-08 in the Oz, Wimbledon and US Opens.
Federer was dominant on grass during most of the 2000s. It wouldn't matter if it was Nadal, Djokovic, or Borg - he played some of his best tennis during that period. Saying he's not the best because Nadal wasn't at his peak yet in those years is as pointless a thought experiment as saying Borg is the GOAT because he would've won more slams than anyone if he was bothered to play the Aussie and not retire at 26. Ultimately, you can only look at actual results and not results that may have happened.