Nadal was the only one who could consistently challenge and beat prime Federer (04-07). Novak wasn't a factor at that time, and Fed did beat him the 2 times they had met (AO and USO 2007).
For people who think I'm being selective about marking a prime, you can look up his win percentages by year. Year-long percentages will be less influenced by things like the form of rivals, etc, since they are mostly determined by winning the smaller matches consistently. He dropped off big-time in 2008, not just to Nadal but generally. And while he had a few bright spots after that (2009, 2012), none are comparable to peak 04-07... till we get to 2017 (92%), a year when Djokovic was again a non-factor.
Win percentages: 04-07
93/95/95/88
81% in 2008, a sharp fall.