Yes
Few points you're missing here:
1. The job wasn't available at the time, while the Chelsea job was.
2. Given his performance with Chelsea, there is no guarantee he would have done well at Tottenham. You can't be this caustic about Potter and think he'd do well at Spurs (a dysfunctional club in its own right that has chewed and spat out Nuno/Conte/Mourinho)
3. Expectations are huge at Spurs to win a trophy, any trophy, and stay in the CL, despite the financial constraints. Chelsea have higher expectations but that comes with more resources.
With that in mind, i don't think anyone can say the Spurs position would have been ideal for Potter, compared to Chelsea. Even with the benefit of hindsight.
Managerial success is not linear. There are examples of managers who made huge leaps and succeeded. There are examples of managers who made lateral or small leaps and failed. There are no guarantees or heuristics.
Which is why I always laugh at this idea of "starting at the lower levels and working your way up". There are managers in the Championship who will never rise above that. Potter could have rejected the Chelsea job, only for him to be fired after things went south at Brighton and that's the end of his chapter.
When a top opportunity comes up, you take it. If you succeed, you've skipped the fence into elite manager territory. If you fail, you have more money in your pocket, you can go back to the clubs you were managing before, and if you do well, eventually another top 6 club will forget the prior mishap and hire you.
Potter won't struggle to get an appointment below the top 6 tier. And once he does, he'll just have to show a decent level of competency before his name is in the ring again. The managerial go-around is not as punitive as you think. Mourinho has been sacked 4 times from top 4 clubs in England.