At the time Portugal had a better squad and they were seen as favorites both in 2004 and 2006 by the neutrals.
People really overrate those England teams because of big names, but at the time players like Gerard, Lampard, Rooney, John Terry, Rio Ferdinand and Ashley Cole were still young and not the finished product they were in their prime. These players hit their prime after Eriksson left. Meanwhile the likes of Beckham and Owen were already in decline. Then there was the goalkeeper problem and lack of wingers. It simply wasn't a well-balanced team and is only rated so highly now because of newer football fans who recognize the star players as legends of Premier League, but don't know the context.
The core of the 2004 Portugal squad were the Porto players who just won the CL and had a lot of experience playing with each other. Add Figo and C. Ronaldo on the wings and it was a great and well-balanced team.