At first glance you're right Uruguay's record doesn't look great - 9 goals conceded, but 4 of those were against that crazy Hungarian side which scored 27! goals that tournament . Also that 1954 tournament was a bit mad in terms of goals scored (from the last article i read the tournament had most goals per game in any world cup), if you look through the results they are full of games where 4+ goals scored.
Also if you look at the goals conceded by the top 4 teams they are as follows:
Hungary - 10
West Germany - 9
Uruguay - 9
Austria - 7
So given the context i wouldn't say they had a particularly bad tournament in terms of defence and he was picked in the all star team for good reason. Unfortunately there is little information on players outside the West Germans and Hungarians for that tournament. According to this article it was on the back of his performances in the 1954 world cup that European teams began to take interest in him:
http://www.world-football-legends.co.uk/index.php/uru/107-santamaria-jose