I've zero knowledge of ice hockey but I know it involves skill. It's quite obvious to see. The only skill involved in American football is the quarter back throwing the ball and to a certain extent the bloke who runs and catches it. The majority of the other players are responsible for blocking, tackling and simply running which doesn't take much skill especially when these players are physically huge.
Having 100-120 players in a squad for every team suggests that there are lot of players with a similar level of skill and if there are so many players like that it doesn't take much to succeed.
The bloated rosters are only in college, and there is a reason for the rosters being so massive.
You have first team players, second team players, third team players, practice players. You have to be able to carry out full team scrimmages. For example, there will be an OP offensive squad, whose job it is to practice against the first team defense every day of the week and to mimic the upcoming oppositions tendencies. There is also an OP defense which do the same thing for the offense. Usually these teams are two deep because they prep the second team as well.
You are looking at a 4 year rate of turn over. Usually players actually play their final two years, so their first two years unless they are spectacular are spent as understudies. If you did not carry these players giving them practice time against the current first team they would simply not be ready or competitive.
In terms of skill, every position in Football requires skill and sometimes extremely specific intricate skill sets. Linemen have to have EXTREMELY coordinated feet. They have to have massive balance and agility for people who can weight in excess of 300 pounds. They need to have specific techniques for blocking or shedding blocks. Their footwork is paramount. If you're big and strong as a bull but can't move your feet you're worthless.
Running backs have to be able to read the field. They have to be extremely fast, extremely powerful, or extremely agile. Like reading the weakness in a rock face, running backs have to find the line of weakness in a defense to exploit a hole or gap.
Linebackers play the same way on the defensive side of the ball. In addition to having sometimes quite complex defensive instructions to follow.
Football is an extremely mental, and extremely physical game. They don't manipulate objects for the most part like a puck or a ball, they manipulate bodies trying to manipulate them back.