I reckon lots of fans just consider his qualities in theory, fast, tall and strong, link it together with excellent that run of form a few years ago, and Mark him as "dominant" and "excellent at defending".
Every time I see him play he's making mistakes of some sort and coming second in situations he shouldn't be. Not blatant enough to register on the Lovren scale, but there is a reason why you are conceding truckloads of goals this season with Smalling as first choice.
He's just not that good a defender. He'd be 4th choice at the top four clubs and he's 1st pick at United. He's just not starting material for the kind of club you want to be, and a solid reflection of why you are where you are now.