The simple fact EU citizens have zero restrictions regarding freedom of movement, increased EU migration directly impacts immigration from outside the EU.
It might seem like a minor issue to you, but to myself and others, having family members 6000 miles apart because they happen to be born in the wrong region is quite a big issue.
I am also aware that the UK is a very attractive place to come and live, which is why we cannot have FM of people on a global scale. It is not practical, so checks have to be made and due to our membership of the EU, those checks have to be on people from outside the EU.
Ideally there would be a simplified and fairer system that allowed people to apply to come and live here on an equal footing, whether EU or not.
It is also one reason for voting leave, not the reason. I shall not be discussing any others mind, as the responses to people genuine issues here isn't worth it.