We were dictating the rules of entry for non-EU citizens, and we didn't curb immigration in any way. Look at
the numbers man. We can say conclusively that being part of the EU wasn't the cause of the immigration issue you perceive, and leaving the EU to dictate the rules of entry will not solve that problem. The facts are clear-cut on that one.
Reasonable people can disagree on whether the amount of immigration we've had in recent years is a good thing or a bad thing, but you can't agree to disagree on the basic facts of the matter. Being in the EU was not the cause of those immigration levels, the primary driver was the Non-EU immigration levels, which we already have control of, and which has increased significantly as a result of Brexit.
Can you agree that leaving the EU isn't a solution to that problem, in light of those facts? Or can you see why it is difficult for people to understand your perspective if you're unable to agree on the same basic set of facts, which might lead them to alternative conclusions about your motivations?