Actually there would, because they would then have access to FoM and could quite easily move outside the UK, effectively using the UK as a means to get to another EU country.
Which is why no EU member does this, because it impacts all EU members.
There has to be some checks and balances in place, but there doesn't need to be a massive cost and an excessive time frame involved.
Consider this. Many people are worried about EU migrants futures in the UK. They can't settle or plan futures because of uncertainty. Well that's what non EU migrants deal with every day.
While more people do come from outside the EU, a majority of them do not remain here, which is due in part to the difficulty in getting citizenship.