When they say ‘control our borders’, I think they just mean the ability to restrict who has the right to work legally in the UK.
I think Brexiters envision a future where the UK has zero (or in any event very low) customs tariffs on all imported goods, regardless of origin, and regulations lower than or equal to the EU. In this future, the UK would have no customs duties to collect from EU imports and no concerns about EU goods being substandard or illegal in Britain. So the Brexiteers perceive no real danger from goods being smuggled into Britain from Ireland. In a sense, they’re not worried about policing the border because they don’t think there will be anything they need to keep out or inspect.
The EU on the other hand has real concerns that Britain could undercut EU regulations and therefore smuggling goods into Ireland is a big concern from a safety, customs and market protection perspective. (See Merkel’s fear about Britain going down the Singapore route:
https://www.theguardian.com/world/2019/sep/11/angela-merkel-stresses-danger-of-britain-becoming-singapore-on-thames-no-deal)
The reason free-market Brexiters think they’ll win in the end is because they think there is a fundamental asymmetry of risk when it comes to trade.
Personally, I don’t agree with their logic - unilaterally eliminating all import tariffs would pretty quickly decimate UK producers and cause mass unemployment or a slashing of wages. I am just explaining their thought process.