That's the problem, ain't it.
Present people with a half decent alternative and the Tories are out (for the time being, at least).
Another problem is that Britain has been divided for years now along geographical and/or demographic lines. If it had been up to the big urban areas, Brexit wouldn't have happened. If it had been up to the younger generations, it wouldn't have happened either.
(I'm pretty sure it wouldn't have happened if the basic turnout among youngish people in London, Manchester and Liverpool had been just slightly higher. The oldish idiots have been allowed to have their way - rather than them being an actual majority in terms of political views).