Bran met Samwell at the Nightfort (that was how they got over the wall).
Bran then went to Crasters Keep (they got captured and was able to escape when Jon attacked it).
The 3 eyed raven cave isn't marked on any map i've seen. Just a rough estimate and they have it roughly between Crasters Keep and the Fist of the First Men (that was the place where Sam found the Dragon Glass).
So the White Walker Army is a long way from the east shore. But then the last time we had a confirmed sighting was at the 3 Eyed Raven cave and that was Season 6 episode 5. So they've had 6 episodes to march back to the coast I guess.