What men? His whole army was butchered (hence Ramsey saying that Jon doesn't have an army anymore when he returned to Winterfell)
It's actually a big problem, the Knights of the Vale are the only military power left on the North and Littlefinger can do whatever he wants, despite Starks being Wardens again
You heard what the other Lords said when he was negotiating for men. They wouldn't back Jon while the Boltons were in power, now the Boltons are gone and the Starks are back to being Wardens of the North you will see them rejoin his Family.
I think judging by the shots of the battle, there were still around 500 of his men left (discounting the archers). So he probably still has a smallish army of loyal men. The question is what does Littlefinger do now, he ultimate goal is to be sat on the Iron Throne so you have to consider from his position that it's like Chess. He just moved his Bishop forward and removed a Knight from the board, so whats his next move to get checkmate? He either destroys the Freys or allies with them to take down the Lannisters.