Thing is, mate, Neville would definitely take it personally when it comes to disruptive influence in the dressing room. And if he truly believes that Pogba is one - he can go overboard in his criticism.
Again that opinion doesn't make it a fact and he could be wrong on Pogba, but him getting overboard is much more probable to have something to do with Pogba's actions on and off the pitch and how he influences the dressing room, rather than race, color or upbringing.
Pogba has been controversial figure under Jose and has made it no secret that he wants away. Accumulated with some poor performances he's rightfully at a spot he would get plenty of criticism, especially from past players that obviously care about the club.
That of course doesn't make Neville, or whoever else criticizes him on football related basis, held accountable for every other mentalist that decides to vent his racist slurs out and call out his name..