I can't speak for others, but I've been asking for direct quotes of words he's said so I can understand how those words can be held in any way accountable for the racist actions of people online or, in the example that mancan92 gave, for the racist words of an idiot in a bar. I've yet to see any. I've never said that you or mancan92 or anyone else has directly labelled Neville racist; but hinting that he is complicit in the racist actions of others is still, to my mind, a very serious charge, and one that requires more than the, at best, circumstantial evidence that has been provided.
I am more than willing to engage in discussions around race (I don't like to say comfortable because that hints at a complacency that I, as a privileged white male, have no right to.) But I think with open dialogues such as those comes a responsibility not to slander, or at the very least, to provide tangible evidence when making accusations, however subtle or indirect they may be.