Pretty much.
White people weren't dumped over the side of boats in droves and left to drown in the North Atlantic because they were perceived as less than human. They weren't subject to segregation in schools fifty or so years ago, they have much more household wealth on average, than black or South Asian families. They face racism from social structures like the police on a much larger scale than white people, these things have a material affect and continued racism through discrimination or slurs is a reflection of these systems.
Calling a white person a racist slur, is an anomaly and the reality is those racist structures don't affect them negatively, in the same way black people are.
White people, in this country (UK) at least, are the dominant class, socially, economically etc. An example, I have never been stopped by the police for a stop and search. Black friends who work with me have, multiple times, been stopped but not me? Must be a real random coincidence.
An easy think here is to replace black people with jewish people when thinking about this subject. For example, can you be anti-semitic to a non-Jewish person? Or is it a non sequitur or oxymoron?
Same when people try to both sides the white people suffer racism argument.
Just my two pennies.