Because being a multimillionaire stops you hearing the n words, or seeing the monkey chants, or being hit by bananas, or being described by your athletic ability rather than your skill and game intelligence no matter how you play, or stops limiting your retirement prospects if you want to go into management?
The problem with trying to explain "white privilege" to people is its name. I guess it could be something like "person from the majority group's non-disadvantage" could work, but that's a bit wordy.