I don't think there are many who are so sure about individual accusations against Jay-Z. What many are sure about is that when people say that Jay-Z has been up to no good then it is extremely plausible, likely even, that he might have been up to no good, especially given that:
Jay-Z has been friend and business associate with Diddy, throughout the time when Diddy was on a sex and violence criminal rampage, some of which we have seen clear video evidence of.
Jay-Z has been friend and business partner of R Kelly when everyone knew he was a rapist pedophile and when he was kidnapping and enslaving young girls. They made a whole album together 2 years after the child abuse tapes were public.
As with Jimmy Savile and Harvey Weinstein, when there are a large number of historical accusations, there will be some which are less reliable and some that may contain inconsistencies but that doesn't necessarily mean they are untrue. And they don't necessarily detract from the overall observable patterns of behaviour.