If it was once, you might have had a point. If there was anything remotely resembling a general balance between soft and harsh decisions across teams, it would be rational to chalk it up to innocuous inconsistency from the referees. But there is no such balance. It's a very consistent and undeniable trend that some teams are treated as harshly as the rules can possibly allow for (and sometimes harsher still), and other teams are afforded every conceivable leniency, with a number of objectively incorrect decisions in their favour that vastly outweighs any realistic chance of happenstance. It's very much valid to point this out, because it has been going on for far longer than mere coincidence can possibly account for.
As bad as refereeing is in the PL, it's every bit as bad that there are fans and pundits who seek to look smarter than the rest by claiming that the things we can all see, every single week, do not in fact take place. It is beyond any possible dispute that certain teams are routinely given a helping hand by the refs, and others are routinely swindled. If it was supposed to be "swings and roundabouts," we're approaching lottery-winning odds, and it's just not rational to believe it. It's abundantly clear that the referees in England make decisions based on which teams are affected. It's undeniable. To say otherwise is frankly a declaration of idiocy and delusion. It has been so consistent for so long that nobody can deny it and expect to be taken seriously.