It is though. Not fair, but it is the way how the world works. When people look at his legacy, they'll see a boxer who hasn't defeated any hall of famer on his career. That is problematic.
I think that there have been many boxers who were great but people avoided them, and no one remember their names. Charley Burley for example apparently was ducked from most top boxers, and he never received a title shot, Sugar Ray Robinson was ducked by many too and needed more than 70 fights to get a title shot. Archie Moore needed 16 years of winning in order to get his title shot.
In GGG's case, it is a bit different cause he isn't chasing a title, instead he holds the belts. It also doesn't help that middleweight is a poor division, with not many quality boxers.