In each of his three seasons I would argue that he has been at the very least, no worse than the "average" performance of the squad.
Year 1: Under Moyes, the entire team played badly, Fellaini included. One can make a case for mitigating circumstances such as playing the first half of the season a wrist injury, and that he had no preseason (something often used to excuse other players if they do not exactly hit the ground running). These factors are dismissed out of hand by the Fellaini haters (buzz term, but appropriate in this case) of course. On the whole, not really any worse than most of the other outfield players, but an easy target for the fans who didnt want him from day one (amongst other factors such as overpaying, and having a shite midfield which Fellaini was never going to be the solution to).
Year 2: One of our better players over the course of the season. The whole "had 4 good games" claim is just biased revisionism. For much of the season, our best play revolved around the Blind/Fellaini/Young triumvirate on the left hand side, he was usually one of our best performers (ie MotM or close) in the big games against the likes of Chelsea and City, including very good performances in a DM role. Did this despite apparently being on the verge of being sold, and the fans being on his back right from the off.
Year 3: Not utilised much initially by LVG except as an emergency striker. Another victim (certainly not the only one) of the switch to 4-2-3-1 which left him without a really suitable position in the team. We have shown an annoying tendency to employ hoofball when he is on the pitch - which is hardly the players fault but the managers. Nonetheless, in a season where we struggle to score or do anything worthwhile, I would suggest that Fellaini has had more impact and looked more likely to actually impact the game in a positive way than the likes of Mata, Herrera, Memphis (apart from a couple of good games) and Rooney (ditto).