I personally wouldn't use the expression "Started going wrong" because I still love the show and find it to be hugely rewarding and entertaining television. Frankly, I was irritated that they had to skim over the battles in the earliest days of the show and I'm delighted that they can now achieve these gigantic technical masterpieces that push back the boundaries of what television is capable of. And I think they've become a lot better at fleshing this world out as the show has grown. There were points in the early seasons where you could tell they'd just rocked up at a country park, or a location you could have just stumbled across yourself. It really took you out of the world whenever that happened.
Observe:
With that being said, there was a slight drop in quality after season 4 ended. I should stress that I do believe that drop was only ever so slight, as opposed to the catastrophic collapse that some viewers (and especially some book readers) would have you believe, but it was there all the same. It's something that writers behind the show even acknowledge themselves - Bryan Cogman stated pretty clearly in a recent interview that he believes the show to have two distinct eras: seasons 1-4 were the "adaptation era", and seasons 5-8 are were the "TV show era".
The "adaptation era" took the model of storytelling that's present in the books and put it to screen. A series of dominoes were slowly placed in a line before they toppled over and inevitably caught out anybody who didn't see them coming. The "TV show era" changed things slightly, using the more traditional 'set-up and payoff' mode of storytelling. Neither method is better or worse than the other (I think seasons 5 & 6 - and 8 so far - stand up to the quality of the "adaptation era"), but any perceived change in storytelling style after four seasons is going to rub people up the wrong way.
Things definitely took a noticeable downturn for season 7, though. Again, not catastrophically. Hell, it didn't even drop to simply being a "good show" - it was still a great one. The main reason for this was that plot efficiency took precedent as the show gave itself less time to piece things together for the final leg. It meant that time we would have usually spent building character without moving the plot forward too much was sacrificed, and the show instead focused on the finish line. Again, that's not objectively bad storytelling (season 7 was still powerful, entertaining TV), but it changed the rules that the show had established for itself.
Frankly, I'm really enjoying season 8 so far because it's returned to the style we saw in seasons 5 & 6 in terms of pacing and budget (especially with episodes like 'A Knight of the Seven Kingdoms'), while containing flashes of the "adaptation era" in the bold decisions it's made. The dominoes collapsed on Jon during 'The Long Night', so it was Arya who stole his moment and killed the Night King. That's the kind of unpredictable but inevitable twist the show used to be famous for. Because of the way the show is written now, the clues to the end have already been given to us, which makes it harder for them to pull off intrigue, but it'll still be a fun ride all the same.
TV shows never stay the same. They're pieces of art in constant flux. Game of Thrones has survived some production changes that would have killed other shows dead. Running out of books to use having relied on them so much initially, then having to condense such a ginormous story and mould it into a logical end-point. I mean, just look at what happened to The Walking Dead after season 4/5. That could have been Game of Thrones but it wasn't - Benioff and Weiss deserve huge credit for that.
