I generally don't favour sacking managers quickly, and its fair to say that Chelsea would struggle to replace him with a quality manager mid-season anyway. But its also clear that Chelsea are in a doom spiral now, and its very difficult to get out of them without a radical change. Unfortunately, in football, the manager is often the thing that's easiest to change for the biggest impact.
While 14 points to 4th place is surely insurmountable, given the form of the teams above, the gap to 6th and (probable) Europa League qualification is definitely doable. As much maligned as it is, with a squad Chelsea's size they need the games to keep coming to keep people happy, and much like us this year, they would start the competition as one of the favourites. Plus a few more home games would keep some money coming in to cover those long contracts.
I personally never rated Potter and felt his Brighton team were overrated, but I don't think this is down to what Potter has done. I don't think he's had any chance at all to try and run the team as he wants, he's done nothing but try and put out fires since he arrived. But the reality is he's reached the point where it would be harder for him to put it right than it would be for someone else coming in. The fans are turning, if not turned, and the players don't seem to be responding to him. You feel like the jolt that a new manager brings could be enough to get a couple of wins on board and turn the mood of the camp entirely.