That's true, and even if I personally believe he can eventually turn it around, if you keep dropping points, especially at the rate Chelsea are currently doing, then he can have no arguments about losing his job. We are currently on a 6 game winless streak, the last time we won a game of any kind was in the middle of January nearly 7 weeks ago.
Most clubs, no matter where they are in the pyramid, would sack the manager for a run like that - 2 wins in 15 and we literally aren't scoring any goals, like at all. No team in the top 4 divisions has scored less goals than us post World Cup, we are 92nd out of 92 teams for goals scored since November. We have sunken to depths the likes of which I haven't seen at Chelsea in my lifetime. Potter's current record is the worst of any Chelsea manager in the last 30-ish years.
As much as I personally like him, his position is becoming increasingly indefensible.