No, not just from watching TV, like I said I can't afford to this year but up until around late October/November I went all the time, saw Spurs, Arsenal and Norwich at home which was 3 of the first 4.
But anyway, that's not the point, you can definitely tell by watching the game whether a fullback keeps his position. Tactics will say they have to get forward, but both should pick their moments better because they can get caught out. With Rafael it's actually more obvious to me than Evra, because you him busting his arse to get back, have you not seen him do it? Head down and sprinting back to defence, it has happened loads.
He's also sometimes not tight enough to his winger, which leaves them open for a pass. Raf had the most tackles/per game last season IIRC, and it doesnt surprise me because his winger is often unmarked, with Rafael not getting tight enough. Not that this bothers me at all, he's immense at tackling so it doesn't really matter, but someone like Gaz would just defend differently.