Playing? He was clearly raged by the news of Cat capturing the imp and immediately started to attack. He lost his cool and wanted to take down Ned there. As for the rest, Ned was clearly in good shape unlike Robert and it had been only nine years since the last battle, plus we saw how the Starks practiced everyday in Winterfell, I don't believe his life was as comfortable as Robert that he would have lost all his swordsmanship. They had a tougher life than Robert who mainly lost his shape due to drinking and having to sit in the throne and not doing any physical work.
It is clear that your opinion of Jamie is made up from something that the rest of us are yet to see.
About fighting, absolutely not. Just look at that video from their fight and tell me who had the upper hand.
It was like watching Sidious fighting Yoda, technically it was a drew but it was clear to see the difference between them.