Winter sports is what we do, pretty much. Football is still the most popular sport, by far, but we're good at winter sports. A lot of people probably only care about them every 4 years, for the Winter Olympics, but it means a lot for those few weeks. We still had a respectable Olympics, but it's true that our cross-country performances early on were talked about as "disastrous". Now imagine the mood in the media and the country in general in 2006, when we took only 2 golds. It wasn't the least ever, but considering we'd taken 9, 10, 10, 13 in the four preceding ones and 9 and 11 in the two since, it was pretty bad.
And then the Olympics were over and most people stopped caring. Mostly it's a stick to beat the Swedes over the head with, when we can. And also to go "haha" at countries like the US with 60 times the population.