Yeah I think you should. The fox dens etc don't really add anything tbh and you do end up with a lot of duplicate charms. Probably just worth doing enough to open up the slots, if that.
I did enjoy the bamboo strikes though. I think them (to increase resolve slots) and the spa's (for health0 are worth doing although you'll be massively overpowered by end game anyway.
@Massive Spanner The yellow birds lead you to everything though (inc story missions). They aren't anything of themselves. Or do you mean the bird cages? I didn't go out of my way to look for them either.
Whoever said the duels were the best bit was right as well. Love em. That whole side quest with the 6 straw hat duelists was cool and the music when you defeat someone is epic.