Mostly mismanagement at local (regional) levels. It's under the power of the communities over here so it's not coordinated on national level, and our federations fail to put competent people in the right places for our youth to grow (and not enough money available to attract foreign coaching talent either). Swimming is the prime example of all that, we had a shitload of swimmers at London and Rio, culminating with a silver medal on the men's 100m free. The guy stops, isn't even asked to join or exercise any function whatsoever even though he indicated he kinda wanted to, and now we had a grand total of 2 swimmers at Tokyo and might not even have a single one in Paris in three years. Quite scandalous because the talent is definitely there, it's jus the rules which are made too complex and the younger generation isn't properly helped in order to develop all their skills (or even find out what they're actually good at).
The few programs we are actually able to put in place have been steadily developed over the last decade, hockey being a good example of that. Other than that we simply rely on some individual brilliance to get us some medals (Van Aert, Derwael, Meesseman for the women's basketball team, ...) and hope for the best.