Which is not actually surprising because the dominant change is the migration from non-poor to poor, not the change of the average score - 1980 has only 9% poor results, 2020 has 40% poor results.

When scores change, at first the average scores within groups will change, but if the scores change enough they will eventually cross group boundaries at which point the group averages become more or less irrelevant and the changes are mostly reflected by changes in group sizes. The group averages are bounded by the range of each group and can only indicate if the group is moving towards one of the boundaries, but as scores cross boundaries they start contributing to different group averages and at opposite ends, e.g. decreasing scores will at first contribute at the lower boundary of one group and then at the upper boundary of the next lower group which may increase the average score of both groups, in one case by loosing a low score relative to the group range, in the other case by gaining a high score within the group range.