Actually, the effect happens just as much during downscaling. It is just not as noticeable, because your artifacts happen at subpixel scale. But they do affect the output colors.

The math does not really know the difference between upscaling and downscaling.