My guess would be that because it's an integration with a sprawling native library, there is a lot of code which touches Python's own C API. That is not entirely stable, and it has a lot more source stability than binary stability:

https://docs.python.org/3/c-api/stable.html

As time passes, it makes sense to support only recent versions of both Qt and Python, hence that matrix.