That's the thing. On-prem K8s doesn't mean deploying a vanilla Kubernetes using instructions from kubernetes.io. There is an entire industry of proprietary solutions for running Kubernetes on-prem. RedHat Openshift, Rancher, Pivotal PKS, VMWare Tanzu come to mind.
I don't know when they decided to do that transition but back when I tried rancher a few years ago (when they were transitionning from rancher 1.x to 2.x) it was a real bug festival. I think the only robust solution at the time was openshift which was well, k8s without being vanilla k8s.
Also most tools that were build to manage k8s cluster were nice to deploy a new cluster, not so much to upgrade a cluster so you would have to create new clusters every time you wanted an upgrade. It can scale when clusters and blast radius is small but can be complicated when it involves contributions from n teams.
For this reason when we were managing our own k8s cluster on prem, we were using kubespray which worked but upgrades were a multiple hours affair.
That's a real good point you mentioned: k8s ecosystem is super young.
And so so much changed in the last 4 years.
But at least for me, the 'easy to use' threshold happend somewhere like 2-3 years ago.
And Gardner for example upgrades quite well.
Rke2 is quite stable for me but rancher integration is still not perfect.
But even doing k8s by hand with Ansible was already double 3 years ago. That's what I started and I had it up and running. I switched to rke2 because I realized that this will not be sustainable/ is not worth it to do it myself on this level.
I haven't used k8s in quite a few years, what would you recommend I look at these days to get a good overview and understand all the different pieces in the ecosystem?
They are very easy to use