This is absolutely true. I can count easily some 20+ components already.
So this is not walk in the park with two willing developers to learn k8s.
The underlying apps (Redis, ES) will have version upgrades.
Their respective operators themselves would have version upgrades.
Essential networking fabric (calico, funnel and such) would have upgrades.
The underlying kubernetes itself would have version upgrades.
The Talos Linux itself might need upgrades.
Of all the above, any single upgrade might lead to infamous controller crash loop where pod starts and dies with little to no indication as to why? And that too no ordinary pod but a crucial pod part of some operator supposed to do the housekeeping for you.
k8s is invented at Google and is more suitable in ZIRP world where money is cheap and to change the logo, you have seven designers on payroll discussing for eight months how nine different tones of brand coloring might convey ten different subliminal messages.
> The underlying apps (Redis, ES) will have version upgrades.
You would have to deal with those with or without k8s. I would argue that without it is much more painful.
> Their respective operators themselves would have version upgrades. > > Essential networking fabric (calico, funnel and such) would have upgrades. > > The underlying kubernetes itself would have version upgrades. > > The Talos Linux itself might need upgrades.
How is this different from regular system upgrades you would have to do without k8s?
K8s does add layers on top that you also have to manage, but it solves a bunch of problems in return that you would have to solve by yourself one way or another.
That essential networking fabric gives you a service mesh for free, that allows you to easily deploy, scale, load balance and manage traffic across your entire infrastructure. Building that yourself would take many person-hours and large teams to maintain, whereas k8s allows you to run this with a fraction of the effort and much smaller teams in comparison.
Oh, you don't need any of that? Great. But I would wager you'll find that the hodge podge solution you build and have to maintain years from now will take much more of your time and effort than if you had chosen an industry standard. By that point just switching would be a monumental effort.
> Of all the above, any single upgrade might lead to infamous controller crash loop where pod starts and dies with little to no indication as to why?
Failures and bugs are inevitable. Have you ever had to deal with a Linux kernel bug?
The modern stack is complex enough as it is, and while I'm not vouching for increasing it, if those additional components solve major problems for me, and they become an industry standard, then it would be foolish to go against the grain and reinvent each component once I have a need for it.
You seem to be misunderstanding. The components that add complexity in this case do not come from running a k8s cluster. They come from the Reclaim the Stack software.
Alright. So let's discuss how much time and effort it would take to build and maintain a Heroku replacement without k8s then.
Besides, GP's criticisms were squarely directed at k8s. For any non-trivial workloads, you will likely use operators and networking plugins. Any of these can have bugs, and will add complexity to the system. My point is that if you find any of those features valuable, then the overall cost would be much less than the alternatives.
The alternative is not to build a different PaaS alternative, but to simply pay Heroku/AWS/Google/Indie PaaS providers and go back to making your core product.
Did you read the reasons they moved away from Heroku to begin with? Clearly what you mention wasn't an option for them, and they consider this project a success.
Talos is an immutable OS; upgrades are painless and roll themselves back upon failure. Same thing for K8s under Talos (the only thing Talos does is run K8s).
Ages ago, I had the notion of booting from removable read-only media. At the time CD-ROM. Like gear for casting and tabulating votes. Or controllers for critical infra.
(Of course, a device's bootloader would have to be ROM too. And boot images would be signed, both digitally and manually.)
Maybe "immutable boot" and immutable OS can be complimentary. Surely someone's already explored this (obv idea). Worth pondering.
Secure Boot can do the "ROM" scenario on conventional read-write media as long as your OS is capable of maintaining a chain of trust and enforce code signature checks before execution. The media is technically writable, but any writes would break said chain of trust on subsequent loads and so the malicious code would fail to execute.
If interacting with a remote system, a TPM can also be used to achieve the same (though if you have TPM you'd generally always have secure boot) - in this case, your OS extends TPM PCRs with the hashes of all the components in its chain of trust, and the remote system uses remote attestation to prove that you indeed booted & executed the expected code before granting you an access token which is never persisted.
In the second case, malicious code would still run but would be unable to pass that authentication step and thu be unable to communicate with the remote system. This is suitable if the machine itself is stateless and not sensitive per-se, and the only security requirement is ensuring the remote system is only accessed by trusted software.
So this is not walk in the park with two willing developers to learn k8s.
The underlying apps (Redis, ES) will have version upgrades.
Their respective operators themselves would have version upgrades.
Essential networking fabric (calico, funnel and such) would have upgrades.
The underlying kubernetes itself would have version upgrades.
The Talos Linux itself might need upgrades.
Of all the above, any single upgrade might lead to infamous controller crash loop where pod starts and dies with little to no indication as to why? And that too no ordinary pod but a crucial pod part of some operator supposed to do the housekeeping for you.
k8s is invented at Google and is more suitable in ZIRP world where money is cheap and to change the logo, you have seven designers on payroll discussing for eight months how nine different tones of brand coloring might convey ten different subliminal messages.