The book Team Topologies discusses Platform Engineering. They consider four core team types, and the platform team is one of those. Some more information about those types can be found here: https://teamtopologies.com/key-concepts-content/what-are-the...
I can definitely recommend this book for a (new) perspective on how platform teams fit into organizations.
This is a typical challenge for platform teams. Because they are responsible for the underlying foundation on which applications are build, a mistake can easily impact all those applications.
An important realization in my opinion is that a platform team is just another development team. They should consider their platform as a product and the developers as their customers. To minimize any downtime they should use the typical mechanisms that developers are also using: automated tests, pair programming, rolling out changes to test environments first, etc.
You can set the DeletionPolicy attribute to "Retain" to work around this S3 issue. CloudFormation will successfully rollback without attempting to remove the S3 bucket. You can then do so manually yourself before trying to deploy it again.
You better do so before deploying again, because the roll forward will break since the resource already exists.
This a major pitfall when using DeletionPolicy=Retain with named resources. It breaks seamless rollbacks/rollforwards. If you rollback, in order to deploy again you need to either delete the named resources with DeletionPolicy=retain that were rolled back, or update your template to rename them all. It is such a huge pain.
True, but it beats the alternative where CloudFormation deletes objects that you didn't want deleted. The underlying issue is that the S3 objects are outside of the CloudFormation scope, thus it takes no risk and doesn't delete your objects.
A nice feature would be a "ForceDelete" deletionpolicy where it would delete the objects. You can even set this initially when creating a stack, and change it to "Retain" later when the stack is stable.
Totally agree btw that it's a huge pain initially, though once you know it it's also not that hard to work around.
My preferred behavior would be for CFN to not barf when rolling forward. In other words, to be able to assume control over a resource that already exists.
I can definitely recommend this book for a (new) perspective on how platform teams fit into organizations.