For AWS specifically, I prefer to have an AWS account specifically dedicated to each service + stage. For example, if I have an image service that handles s3 uploads (say Lambda, S3, Cloudfront and API Gateway), then I'd deploy a "test" environment to a dedicated AWS account and run tests against that. Since it's fully serverless, it only costs a few pennies to test (or free).
That means that every person who runs the tests needs credentials for that AWS account. That obviously won’t work for an open source project. Even for a company project, how do you distribute those secrets? It adds friction for developers getting their local dev environment setup.
Not only that, but you now need network access to run tests. A network blip or a third party service outage now makes your tests fail.
There is also the possibility that an aborted test run might leave state in s3 that you are now paying for. Someone hits Ctrl-c during a test run and now you have a huge AWS bill.
> That obviously won’t work for an open source project
On the contrary - I was an employee at Serverless Inc, working on the Serverless Framework for the last two years, we used this pattern extensively (and very successfully) in our open source repos.
We used part of our enterprise SaaS product to provision temporary credentials via STS and an assumable role, and it works great. You could do the same thing with something like HC Vault.
For Lambda, S3, DynamoDB, the perpetual free tier means we've never paid to run our own tests. API Gateway isn't free (after 1 year), but it's still pennies per month. We've had several cases where tests stuck around a long time, but a billing alert and occasionally some CloudFormation stack cleanup takes care of that.
We still have offline unit tests which test business logic, but everything else runs against the cloud - even our development environments ship code straight to lambda.
Speaking from personal experience, our team wasted far more money tinkering with local dev environments and trying to replicate the cloud than we ever did simply using it to develop.
The blog post in the parent comment lays out our experience and my thoughts, but because of the pretty generous free tier, I don't think we've ever paid a penny for a build/dev/test AWS account.
I try not to develop locally at all anymore. If you're looking for more practical advice, perhaps this will help: https://dev.to/aws-builders/developing-against-the-cloud-55o...