Don’t code with mocks period.
Structure your code such that it has a functional core and imperative shell. All logic is unit testable (functional core). All IO and mutation Is not unit testable (imperative shell).
Mocks come from a place where logic is heavily intertwined with IO. It means your code is so coupled that you can’t test logic without touching IO.
IO should be a dumb layer as much as possible. It should be Extremely general and as simple as fetching and requesting in the stupidest way possible. Then everything else should be covered by unit tests while IO is fundamentally not unit testable.
If you have a lot of complex sql statements or logic going on in the IO layer (for performance reasons don’t do this if you can’t help it) it means your IO layer needs to be treated like a logical layer. Make sure you have stored procedures and unit tests written in your database IO language. You code should interface only with stored procedures as IO and those stored procedures should be tested as unit tests in the IO layer.
> IO should be a dumb layer as much as possible. It should be Extremely general and as simple as fetching and requesting in the stupidest way possible. Then everything else should be covered by unit tests while IO is fundamentally not unit testable.
Another way to put this is to ignore all the cool problems your database can solve for you and hand roll it instead. This also is a recipe for riddling your codebase with data races.
You're trading fairly easy problems for very hard problems. No thanks.
When running tests, optimize your database configuration for running tests against it, and make it possible to run those tests in parallel.
> Don’t code with mocks period. Structure your code such that it has a functional core and imperative shell. All logic is unit testable (functional core). All IO and mutation Is not unit testable (imperative shell).
I'm not sure what your definition of mocks is here.
Even if you follow a principle where all IO and mutations is not unit testable, you still need test doubles to plug in the interface of your testable functional core. Because you need to pass data through to exercise code paths. How do you call the component that implements the interface of your dumb IO layer?
For every “I made this way more efficient/clean with a stored procedure” there’s some incident where the stored procedure was left off, fired too many times, or just caused some problem due to its lack of visibility to the end user in the codebase
Exactly. I only break out SPs when I absolutely need to squeeze as much performance from my data store as possible. I'd rather pay for more horizontal scaling than push my logic across a system boundary where it's difficult to monitor and debug, because bugs in SPs are probably the single costliest class of bug in a prod system.
I’ve always felt this was partly database vendors fault for not having tooling that integrates well. I think there is some theoretical world where pgTAP and other database specific test harnesses work fairly seamlessly with language runtimes via some well designed SDK. Tracing would need to walk into the db call stack from the application code.
Clearly this is harder to get right than expected since there aren’t many places doing stuff like that, that I’m aware of at least. Maybe if some of these database vendors built out better API’s for things like that with the expectation other language runtimes would be calling them the situation would be better. But it seems a lot of the old school db world is perfectly happy to build their products as if they were a black box from the application’s perspective
At least with Lambda you can use mainstream programming languages. You can also have single lambdas that handle multiple endpoints or events. It’s a bit different.
I am actually at a point in my career that I won’t work for a company that is dependent on stored procedures. Rollbacks, versioning, branching, testing and everything else is a pain once you introduce stored procedures.
As a lead for mostly green field applications over the past decade, my one no compromise architectural edict is no stored procedures for online production code.
I might use it for background utilities or one off maintenance.
Is this about having application logic in multiple places, or having application logic co-located with the data? Does it make a difference if the procedures are database triggers vs being directly called by the application code?
Would splitting off some of the logic into a separate service (that does its own database accesses) have the same issues?
Say I have functionality to get customer data. If the sql logic is in code, I can create a new branch and work on it, merge my changes, I can do blue/green deployments. A rollback if you’re using Kubernetes is a simple matter of reverting back to the configuration file containing a reference to the old Docker container etc.
How do you roll back a dozen stored procedures quickly? How do you just create a new branch or test a regression?
With all of the logic in code, it’s a simple matter of checking out a previous commit.
Triggers are even worse - “spooky actions at a distance”.
> How do you roll back a dozen stored procedures quickly?
Liquibase rollback feature. The SQL changesets and rollback strategy is all defined in my .sql liquibase files and goes inside my git repo alongside my scala/python code the associated CICD automated integration tests + deployments.
Blue Green deployment can handled via bidirectional active-active database replication across your two disaster recovery databases; you deploy the newer stored procedures on one but not the other. Or you have sharded databases for some but not all customers and deploy blue/green with appropriate blue/green sharding. Or you could have separate version numbers for the stored procedure names (perhaps with a wrapper stored procedure that works differently on different shards/databasenames to isolate the impact from your non-SQL codebase.) Or just put a form of feature flags in your stored procedures; it's not 100% blue green but with minor integration test coverage it can be quite safe and allow you to run two code bases at once even in a single database.
Agree with you on triggers (and I agree to a point that stored procedures are often not the right approach despite my above rebuttal... but it is quite straightforward with the right tooling/mindset. For high-volume analytics with aggregations you might be better off leaving the data in columnar database SQL and doing some stored procedures if the code was complex enough; but for CRUD apps I would strongly discourage stored procedures.)
I agree with Liquibase for source control of schema.
But you don’t see how much harder this is for developers with feature flags in stored procedures, etc over standard git? There is tooling around feature flags for code
I think we are in violent agreement though, for OLAP and analytics, I wouldn’t care as much. Because of the way that Redshift works (columnar store) and how it compiles and optimizes queries across a cluster, I wouldn’t care about stored procedures.
If I need to fork code am I going to then make copies of all of the stored procedures corresponding with the code and now have getCustomer1, getCustomer2… and doing the same with all of your procedures?
And then do I need to change all of my stored procedure references in my code? Do I merge my changes back to the main one when done?
Isn’t just doing a “git branch” easier?
Rolling back, is just doing a git revert in your separate Kubernetes repository (you are doing proper Gitops aren’t you?), recommitting your code and your cluster is now using your old Docker container.
If you aren’t using K8s, you just push your old code.
I'm not really following what you are saying. Sprocs are in the same repo as any other code, a branch is a branch, why do you think you can't create a branch that contains sprocs?
Am I then going to deploy the stored procedures to a shared database that other developers are using with their own branch or am I going to have my own database instance?
Are all those developers running different branches of code with automated schema changes on the same database? What if someone adds or removes a field? Stored procedures are not the issue here
If I have logic that returns customer data in code, I can branch the code, make and test my changes locally and other developers can do the same without any conflicts.
Now imagine the next major release requires 50 different sql changes. Would that be easier to push and maintain one git branch or 50 stored procedures?
Again how do you rollback easily and effectively when your release depends on multiple store procedures?
Adding a nullable field shouldn’t be a breaking change, removing a field would be.
> Now imagine the next major release requires 50 different sql changes. Would that be easier to push and maintain one git branch or 50 stored procedures?
Why aren't those 50 stored procedure changes also in your one branch?
It kind of sounds like you only put some types of code in git, as opposed to everything. Is that correct?
And now we are getting back to having to deploy those 50 stored procedures to a database to test your code when you branch.
Are you going to use a shared dev database? What happens when other developers are also making changes to other branches?
I’m assuming you need to test with a representative size data set with a representative size database server.
Just like with traditional Kubernetes Gitops where you have your source code in one repo and a pipeline to create your docker containers and your K8s configuration in another repo to manage your cluster and update the referenced Docker images, even if you don’t have stored procs, you would keep your schema change sql files in another repo and “version” your database separately with a separate pipeline.
> And now we are getting back to having to deploy those 50 stored procedures to a database to test your code when you branch.
Are you going to use a shared dev database? What happens when other developers are also making changes to other branches?
Valid questions, but there are also valid answers, the primary one is of course "it depends".
Sometimes a shared DB works fine, as long as you manage other work being done to avoid conflicts.
Sometimes an isolated DB is needed for either just this work, or this work plus related project work isolated from other projects.
We do all of the above, and yes it takes effort, but that effort exists regardless of sprocs or not due to the nature of most of our projects and changes, rarely do we have small isolated work.
But I'm guessing our environment is not the same as your based on your statements, this is our environment:
1-Medium sized enterprise which means we run many apps from vendors, some on prem, some in cloud, as well as internally developed apps - in other words it's not a homogeneous environment and we don't control all aspects.
2-Functionality that frequently either sits on the side of operational systems, or in between the operational systems or as a layer on top that is unifying the business activity across systems and realizing a higher level of abstraction and activity.
3-Focus on end to end testing due to higher bug detection rate (i.e. around 70% bug detection) vs unit testing (i.e. around 30% bug detection), and due to work flowing through multiple apps frequently.
> even if you don’t have stored procs, you would keep your schema change sql files in another repo
Our schema changes are typically in the branch being merged and deployed. Our deployment process has pre and post DB actions sections.
But some changes are too big for that type of thing and require multiple day deployments with various business and technical actions happening in a specific sequence across multiple different apps (e.g. hold new activity for set of skus, stores and web site complete final transactions, clear DCs, switch to new process where those skus have a new unique flow through the systems and the enterprise).
This throws away most of what your database can do for you. I personally like to do authorization at the database layer. There are a tons of other things a DB can do, like trigger a webhook. Are you going to frequently pull from your DB for live updates?
I think OP issue is that he didn't understand what mock tests actually tested against. They are called mock for a reason. There should be a better way to provision a database when running tests and have a full simulated environment.
I think you've swung the other way a bit too far, I've never seen application logic split between the app and database not become a huge PITA.
I would never have my DB trigger a webhook for me but I would have it act as a message broker back to my own code that triggers a webhook. I don't want my DB ever initiating an outbound connection.
> This throws away most of what your database can do for you. I personally like to do authorization at the database layer.
The whole point of writing maintainable software is to throw away most of what technologies can do, because you are way better off if you do not make the mistake of following that path.
For example, at a first glance doing auth at the database layer is a huge mistake. You design your systems with a few nodes having exclusive access to the DB to ensure your access is secure without having to waste the overhead of checking credentials in each db access. The main design trait is speed. But you do you.
> There are a tons of other things a DB can do, like trigger a webhook.
There are plenty of reasons and hard learned lessons behind principles such as separation of concerns.
so what you should do instead is write a load of pure functions and intersperse them with impure functions, and when it comes to testing that, if someone writes a different impure function than the one you wanted, you just fail the test and trigger a post to HN: "don't code with mocks."
You failed to read and/or understand what I wrote. Also how do I fail a test for a new code addition I didn’t even write?
Unit tests dont test impure functions all they do is test pure functions. Impure functions aren’t unit tested period. So when someone writes an impure function I don’t like what happens? Nothing. It wasn’t part of unit tests annyway, understand?
If you’re referring to integration tests well a newly written impure function isn’t part of integration tests anyway so nothing happens in this case either.
I found your last comment about how I’m triggered to write such a comment rude and lacking in even understanding what I wrote. Please try to understand my point before saying something like that… if you can’t please refrain from replying in such a way as it’s against the rules here. Thanks for reading.
That’s an integration test. Your external api returns a number, let’s say 10. You have two local functions. One that is very general and fetches that number, and another that transforms that number. Your local transformation function is the one that is tested. The other function is part of integration tests and in general can’t be unit tested.
Exactly, there's a layer (integration testing) where I usually do need to have a test against a mock to validate my function doing the external call is passing a request I expect. I can break it down into a parameter supplier function that will generate the proper request params and test that but I don't see any gain in abstracting that away simply to avoid mocks.
You use a "VCR" module to record the real request and response and then play it back in your unit test. Boom you get the power of an integration test with the speed and determinism of a unit test. Then you turn off the VCR in CI to catch if/when the upstream API changes on you and now it's a real integration test.
The problem with write tests, not too many, mostly integration is that unit tests are too damn good at giving you instant feedback and a tight dev loop. So we turned integration tests into unit tests. Combined with SQLite :memory: for unit tests and Postgres for the final integration tests we don't have to mock anything.
Source: Currently use this method at $dayjob with great success.
My question was a rhetorical one, I do use request-replayers and other techniques (I've been through a bunch in the past 20+ years as they were developed).
I was challenging the absolutism of "never use mocks", it's just another technique and can be applied easily in integration tests if the guidelines are well set in the team on how to not use them.
Also, I do not do external calls in the CI/CD pipeline, it inevitably makes tests brittle and flaky.
Great advice. I follow it in my coding efforts and it has never failed me. Great book about this: Unit Testing Principles, Practices, and Patterns, Vladimir Khorikov, 2020
The imperative shell is untestable with unit tests. You don’t test it this way period.
Think about it. This IO layer should be so thin that all you’re doing is fetching and getting so this layer does a direct external call. When you mock this entire thing is basically what is mocked.
So if you mock or don’t mock if you write your code with the pattern of functional core and imperative shell the imperative shell is what is mocked and is never tested via unit tests regardless. You cannot test this.
Another way to think about it, is like this. Write your code in a highly decoupled way such that all logic or transformations done in your code can operate independently from things you have to mock. Then for the things you have to mock, what’s the point of mocking them in the first place when your logic can be tested independently from the mocks?
What’s the point of unit testing code that touches IO and is supposed to be mocked? You’d just be writing mocks and testing mocks.
Don’t write functions like this:
fetchNumberFromIOAndAdd2
Do write two functions like this:
fetchNumberFromIO
addTwo(x)
The later function can be unit tested. The former can’t be unit tested and there’s no point in unit testing it. The former function is what lives in the imperative shell.
What you see in a lot of books are authors promoting patterns like this:
addTwo(database object)
This kind of pattern forces you mock by coupling IO with logic.
suppose you are writing some software that interfaces with personal finance SAAS. except you want your users to be able to interact with more than one. so you sure as shit better be testing that your abstractions are making the correct calls, especially if customer money is involved.
you also really want to be testing for unreliability, for example, if your saas call is taking too long, you dont double tap, if a tx looks like it fails but it didnt actually, you know that on the second attempt, etc, etc.
"the shell is untestable" is not an acceptable answer.
and you cant wriggle out of this by saying "well that's an integration test". you said in your op "don't write mocks". thar is terrible advice in general. im generally a big fan of imperative shell functional core, but you MUST mock your shell, especially if absolute correctness is required (people's lives or money at stake)
No. You completely missed what I said. I differentiated between unit tests and integration tests. Your imperative shell is not testable in the context of unit tests. However you must test it via integration tests.
Mocks are a utility for unit tests hence the topic of this post. We are talking about unit tests and mocking aspects of the unit test that can’t be tested via unit tests.
Integration tests are another topic. With integration tests or end to end tests you test everything. You don’t mock. Mocking is not really a valid concept in the context of integration tests.
that is not what you said. you wrote: "Don’t code with mocks period"
> I differentiated between unit tests and integration tests.
you did not. your original comment doesnt mention integration tests at all. i quote:
"Structure your code such that it has a functional core and imperative shell. All logic is unit testable (functional core). All IO and mutation Is not unit testable (imperative shell).
Mocks come from a place where logic is heavily intertwined with IO. It means your code is so coupled that you can’t test logic without touching IO.
IO should be a dumb layer as much as possible. It should be Extremely general and as simple as fetching and requesting in the stupidest way possible. Then everything else should be covered by unit tests while IO is fundamentally not unit testable.
If you have a lot of complex sql statements or logic going on in the IO layer (for performance reasons don’t do this if you can’t help it) it means your IO layer needs to be treated like a logical layer. Make sure you have stored procedures and unit tests written in your database IO language. You code should interface only with stored procedures as IO and those stored procedures should be tested as unit tests in the IO layer
> Mocking is not really a valid concept in the context of integration tests.
what the fresh hell are you talking about? its a common practice to set up mocks (using dependency injection with expects) for an external API call in an integration test. in 10 years of professional software development I've mocked in integration tests at five empolyers.
Rarely used unless necessary is what I meant. The point of the integration test is to avoid mocking and I don’t want to argue a pedantic point with you and I also don’t like your attitude so I’m ending this section of the thread don’t bother replying.
hey man the least you could say us "yeah i wrote something that I didn't mean, my bad" when i called you out. instead you dug in, moved goalposts, and claimed you wrote shit you didn't.
it started with literally bad professional advice. junior developers reading these forums need to not have professional habits influenced by the sort of behavior on display by you here.
and a ChatGPT transcript? really? just do a simple search, you will find tons of articles advocating for mocking external apis in integration tests.
Ok. My bad. I have no problem saying that. But I cut off the convo because I don’t like your attitude. Didn’t like this reply either. Let’s just end it here I have nothing further to say to you and I don’t care about your technical opinions. Good day.
The jargon around all this stuff is terrible. What I would probably write is, like, a fake version of that API, one which runs out of process from the library or test, and which can be "programmed" from the test set up -- "expect these requests, serve these responses", etc.
Is that a mock? I don't know. It sure does resemble one, but it doesn't use dependency injection or other runtime tricks which I'd associate with classic "mocks"; it's just another program you write to support the tests. The imperative shell doing I/O needs to be configurable to support this (think a "widgets API base URL"), but that's it.
Yes that still a mock. And thr point is to structure your code in such a way that there is little to no value left in testing that API call independently, because all of the related logic has been tested in a unit test.
Once you've done that, you can just run a simple integration or end-to-end test to cover that API. You never need to manually mock it (like you suggested), or use a mocking tool.
that is a fake. A mock is a device for verifying you called the expected funchtions. A fake emulates the api. Importantly if you change funcioms the fake will still pass if they mean the same - think of write which might take a buffer+length [C style], a list or a string (often there would be more than 10 variations)
following in that theme, just doing some digging [1] turns up the following for those that might have been confused like i was:
Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test.
Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.
Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.
The jargon is terrible because I think you misunderstood my point. You don’t mock period is my philosophy but your talking about mocking an entire api here.
Mocks come from a place where logic is heavily intertwined with IO. It means your code is so coupled that you can’t test logic without touching IO.
IO should be a dumb layer as much as possible. It should be Extremely general and as simple as fetching and requesting in the stupidest way possible. Then everything else should be covered by unit tests while IO is fundamentally not unit testable.
If you have a lot of complex sql statements or logic going on in the IO layer (for performance reasons don’t do this if you can’t help it) it means your IO layer needs to be treated like a logical layer. Make sure you have stored procedures and unit tests written in your database IO language. You code should interface only with stored procedures as IO and those stored procedures should be tested as unit tests in the IO layer.