But what ORMs encourage you to do is to load a bunch of
objects and then use procedural code to do the real thing.
The exact same thing happens when you directly express the query in SQL. You load a bunch of objects and post-process them, to do what SQL can't do. Unless you count the various languages you can use in stored procedures in RDBMS's, but those aren't set-oriented SQL either.
If I want to do something that SQL can't do I don't use SQL. Using SQL is not an end in itself.
My rule of thumb is pretty simple. I use whatever takes fewer and/or simpler lines of code unless it's a lot slower.
The raging scalability debate is a different matter. I'm sure if you're Google there are good reasons to write more lines of code in order to scale better. I'm not Google so I can prioritize productivity over scalability.