I've had a very good experience with pg queuing. I didn't even know `skip locked` was a pg clause. That would have... made the experience even better!
I am afraid I've moved to a default three-way architecture:
- backend autoscaling stateless server
- postgres database for small data
- blobstore for large data
it's not that other systems are bad. its just that those 3 components get you off the ground flying, and if you're struggling to scale past that, you're already doing enormous volume or have some really interesting data patterns (geospatial or timeseries, perhaps).
Quite correct. Made an error. Carryover from a prior job where we mucked with weather data... I was thinking more along the lines of raster geo datasets like sat images, etc. Each pixel represents one geo ___location, with a carrying load of metadata, then a timeseries of that snapshot & md, so timeseries-raster-geomapped data basically.
I don't remember anymore what that general kind of datatype is called, sadly.
I am afraid I've moved to a default three-way architecture:
- backend autoscaling stateless server
- postgres database for small data
- blobstore for large data
it's not that other systems are bad. its just that those 3 components get you off the ground flying, and if you're struggling to scale past that, you're already doing enormous volume or have some really interesting data patterns (geospatial or timeseries, perhaps).