> you need to get some data from a few sources (databases etc) do some computation on each set and then do some sort of mapping to get the resulting set. You may want those computations to run in parallel and idealy you'd like to start mapping as soon as each computation function starts producing results.
This definitely sounds like a typical situation for handling the job in a queue (plus with an async library like spatie/async to retrieve data in parallel), though I see the advantages and convenience of using a natively-async language here.
An example (not the best one maybe but I'm not the right guy to come up with good examples immediately) from a year ago:
We have to integrate company M and W (one is a marketplace the other one is a big store, think Walmart or something).
Company W mostly uses software similar to SAP-whatever and have a small team resposible for building 'helper' services in place where SAP can't do the job.
Company W can't communicate with M in any other way except through your generic http API they provide.
So every now and then M has to send a request to W and W has to prepare a pretty large XML response. (obviously the data is split in some way and we are not talking about tens of GBs but even so W has has to fetch the data from more than one data source, process it and send it)
In some cases you can simply have a cache\a view\whatever for this (so you only have to fetch prepared data) but in some cases you can not.
PS: this is if we are talking about HTTP communication. Or maybe some real-time communication where you can't really respond with "hey, we are getting your data prepared so just wait for a bit and re-request it with this nice jobID at a later time"
Well, for that use case, it would be typical in my industry for party A to send a callback URL to party B, so that B can POST the required information back to A after doing the multi-step processing. It's not really a done thing to make a synchronous HTTP request and wait say a minute or more for the response. Maybe that's just different expectations in different industries, though.
This definitely sounds like a typical situation for handling the job in a queue (plus with an async library like spatie/async to retrieve data in parallel), though I see the advantages and convenience of using a natively-async language here.