Perhaps start here:

https://www.nngroup.com/articles/overuse-of-overlays/

Then ux.stackexchange.com will undoubtedly have some useful pointers as well....

Although some tailored google searches will also help.

Back in my UX-only days, I used NN Group a lot in supplementing my own A/B testing - it was a bittersweet relationship, as they provide a suite of analytics-based trackers on many of the websites I visit.

Read "The Humane Interface". The short answer is, almost never. Try to avoid them.

I am so glad to see someone write this. I've worked with too many designers who stack modals onto modals.

Agree. I figured the person asking the question was being amusing.

Here's one of many examples I found on the first page of Google.

https://www.smashingmagazine.com/2009/05/modal-windows-in-mo...