Most "side effects" of psychedelics occur in the short-term, while under the influence of the substance. I will focus on LSD, as it's the substance I am most familiar with. The side effects are mostly what people call the "body load". Most notably there is vasoconstriction, increased body temperature, increased blood pressure and heart rate, and of course, the possibility of a bad trip or triggering latent schizophrenia/psychosis or bipolar disorder in those with a predisposition.

Edit: There are also interactions with certain medications that can result in things like seizures, and very rarely people report "flashbacks" or HPPD.

Overall, in my extensive experimentation both with traditional psychiatric medications and with LSD... I think the side effects of the substance are overstated. I have far worse reactions to psychiatric medication. (Edit: your mileage may vary of course, different people will react differently to substance, illicit or otherwise - I don't mean to downplay the negative experiences that DO occur with psychedelic usage)