I know a kid whose throat swelled up, couldn't breathe for several minutes, and suffered brain damage after a one time incident of eating peanuts... Apparently, non-negligible (over 1% [1]) portion of the population reacts this way.
It depends on how much risk averse you are - I know that, for me, trying out psychedelics is not worth the 1% risk of going into psychotic breakdown, being admitted to psychiatric ward or even messing myself up permanently. For others it might be worth it.
Lesson for everybody: Don't eat nuts!
[1] http://www.mountsinai.org/about-us/newsroom/press-releases/r...
Your argument against psychedelics is not persuasive in the manner presented.