What is the worst that can happen?

Seriously, I think I might know (a fire) but I do not.

Is there any possibility these can go ban! and spread nasty stuff around the neighbourhood?

I doubt they can go thermonuclear bang! but I am not a physicist.

What is the worst than can happen?

Well, neutron bombardment from fusion means you get some radioactive isotopes. It's not as bad as spent fuel but still something to avoid.

But that doesn't really matter. This is expensive technology you don't want to blow up. Yes, more expensive than rockets.