ARM is pretty new as a remotely high-performance thing. x86 is used a bit; the Hubble space telescope has a radiation hardened 486, for instance (originally a 386). A good few spacecraft use radiation hardened PowerPCs, M68Ks, and other common designs. Even that ESA chip isn't anything hugely custom; it's a SPARC-V8 variant.
But it is certainly custom hardware and a custom OS, because you need radiation-hardened chips (even for earth-satellites) and extreme reliability.
[1] http://en.wikipedia.org/wiki/LEON