The core of TeX (written by Knuth) is actually quite small for what it does - approximately 24kLOC when translated from Pascal to C. The "TeX systems" that are around today are huge because of all the other things that dwarf it, and honestly I don't think everyone needs all or even a majority of that - I know that the parts of TeX that I use to turn a .tex into a .ps can fit on a single floppy.

There's also the fact that LaTeX is often bundled in, and that is another "monstrosity" in itself. (I exclusively use plain TeX, so I have no need for that either.)