There are some popular textbooks on computer architecture; Patterson's stuff is good, but there are better texts. Some techno-hobbyist sites (like Ars Technica) do CPU reviews that get into various microarchitectures. Datasheets from processor manufacturers are free and often have interesting details.
Going after things at a really low level, books like The Art of Electronics take you from transistors (and through things you probably don't care about, like power supplies and amplifiers) to digital electronics, to simple computer system design. You can probably find value by skimming the component-level stuff, skipping the goopy analog chapters, and just reading the digital electronics sections. (This is more or less how I started with personal computers in 1974 or so).
Can you give some examples and why/in what respect they are better than (David) Patterson's books? Or are those "better texts" the Ars Technica CPU reviews and manufacturer datasheets that you mentioned in the subsequent sentence?
I liked John Shen's Modern Processor Design and Noam Nisan's The Elements of Computing Systems.
The websites I mentioned have hobbyist-level details that are pretty interesting (but in general won't teach you much about things like concurrency hazards and branch prediction and so forth).
Going after things at a really low level, books like The Art of Electronics take you from transistors (and through things you probably don't care about, like power supplies and amplifiers) to digital electronics, to simple computer system design. You can probably find value by skimming the component-level stuff, skipping the goopy analog chapters, and just reading the digital electronics sections. (This is more or less how I started with personal computers in 1974 or so).