Excellent point, especially with the tons of semi rich people. Its often overlooked how young SV really is. The first gold rush was in semiconductors, and the people that made money there invested in software and the cycle continued. What essentially happened is that these "semi-rich" people, as you say, were perfect investors in new startups:

1. They had the money to get the startup going 2. They had the technical expertise to (mostly) select good investments and cut through the bullshit. 3. They had experience with startups themselves; they were excellent mentors and had industry connections besides.

Now the system is in place, it seems rather magical.