> You've read too much Tyler Cowen and imagine AGI only adding .5% to GDP per year

I did not know who Tyler Cowen is.

I'm not making any assumptions about how much growth we would get from machine intelligence, I'm simply stating that the value you can expect from that whole new sector is gonna be much closer to the cost in silicon and energy to provide it than the former value of the human labor it replaces (history reinforces that view).

In my view, building machine intelligence makes you into a (groundbreaking, innovative) commodity provider medium-term at best, not a god.

I'm personally extremely skeptical of singularity scenarios in general.

Most models of it implicitly assume a sub-exponential, somewhat proportional or even constant relation between achieved "intelligence/utility" and required energy (thanks to "self improvement"), which is an extremely flawed assumption in my view. But what kind of singularity do you actually believe in?

I do agree that machine intelligence is a significant existential risk for our species, though.

>I'm simply stating that the value you can expect from that whole new sector is gonna be much closer to the cost in silicon and energy to provide it than the former value of the human labor it replaces (history reinforces that view).

Again lacking imagination. This not only replaces human labor but enables entirely new types of endeavors. Did ICs simply replace the value of Vacuum tubes? Did the Internet simply replace the value of Snail Mail or Libraries? You just aren't framing the problem correctly.

>I'm personally extremely skeptical of singularity scenarios in general.

Naturally...

>Most models of it implicitly assume a sub-exponential, somewhat proportional or even constant relation between achieved "intelligence/utility" and required energy (thanks to "self improvement"), which is an extremely flawed assumption in my view.

Until you get close to the efficiency of a human brain this is a total nothingburger and AI undergoing RSI has enormous room to improve purely within the algorithmic space. Its virtually certain that an AGI could run on average consumer hardware.

> This not only replaces human labor but enables entirely new types of endeavors.

I completely agree with that! I think we are talking past each other a bit, because to me all your examples seem supportive of my main point: None of those revolutions in tech really made their inventors "break capitalism", but instead the "commodity providers" of the digital age made good business together with all the new industry that sprouted up and used the tech...

> Its virtually certain that an AGI could run on average consumer hardware.

Sure. But then what? I can totally see you getting the equivalent of 10 human white-collars for a kilowatt within a decade or two. (Sidenote: I don't believe we'll ever get past the 0.1W/human-equivalent brain in efficiency, and even that is an optimistic estimate that I would not stake anything on).

You might be able to further improve those systems, getting up to a hundred or even a thousand clever humans-- a veritable army of consultants/contractors at everyones fingertips.

And thats great! But an army of consultants for almost free is NOT a singularity-- it is extremely doubtful to me that you are even gonna get "the sum of its parts" out of such an arrangement; expecting such a construct to be capable of rapid self improvement is extremely optimistic (just compare real-life armies of consultants, which are typically agreed to be not capable of rapid self improvement, even if you throw exponentially increasing amounts of them at it).

Just like real life armies of consultants, we will have to be careful that the work is actually aligned with our interests though, and having many more consultants than actual people is dangerous business already...