> I think the only way to start an AI chip company is to start with the software. The computing in ML is not general purpose computing. 95% of models in use today (including LLMs and image generation) have all their compute and memory accesses statically computable.
Agree with this insight. One thing Nvidia got right was a focus on software. They introduced CUDA [1] back in 2007 when the full set of use cases for it didn't seem very obvious. Then their GPUs had Tensor cores, and more complementary software like TensorRT to take full advantage of them post deep learning boom.
Right as Nvidia reported insane earnings beat too [2]. Would love more players in this space for sure.
I know the founders of an AI chip company that taped out and got working chips on their first go. They got their chip done, it’s pretty solid. Chip has great perf and is super power efficient, a solid delivery. I knew they'd nail it and they did.
The SW story is a train wreck, though. The problem basically was that they couldn’t hire any good SW people. As I said I know the founders. They are both genuinely decent guys, they put their own money in so they have some (well, minimal) skin in the game, and they know a ton of expert-level embedded and systems coders with between 20 and 40 years of hard core experience. As far as I can tell, they weren't really able to get anyone that we know in common to join. I certainly did not, and no one I know did either. Last I heard they'd had to hire third choice guys in Europe to do the work and it wasn't going well.
There's a pretty good reason for it, and it comes down to a sociological problem. HW people don’t value SW people. It's just basically true and has been true everywhere I've looked. Maybe if you're doing a system (like a router or maybe a drone) then the HW people will begrudgingly admit that the SW is a major part of the delivery, but that isn't true for chip companies (including chips-on-reference-boards).
You can rest assured that at a chip company, all of the high comp people in the company are going to be on the ASIC team and the SW team will never be on the same tier. The argument is always the same, no matter how many times it bites the companies on the ass and sends them careening into the dumpster: “yes, but the chip without SW is the chip! we can buy SW, if we have to. SW without the chip has zero value.”
Almost every chip company ends up like that, and the kind of low level, experienced SW people that work in the space know to avoid them and work at systems companies instead.
As far as I've been able to determine, with _maybe_ the exception of Cerebras - maybe - this is the situation that has played out at all of the 201x AI chip companies. They get founded by ASIC guys, most of whom have more than a small chip on their shoulder about the relative value of ASICs-vs-SW. These guys are all ex-SGI, ex-Sun, ex-Google, ex-Nvidia, ex-Intel HW guys who saw SW people making a lot more, not just in broader industry terms over the last few years, but at hardware-focused companies. In general, ASIC guys make less than SW guys unless they are the very narrow set of top level architects. IMHO from a value creation standpoint, that is _super unfair_ and I am not here to justify it, but it is how it is. The result poisons ASIC companies. SW people who know what needs to be don't won't go to them most of the time, for good reason, and so they fail.
So I will say, given that, starting with SW first is brilliant.
I also know the founder of an AI chip company (ex-AMD), they taped out in 2022, got working chips on the first try. Miraculously, they hired a good software team, some even with previous compiler experience. In their brochure they write things like:
> In 2022, FuriosaAI remained the only startup to submit results in MLPerf Inference... This time, through purely enhancements in the compiler, our team was able to double the performance on the exact same silicon.
were the SW folks offered compensation commensurate with their low perceived status? like HW gets C/cofounder, SW gets early employee package. Because that's a different story about money and ownership. of course good people reject wage labor.
I can only speak for myself, but their offer was underwhelming and I was easily one of the first five people they called. I’ve founded and exited companies myself, so when I walked them through why the founding SW engineer offer wasn’t worth my time giving the risk profile, and laid out there size and type of SW engineers they were going to need. They understood, but they also trotted out the “but we really have to make sure we hire the right chip guys.”
Fair enough. I’m old so these conversations are never personal. I tried to help them by steering younger, less experienced but very high potential engineers toward them, but in the end they failed utterly to put together a viable SW team.
I don’t know any of their investors in terms that would let me ask, but I do know a bunch who passed, and most of them had concluded that it would end up just being more silicon on a crowded market. Being 10-20x better than nvidia isn’t the point if the market is about to be flooded with a dozen other chips against whom you are maybe 1.5-3x better. Without nailing the go to market needs, which means “make the stuff people have work, don’t make the customer learn new” etc. you have nothing. That’s all a Sw problem.
It’s actually worse, because the engineers in the space are actually pretty bad. A lot of what they have actually barely works to begin with, being a bunch of cobbled together python frameworks of dubious engineering quality and all of the hassle of the ecosystem. So the amount of mental space for “different” is almost zero even if you ignore that they’ve been burned (AMD) before, which you can’t.
Agree with this insight. One thing Nvidia got right was a focus on software. They introduced CUDA [1] back in 2007 when the full set of use cases for it didn't seem very obvious. Then their GPUs had Tensor cores, and more complementary software like TensorRT to take full advantage of them post deep learning boom.
Right as Nvidia reported insane earnings beat too [2]. Would love more players in this space for sure.
[1] - https://en.wikipedia.org/wiki/CUDA [2] - https://www.cnbc.com/2023/05/24/nvidia-nvda-earnings-report-...