A cynical way to look at it is that we're pretty close to the ultimate limits of what LLMs can do and now the stake holders are looking at novel ways of using what they have instead of pouring everything into novel models. We're several years into the AI revolution (some call it a bubble) and Nvidia is still pretty much the only company that makes bank on it. Other than that it's all investment driven "growth". And at some point investors are gonna start asking questions...
A very simple observation, our brains are vastly more efficient. Obtaining vastly better outcomes from lesser input. This evidence means there's plenty of room for improvement without a need to go looking for more data. Short term gain versus long term gain like you say, shareholder return.
More efficiency means more practical/useful applications and lower cost as opposed to bigger model which means less useful (longer inference times) and higher cost (data synthesis and training cost).
At a fundamental level, brains don’t operate on floating point numbers encoded in bits.
They have chemicals to facilitate electrochemical reactions which can affect how they respond to input. They don’t throw away all knowledge of what they just said. They change continuously, not just in fixed training loops. They don’t operate in turns.
I could go on.
Honestly the number of people who just heard “learning,” “neural networks,” and “memory” and assume that AI must be acting like a biological brain is insane.
Fundamentally and physically are two different things. A logic gate is a logic gate if it's in neurons or silicon.
Are abacus and calculators solving different things? No.
You're proving my point, things like them changing continuously are exactly what I mean when I say the brain is more efficient. Where there's a will theres a way and our brains are evidence that it can be done.
You're saying that because two different objects can to solve the same problem, they must work the same way.
An abacus and a calculator were both made to solve relativly simple math problems, so they must work in the same way, right?
And apple and an orange are ways to store sugar for plants, so they must be the same thing, right?
No. That's not how any of this works.
An abacus and a calculator are two different tools that solve the same problem.
They don’t act like each other just because the abstract outcome is the same
> You're proving my point, things like them changing continuously are exactly what I mean when I say the brain is more efficient.
I don't see how that proves that neural networks act like brains.
It's also not just a difference in terms of efficiency, it's the fundamental way that statistical models like neural networks are trained. Every time their trained, it's a brand new model, unlike a brain, which is still the same brain.
Also, neural networks and brains were NOT made to solve the same problems... even if your argument made any sense, it doesn't fit here.
No I'm not saying they must work the same way. I'm saying it's evidence there is a more efficient way as they both solve the same problem and one is more efficient (in truth both are more efficient in different areas).
At an abstract level they can be doing the same thing. What does a simulator do?
Think a little further yes, currently it's a brand new model each time but why will it be this way forever? Its an engineering problem one that we can solve and the brain is evidence it can be done.
Neural networks were originally inspired by the brain. Yes, they've deviated but there's absolutely no reason they can't take further inspiration.
So you’re just abstracting everything to the point where everything is a “something solver” and if two things can solve the same something, one must be a better version of the other?
Abstracting everything to the point of meaninglessness isn’t a worthwhile exercise.
