Just to get things right. The big AI LLM hype started end of 2022 with the launch of ChatGPT, DALL-E 2, ....
Most people in society connect AI directly to ChatGPT and hence OpenAI. And there has been a lot of progress in image generation, video generation, ...
So I think your timeline and views are slightly off.
> Just to get things right. The big AI LLM hype started end of 2022 with the launch of ChatGPT, DALL-E 2, ....
GPT-2 was released in 2019, GPT-3 in 2020. I'd say 2020 is significant because that's when people seriously considered the Turing test passed reliably for the first time. But for the sake of this argument, it hardly matters what date years back we choose. There's been enough time since then to see the plateau.
> Most people in society connect AI directly to ChatGPT and hence OpenAI.
I'd double-check that assumption. Many people I've spoken to take a moment to remember that "AI" stands for artificial intelligence. Outside of tongue-in-cheek jokes, OpenAI has about 50% market share in LLMs, but you can't forget that Samsung makes AI washing machines, let alone all the purely fraudulent uses of the "AI" label.
> And there has been a lot of progress in image generation, video generation, ...
These are entirely different architectures from LLM/chat though. But you're right that OpenAI does that, too. When I said that they don't stray much from chat, I was thinking more about AlexNet and the broad applications of ML in general. But you're right, OpenAI also did/does diffusion, GANs, transformer vision.
This doesn't change my views much on chat being "not seeing the forest for the trees" though. In the big picture, I think there aren't many hockey sticks/exponentials left in LLMs to discover. That is not true about other AI/ML.
>In the big picture, I think there aren't many hockey sticks/exponentials left in LLMs to discover. That is not true about other AI/ML.
We do appear to be hitting a cap on the current generation of auto-regressive LLMs, but this isn't a surprise to anyone on the frontier. The leaked conversations between Ilya, Sam and Elon from the early OpenAI days acknowledge they didn't have a clue as to architecture, only that scale was the key to making experiments even possible. No one expected this generation of LLMs to make it nearly this far. There's a general feeling of "quiet before the storm" in the industry, in anticipation of an architecture/training breakthrough, with a focus on more agentic, RL-centric training methods. But it's going to take a while for anyone to prove out an architecture sufficiently, train it at scale to be competitive with SOTA LLMs and perform enough post training, validation and red-teamint to be comfortable releasing to the public.
Current LLMs are years and hundreds of millions of dollars of training in. That's a very high bar for a new architecture, even if it significantly improves on LLMs.
ChatGPT was not released to the general public until November 2022, and the mobile apps were not released until May 2023. For most of the world LLM's did not exist before those dates.
This site and many others were littered with OpenAI stories calling it the next Bell Labs or Xerox PARC and other such nonsense going back to 2016.
And GPT stories kicked into high gear all over the web and TV in 2019 in the lead-up to GPT-2 when OpenAI was telling the world it was too dangerous to release.
Certainly by 2021 and early 2022, LLM AI was being reported on all over the place.
>For most of the world LLM's did not exist before those dates.
Just because people don't use something doesn't mean they don't know about it. Plenty of people were hearing about the existential threat of (LLM) AI long before ChatGPT. Fox News and CNN had stories on GPT-2 years before ChatGPT was even a thing. Exposure doesn't get much more mainstream than that.
As another proxy, compare Nvidia revenues - $26.91bln in 2022, $26.97bln in 2023, $60bln 2024, $130bln 2025. I think it's clear the hype didn't start until 2023.
You're welcome to point out articles and stores before this time period "hyping" LLM's, but what I remember is that before ChatGPT there was very little conversation around LLM's.
Since I almost considered getting a paid AI service, with Kagi I get the freedom to choose different models + I get a nice interface for search, translate, ...
With Kagi the AI service also does not know who I am.
I'm quite happy so far, also the Android app works fine. 95% of the time I don't open a browser but instead the app to answer my questions.
The privacy feature somehow did not work in my firefox browser yet.
I think it actually depends what you define as "pixel". Sure, the pixel on your screen emits light on a tiny square into space.
And sure, a sensor pixel measures the intensity on a tiny square.
But let's say I calculate something like:
# samples from 0, 0.1, ..., 1
x = range(0, 1, 11)
# evaluate the sin function at each point
y = sin.(x)
Then each pixel (or entry in the array) is not a tiny square. It represents the value of sin at this specific ___location. A real pixelated detector would have integrated sin from `y[u] = int_{u}^{u + 0.1} sin(x) dx` which is entirely different from the point wise evaluation before.
So for me that's the main difference to understand.
For reference, I think a common approximation is one token being 0.75 words.
For a 100 page book, that translates to around 50,000 tokens. For 1 mil+ tokens, we need to be looking at 2000+ page books. That's pretty rare, even for documentation.
It doesn't have to be text-based, though. I could see films and TV shows becoming increasingly important for long-context model training.
Synthetic data requires a discriminator that can select the highest quality results to feed back into training. Training a discriminator is easier than a full blown LLM, but it still suffers from a lack of high quality training data in the case of 1M context windows. How do you train a discriminator to select good 2,000 page synthetic books if the only ones you have to train it with are Proust and concatenated Harry Potter/Game of Thrones/etc.
Despite listing all presently known bats, the majority of "list of chiropterans" byte count is code that generates references to the IUCN Red List, not actual text. Most of Wikipedia's longest articles are code.
I changed recently from X1 Yoga Gen 2 to X1 Yoga (2-in-1) Gen 9.
The decline in quality is so clearly visible, the external pen is poorly designed and died after 6 months.
The hinges are as loose as in a 600$ laptop. The whole thing makes cracking noises under slight stress. Also, Linux compatibility is poor, my Webcam still does not work. Battery lifetime is a joke too.
Lenovo made this Laptop worse than 7 years ago, and it's their top line model for > 2000$. It's such a shame and sad to see. There's no very good alternative with integrated touchscreen and stylus.
To be fair yoga models always had issues, never fully Linux compatible and never had a long lifetime. Not sure what this series is supposed to be, but I wouldn't recommend it.
Yeah. I never really even considered Lenovo's new additions to the ThinkPad line. I focused on the machines that were continued from the original THinkpad series (T, R and X).
> AI can solve math puzzles better than 99.9% of population
I've studied electronic engineering and then switched to software engineering as a career, and I can say the only time I've been exposed to math puzzles were in academic settings. The knowledge is nice and help with certain problem solving, but you can be pretty sure I will reach out to a textbook and a calculator before trying to brute-force one such puzzle.
The most important thing in my daily life is understand the given task, do it correctly, and report about what I've done.
Yep this exactly. If I ever feel that I'm solving a puzzle at work, I stop and ask for more information. Once my task is clear, I start working on it. Learned long ago that puzzle solving is almost always solving the wrong problem and wasting time.
Maybe that's where the disconnect comes from. For me understanding comes before doing. And coding for me is doing. There may be cases when I assumes that my understanding was complete and the feedback (errors during compiling and testing) told me I'm wrong, but that just triggers another round of research to seek understanding.
Puzzle solving is only for when information are not available (reverse engineering, closed systems,...) but there's a lot of information out there for the majority of tasks. I'm amazed when people spend hours trying to vibe code something, where they could spend just a few minutes reading about the system and comes up with a working solution (or find something that already works).
While I do see this argument made quite frequently, doesn't any professional effort center in procedures employed particularly to avoid mistakes? Isn't this really the point of professional work (including professional liabilities)?
> LLMs can write boilerplate code at least an order of magnitude faster than I can.
This is my biggest fear with everyone adopting LLMs without considering the consequences.
In the past, I used "Do I have to write a lot of boilerplate here?" as a sort of litmus test for figuring out when to refactor. If I spend the entire day just writing boilerplate, I'm 99% sure I'm doing the wrong thing, at least most of the time.
But now, junior developers won't even get the intuition that if they're spending the entire day just typing boilerplate something is wrong, instead they'll just get the LLM to do it and there is no careful thoughts/reflections about the design and architecture.
Of course, reflection is still possible, but I'm afraid it won't be as natural and "in your face" which kind of forces you to learn it, instead it'll only be a thing for people who consider it in the first place.
The question to ask yourself is what value one gets from that refactor. Is the software faster? Has more functionality? Cheaper to operate? Reduce time to market. These would be benefits to the user, and I'd venture to say the refactor does not impact the user.
The refactor will impact the developer. Maybe the code is now more maintainable, or easier to integrate, or easier to test. But this is where I expect LLMs will make a lot of progress - - they will not need clean, well structured code. So the refactor, in the long run, is not useful to a developer with an LLM sidekick.
I agree with you but as a thought exercise: does it matter if there is a lot of boilerplate if ultimately the code works and is performant enough?
Fwiw, most of the time I like writing code and I don't enjoy wading through LLM-generated code to see if it got it right. So the idea of using LLMs as reviewers resonates. I don't like writing tests though so I would happily have it write all of those.
But I do wonder if eventually it won't make sense to ever write code and it will turn into a pastime.
> I agree with you but as a thought exercise: does it matter if there is a lot of boilerplate if ultimately the code works and is performant enough
Yeah it matters because it is almost guaranteed that eventually a human will have to interact with the code directly so it should still be good quality code
> But I do wonder if eventually it won't make sense to ever write code and it will turn into a pastime
Even the fictional super-AI of Star Trek wasn't so good that the engineers didn't have to deeply understand the underlying work that it produced.
Tons of Trek episodes deal with the question of "if the technology fails, how do the humans who rely on it adapt?"
In the fictional stories we see people who are absolute masters of their ___domain solve the problems and win the day
In reality we have glorified chatbots, nowhere near the abilities of the fictional super-AI, and we already have people asking "do people even need to master their domains anymore?"
I dunno about you but I find it pretty discouraging
Most people in society connect AI directly to ChatGPT and hence OpenAI. And there has been a lot of progress in image generation, video generation, ...
So I think your timeline and views are slightly off.
reply