doesn't it seem like these models are getting to the point where even conceiving their training and development is less and less possible for the general public?
I mean, we already knew only a handful of companies with capital could train them, but at least the principles, algorithms, etc. were accessible to individuals who wanted to create their own - much simpler - models.
it seems that era is quickly ending, and we are entering the era of truly "magic" AI models that no one knows how they work because companies keep their secret sauces...
Recent developments like V3, R1 and S1 are actually clarifying and pointing towards more understandable, efficient and therefore more accessible models.
I don't think it's realistic to expect to have access to the same training data as the big labs that are paying people to generate it for them, but hopefully there will be open source ones that are still decent.
At the end of the day current O1-like reasoning models are still just fine-tuned LLMs, and don't even need RL if you have access to (or can generate) a suitable training set. The DeepSeek R1 paper outlined their bootstrapping process, and HuggingFace (and no doubt others) are trying to duplicate it.
We have been in the 'magic scaling' era for a while now. While the basic architecture of language models is reasonably simple and well understood, the emergent effects of making models bigger are largely magic even to the researchers, only to be studied emperically after the fact.
I mean, we already knew only a handful of companies with capital could train them, but at least the principles, algorithms, etc. were accessible to individuals who wanted to create their own - much simpler - models.
it seems that era is quickly ending, and we are entering the era of truly "magic" AI models that no one knows how they work because companies keep their secret sauces...