An interesting development to look forward to will be hooking them up to search engines. The proprietary models already do this, and the open equivalents are not far behind; the recent Qwen models are not as great at knowledge, but are some of the best at agentic functionality. Exciting times ahead!
Many tasks that one might want to give a model end up implicitly including search as a subtask. For example, "plan me a trip to Santiago" obviously requires the model to understand details about the real city of Santiago. Less obviously, "write me a Python script to do ..." requires they understand APIs, libraries, etc., the same things you might ask a search engine to pull up. The tasks which do not require a coherent + mostly-correct exterior-world-model are relatively few -- text processing (e.g. "proofread this") is a big one; calculation tasks fit, but LLMs are also bad at those.
Exactly, I think all those base models should be weeded out from this nonsense, kardashian-like labyrinths of knowledge complexities that just makes them dumber by taking space and compute time. If you can google out some nonsense news, it should stay there in search engines for retrieval. Models should be good at using search tools, not at trying to replicate their results. They should start from logic, math, programming, physics and so on, similar to how education system is suppose to equip you with. IMHO small models can give this speed advantage (faster to experiment ie. with parallel diverging results, ability to munch through more data etc). Stripped to this bare minimum they can likely be much smaller with impressive results, tunable, allow for huge context etc.
