The global economy has depended on finessing quasi-stochastic black-boxes for many years. If you have ever seen a cloud provider evaluate a kernel update you will know this deeply.
For me the potential issue is: our industry has slowly built up an understanding of what is an unknowable black box (e.g. a Linux system's performance characteristics) and what is not, and architected our world around the unpredictability. For example we don't (well, we know we _shouldn't_) let Linux systems make safety-critical decisions in real time. Can the rest of the world take a similar lesson on board with LLMs?
Maybe! Lots of people who don't understand LLMs _really_ distrust the idea. So just as I worry we might have a world where LLMs are trusted where they shouldn't be, we could easily have a world where FUD hobbles our economy's ability to take advantage of AI.
Yes, but if I really wanted, I could go into a specific line of code that governs some behaviour of the Linux kernel, reason about its effects, and specifically test for it. I can't trace the behaviour of LLM back to a subset of its weights, and even if that were possible, I can't tweak those weights (without training) to tweak the behaviour.
No, that's what I'm saying, you can't do that. There are properties of a Linux system's performance that are significant enough to be essentially load-bearing elements of the global economy, which are not governed by any specific algorithm or design aspect, let alone a line of code. You can only determine them empirically.
Yes there is a difference in that, once you have determined that property for a given build, you can usually see a clear path for how to change it. You can't do that with weights. But you cannot "reason about the effects" of the kernel code in any other way than experimenting on a realistic workload. It's a black box in many important ways.
We have intuitions about these things and they are based on concrete knowledge about the thing's inner workings, but they are still just intuitions. Ultimately they are still in the same qualitative space as the vibes-driven tweaks that I imagine OpenAI do to "reduce sycophancy"
For me the potential issue is: our industry has slowly built up an understanding of what is an unknowable black box (e.g. a Linux system's performance characteristics) and what is not, and architected our world around the unpredictability. For example we don't (well, we know we _shouldn't_) let Linux systems make safety-critical decisions in real time. Can the rest of the world take a similar lesson on board with LLMs?
Maybe! Lots of people who don't understand LLMs _really_ distrust the idea. So just as I worry we might have a world where LLMs are trusted where they shouldn't be, we could easily have a world where FUD hobbles our economy's ability to take advantage of AI.