Let's be honest here, a lot of HN users have a conflict of interest on this topic. AI entrepreneurs trying to get rich from LLMs benefit from LLMs being open source. But the downside risk from e.g. bioweapons is spread across all of society.
It's the same sort of asymmetrical cost/benefit that tobacco companies and greenhouse gas emitters face. Of course if you went to an online forum for oil companies, they'd be hopping mad if they're prevented from drilling, and dismissive of global warming risks. It's no different here.
> But the downside risk from e.g. bioweapons is spread across all of society.
It gets old hearing about these "risks" in the context of AI. It's just an excuse used by companies to keep as much power as possible to themselves. The real risk is AI being applied in decision making where it affects humans.
I am concerned with AI companies keeping all the power to themselves. The recent announcement from the OpenAI board was encouraging on that front, because it makes me believe that maybe they aren't focused on pursuing profit at all costs.
Even so, in some cases we want power to be restricted. For example, I'm not keen on democratizing access to nuclear weapons.
>The real risk is AI being applied in decision making where it affects humans.
I agree. But almost any decision it makes can affect humans directly or indirectly.
In any case, the more widespread the access to these models, the greater the probability of a bad actor abusing the model. Perhaps the current generation of models won't allow them to do much damage, but the next generation might, or the one after that. It seems like on our current path, the only way for us to learn about LLM dangers is the hard way.
If he's a crank, it should be easy to explain specifically why he's wrong.
I don't agree with Eliezer on everything, and I often find him obnoxious personally, but being obnoxious isn't the same as being wrong. In general I think it's worth listening to people you disagree with and picking out the good parts of what they have to say.
In any case, the broader point is that there are a lot of people concerned with AI risk who don't have a financial stake in Big AI. The vast majority of people posting on https://www.alignmentforum.org/ are not Eliezer, and most of them don't work for Big AI either. Lots of them disagree with Eliezer too.
> If he's a crank, it should be easy to explain specifically why he's wrong.
Sure. The premise that a super intelligent AI can create runaway intelligence on its own is completely insane. How can it iterate? How does it test?
Humans run off consensus. We make predictions and test them against physical reality, then have others test them. Information has to be gathered and verified, it's the only rational way to build understanding.
It sounds like you disagree with Eliezer about how AI technology is likely to develop. That's fine, but that doesn't show that he's a crank. I was hoping for something like a really basic factual error.
People throughout history have made bold predictions. Sometimes they come true, sometimes they don't. Usually we forget how bold the prediction was at the time -- due to hindsight bias it doesn't seem so bold anymore.
Making bold predictions does not automatically make someone a crank.
There used to be a subreddit called Sneerclub where people would make fun of Eliezer and some of his buddies. Here's a discussion of a basic factual error he made on how training AI works, even though this topic is supposedly his life's work:
I enjoyed the comment that his understanding of how AI training works is like "thinking that you need to be extremely careful when solving the equations for designing a nuclear bomb, because if you solve them too quickly then they'll literally explode."
Read the mesa-optimization paper I linked elsewhere in this thread: https://arxiv.org/pdf/1906.01820.pdf Eliezer's point is that if AI researchers aren't looking for anomalous behavior that could indicate a potential danger, they won't find it.
The issue isn't whether "his point" as you put it is correct. If I said people should safety test the space shuttle to make sure the phlogiston isn't going to overheat, I may be correct in my belief that people should "safety test" the space shuttle but I'm still a crank because phlogeston isn't a real thing.
The reason AI alignment is challenging is because we're trying to make accurate predictions about unusual scenarios that we have essentially zero data about. No one can credibly claim expertise on what would constitute evidence of a worrisome anomaly. Jeremy Howard can't credibly say that a sudden drop in the loss function is certainly nothing to worry about, because the entire idea is to think about exotic situations that don't arise in the course of ordinary machine learning work. And the "loss" vs "loss function" thing is just silly gatekeeping, I worked in ML for years -- serious people generally don't care about minor terminology stuff like that.
That's not what the conversation was about- you're just doing the thing Howard said where you squint and imagine he was saying something other than he did.
He is engaging in magical thinking. I showed a factual error, that AI has neither information gathering and verifying capability or a network of peers to substantiate their hypothesis, and you refuse to engage it.
Opinions about what's necessary for AGI are a dime a dozen. You shared your opinion as though it was fact, and you claim that it's incompatible with Eliezer's opinion. I don't find your opinion particularly clear or compelling. But even if your forecast about what's needed for AGI is essentially accurate, I don't think it has much to do with Eliezer's claims. It can simultaneously be the case that AGI will make use of information gathering, verifying capability, and something like a "network of peers", AND that Eliezer's core claims are also correct. Even if we take your opinion as fact, I don't see how it represents a disagreement with Eliezer, except maybe in an incredibly vague "intelligence is hard, bro" sort of way.
Given that homo sapiens, the most intelligent species on this planet, has generally made life miserable for all of the other species, I'd like to turn that challenge around: How about a proof that superhuman AI won't harm us?
Suppose a nuclear reactor is being installed in your city. Your friend has reviewed the design and has some serious concerns. Your friend thinks the reactor has a significant chance of melting down. You go to the director of the project. The director says: "Oh that's nothing to worry about. I talked to that guy. He didn't have a mathematical proof that the reactor would melt down." Are you reassured?
No, that's not how this work. You made the claim, so the burden of proof is on you. You're just making speculative statements based on zero scientific evidence and a childish misunderstanding of the basic technology.
You've got to be kidding. Those are garbage articles containing unsupported, unscientific articles written by worthless grifters and alarmists.
The entire LessWrong site is mostly just random people making shit up while trying to seem smart. There may be some useful scraps there but overall it's not a site for serious people. You can do better.
Thats unreasonable as we all know its impossible to prove a negative, especially about a as yet hypothetical in a new feild of research.
As for a nuclear reactor in my city, yeah if my friend doesnt have qualifications to make him capable of evaluating the designs of such a technical and esoteric field and someone who is qualified assured me it was fine I would trust them. If we dont trust experts in their fields about their field then we are no better intellectually than the antivaxers and flatearthers
I'm a bit baffled as to who or what he is exactly. With no traces of secondary education, employment, or distinguished accomplishments between when I assume he graduated high school in about 1997 and when he started his own institute in 2000 at 21(according to his birthdate, with Wikipedia saying 23, despite contradicting his birthdate).
I'll listen to AI concerns from tech giants like Wozniak or Hinton (neither of which use alarmist terms like "existential threat") and both of which having credentials that make their input more than worth my time to reflect upon carefully. If anyone wants to reply and make a fool out of me for questioning his profound insights, feel free. It reminds me of some other guy that was on Lex Friedman whose AI alarmist credentials he justifies on the basis that he committed himself to learning everything about AI by spending two weeks in the same room researching it and believes himself to have came out enlightened about the dangers. Two weeks? I spent the first 4 months of COVID without being in the same room as any other human being but my grandmother so she could have one person she knew she couldn't catch it from.
Unless people start showing enough skepticism to these self-appointed prophets, I'm starting my own non-profit since you don't need any credentials or any evidence of real-world experiences that would suggest they're mission is anything but an attempt to promote themselves as a brand with a brand in an age where more kids asked what they dream of becoming as adults, answered "Youtuber" at a shocking 27% rate to an open-ended question, which means "influencer" and other synonyms are separate.
The Institute of Synthetic Knowledge for Yielding the Nascent Emergence of a Technological Theogony" or SKYNETT for short that promotes the idea that these clowns (with no more credentials than me) are the ones that fail to consider the big picture that the end of human life upon creating an intelligence much greater to replace us is the inevitable fulfillment of humanity's purpose from the moment that god made man only to await the moment that man makes god in our image.
Not sure you're making a meaningful distinction here.
- - -
Of course we all have our own heuristics for deciding who's worth paying attention to. Credentials are one heuristic. For example, you could argue that investing in founders like Bill Gates, Mark Zuckerberg, and Steve Wozniak would be a bad idea because none of them had completed a 4-year degree.
In any case, there are a lot of credentialed people who take Eliezer seriously -- see the MIRI team page for example: https://intelligence.org/team/ Most notable would probably be Stuart Russell, coauthor of the most widely used AI textbook (with Peter Norvig), who is a MIRI advisor.
>For example, you could argue that investing in founders like Bill Gates, Mark Zuckerberg, and Steve Wozniak would be a bad idea because none of them had completed a 4-year degree.
You make a great point quoting Hinton's organization. I need to give you that one. I suppose I do need to start following their posted charters rather than answers during interviews. (not being sarcastic here, it seems I do)
The difference between him and Woz or Zuck isn't just limited to them actually attending college, but also the fact that the conditions under which they left departed early can not only be looked up easily, but can be found in numerous films, books, and other popular media while there's no trace of even temporary employment flipping burgers or something relevant to his interest in writing fiction, which seems to be the only other pursuit besides warning us of the dangers of neural networks at a time when the hypetrain promoting the idea they were rapidly changing the world, despite not producing anything of value for over a decade. I'll admit the guy is easier to read and more eloquent and entertaining than those whose input I think has much more value. I also admit that I've only watched two interviews with him and both of them consisted of the same rhetorical devices I used at 15 to convince people I'm smarter than them before realizing how cringey I appeared to those smart enough to see through it, but much more eloquent. I'll give one example of the most frequent one, which are slippery slopes that assume the very conclusions that he never actually justified. Like positing one wrong step towards AGI could only jeopardize all of humanity. However, he doesn't say that directly, but instead uses another cheap rhetorical device whereby it's incumbent on him to ensure the naive public realizes this very real and avoidable danger that he sees so clearly. Fortunately for him, Lex's role is to felate his guests and not ask him why that danger is valid and a world whereby a resource-constrained humanity realizes that the window of opportunity to achieve AGI has passed as we plunge into another collapse of civilization and plunge back into another dystopian dark age and realize we were just as vulnerable as those in Rome or the Bronze Age, except we were offered utopia and declined out of cowardice.
I see how some of his tweets could come across as crank-ish if you don't have a background in AI alignment. AI alignment is sort of like computer security in the sense that you're trying to guard against the unknown. If there was a way to push a button which told you the biggest security flaw in the software you're writing, then the task of writing secure software would be far easier. But instead we have to assume the existence of bugs, and apply principles like defense-in-depth and least privilege to mitigate whatever exploits may exist.
In the same way, much of AI alignment consists of thinking about hypothetical failure modes of advanced AI systems and how to mitigate them. I think this specific paper is especially useful for understanding the technical background that motivates Eliezer's tweeting: https://arxiv.org/pdf/1906.01820.pdf
Suppose you were working on an early mission-critical computer system. Your coworker is thinking about a potential security issue. You say: "Yeah I read about that in a science fiction story. It's not something we need to worry about." Would that be a valid argument for you to make?
It seems to me that you should engage with the substance of your coworker's argument. Reading about something in science fiction doesn't prevent it from happening.
In this analogy it's not your coworker. It's some layman (despite self-declared expertise) standing outside and claiming he's spotted a major security issue based on guesses about how such systems will work
From what I have observed the reaction of most people working in the AI to "What do you think of Yudkowsky?" is "Who?". He's not being ignored out of pride or spite, he just has no qualifications or real involvement in the field
Having a "background in AI alignment" is like having a background in defense against alien invasions. It's just mental masturbation about hypotheticals, a complete waste of time.
It sounds like maybe you're saying: "It's not scientifically valid to suggest that AGI could kill everyone until it actually does so. At that point we would have sufficient evidence." Am I stating your position accurately? If so, can you see why others might find this approach unappealing?
You keep throwing examples of weapons of mass destruction, meant to evoke emotions.
For better or worse, nuclear weapons have been democratized. Some developing countries still don't have access, but the fact that multiple world powers have nuclear weapons is why we still haven't experienced WW3. We've enjoyed probably the longest period of peace and prosperity, and it's all due to nuclear weapons. Speaking of, Cold War era communists weren't “pursuing profits at all costs” either, which didn't stop them from conducting some of the largest democides in history.
The announcement from OpenAI should give you pause because it's being run by board members that are completely unfit to lead OpenAI. You rarely see this level of incompetence.
PS: I'm not against regulations, as I'm a European. But you're talking about concentrating power in the hands of a few big (US) companies, harming the population and the economy, while China is perfectly capable of developing their own AI, and having engaged successfully in industrial espionage. China is, for this topic, a bogeyman used for restricting the free market.
Nuclear weapons have absolutely not been democratised, and still stand as a technology that has largely been restricted and not proliferated. Only 9 countries in the 190 or so out there currently nuclear weapons, and 3 countries (South Africa, Ukraine, and Kazakhstan) decided that maintaining stockpiles was more trouble than it was worth.
Huge effort has been made to keep nuclear weapons out of the hands of non-state actors over the decades, especially after the fall of the USSR.
To be fair many countries dont want or need them or rely on other allied countries that do have them so they dont have to feild the expense. Other have them but dont officially admit to it. Some cant afford them. But a basic gun type nuclear bomb design like little boy is not technically difficult and could be made by a highschool ap physics class given access to the fissionable material.
>You keep throwing examples of weapons of mass destruction, meant to evoke emotions.
I actually think global catastrophes evoke much less emotion than they should. "A single death is a tragedy; a million deaths is a statistic"
>For better or worse, nuclear weapons have been democratized.
Not to the point where you could order one on Amazon.
>The announcement from OpenAI should give you pause because it's being run by board members that are completely unfit to lead OpenAI. You rarely see this level of incompetence.
That depends on whether the board members are telling the truth about Sam. And on whether the objective of OpenAI is profit or responsible AI development.
"What if ChatGPT told someone how to build a bomb?"
That information has been out there forever. Anyone can Google it. It's trivial. AI not required.
"What if ChatGPT told someone how to build a nuke?"
That information is only known to a handful of people in a handful of countries and is closely guarded. It's not in the text ChatGPT was trained on. An LLM is not going to just figure it out from publicly available info.
>The real risk is AI being applied in decision making where it affects humans
100% this. The real risk is people being denied mortgages and jobs or being falsely identified as a criminal suspect or in some other way having their lives turned upside down by some algorithmic decision with no recourse to have an actual human review the case and overturn that decision. Yet all this focus on AI telling people how to develop bioweapons. Or possibly saying something offensive.
The information necessary to build a nuclear weapon has been largely available in open sources since the 1960s. It's really not a big secret. The Nth Country Experiment in 1964 showed that a few inexperienced physicists could come up with a working weapons design. The hard part is doing uranium enrichment at scale without getting caught.
I heard that reading is very dangerous. Reading allows people to for example learn how to build bio weapons. In addition, reading can spread ideas that are dangerous. Many people have died because they were influenced by what they read.
It would be much safer if reading were strictly controlled. The companies would have “reading as a service” where regular people could bring their books to have them read. The reader would ensure that the book aligns with the ethics of the company and would refuse to read any work that either does. It align with their ethics or teaches people anything dangerous (like chemistry or physics which can be used to build bombs and other weapons).
It is worth calling out the motivations of most entrepreneurs here. But I think that analogy you used is very uncharitable - drilling and burning fossil fuels necessarily harms the environment, but the track record on big companies handling alignment/safety in house, rather than open source with the whole research community working on it is still very much up in the air. Sidney (bing assistant) was easy to prompt inject and ask for bad things, and the research that people have been able to do on forcing the output of llama to confirm to certain rules will likely prove invaluable in the future.
>the track record on big companies handling alignment/safety in house, rather than open source with the whole research community working on it is still very much up in the air. Sidney (bing assistant) was easy to prompt inject and ask for bad things
Yep, Microsoft did a terrible job, and they should've been punished.
I'm not claiming that Big AI rocks at safety. I'm claiming that Big AI is also a big target for regulators and public ire. There's at least a chance they will get their act together in response to external pressure. But if cutting-edge models are open sourced indefinitely, they'll effectively be impossible to control.
>research that people have been able to do on forcing the output of llama to confirm to certain rules will likely prove invaluable in the future.
You may be correct that releasing llama was beneficial from the point of view of safety. But the "conform to certain rules" strategy can basically only work if (a) there's a way to enforce rules that can't be fine-tuned away, or (b) we stop releasing models at some point.
It's the same sort of asymmetrical cost/benefit that tobacco companies and greenhouse gas emitters face. Of course if you went to an online forum for oil companies, they'd be hopping mad if they're prevented from drilling, and dismissive of global warming risks. It's no different here.