Yudkowsky has a very shallow understanding of machine learning/AI, but loves to write in a conspiratorial way with in-group terminology to make people think that he's onto some dark secrets.
What he's describing is common knowledge. This technique is called knowledge distillation (there are some small differences, but the idea is the same). A large model (here, openai's davinci) is used as a teacher model to train a smaller student model (llama). This is used practically everywhere in industry to improve inference times.
(1) This is not knowledge distillation, which requires knowing the logits and not just the output text.
(2) "Training a smaller model off davinci using the logits" would have been vastly more expensive than what they did, which was to fine-tune another pretrained foundation model, which happened to be smaller, using output of another foundation model, which happened to be larger. The relative size isn't the key idea, it's that they were able to extract the fine-tuning for instruction-following (which is relatively computationally cheap to train, but requires a custom expensive small dataset) and port it from one foundation model to another. This is what has implications for the companies that planned to have this custom fine-tuning as part of their competitive moat.
(3) I understand that ad hominem is against the culture that Hacker News tries to cultivate here. Consider reading or rereading the local guidelines on discussion, as well as reading or rereading an article on knowledge distillation, the Stanford publication of Alpaca, and my tweet.
1) It is a general version of knowledge distillation. For example, this paper from 2016 describes the same technique: Sequence-Level Knowledge Distillation [0]
> This sequence-level approximation leads to a simple training procedure wherein the student network is trained on a newly generated dataset that is the result of running beam search with the teacher network
2) Fine-tuning is a step in the training process. Language models are first pre-trained, then fine-tuned. This is a pedantic quibble.
3) It is unsurprising that you don't understand ad hominem. Giving background information and pointing out the style of writing is relevant to arguments made.
It's arguable that saying "EY has a very shallow understanding of ML" is even lower than ad hominem (which is DH1) on the pg scale [4], since pg specifically gives "The author is a self-important dilettante." as an example of DH0.
Literally retraining on inputs and outputs without doing anything clever, does not to me seem to deserve quite such a fancy name. How do we end up with a smaller model if the dataset formats are exactly similar, except in some other special case, like using much more generated data to train the smaller model? Of course there are other deservedly clever things you could do besides matching logits, that would enable a smaller student model.
Here, the Stanford authors are not doing anything clever to enable a smaller model. They're just yoinking the fine-tuning onto what happens to be a smaller model. The destination model being smaller is not the point. The cheap yoinking of just the instruction tuning is the point. They used a small model as the destination because that was cheaper.
Sure, but the value of logit matching is that it allows the smaller model to access richer representations that it has sufficient capacity to express but insufficient capacity to learn. In this case, we're not trying to distill the full capabilities of the large model. We just want to mimic instruction following, and we don't have any reason to believe the smaller model has insufficient capacity. So doesn't the lesson of knowledge distillation tell us that this should work just by matching low temperature logits (in other words, matching argmax only)? I don't see how this is fundamentally different from the standard knowledge distillation setting - it's just an easier instance of it
I threw a null cultural reference exception on that one. Can you please clarify?
Edit: I know the "Masks of Nyarlathotep" and what a shoggoth is; I just don't know what is the "mask of the shoggoth".
Edit 2: Really, the one thing I wanted to be able to do with a language generator has always been to get it to GM a CoC campaign for me. It's still the case that there's no chance of that. Just as an aside.
It's referred to as distillation in the literature in separate sub-fields (e.g. LRspeech, a paper about low-resource speech-to-text & text-to-speech from Microsoft Research, 2020, uses datasets generated by a large model to train a small one and refers to this as knowledge distillation: https://arxiv.org/abs/2008.03687 )
> Yudkowsky has a very shallow understanding of machine learning/AI, but loves to write in a conspiratorial way with in-group terminology to make people think that he's onto some dark secrets.
Perhaps I am a little behind the times, but what are reasons/achievements/credentials behind his huge, prophet like status within the cult of rationalism? Beyond writing a popular Harry Potter fanfic, I mean.
I think EY's big contribution to the LessWrong/Rationalist cult/movement/ethos is writing the "Sequences," which purport to be a guide to training yourself to think rationally. (https://www.lesswrong.com/tag/original-sequences)
The rationalists are an interesting group. I like a lot of the vibe of the whole thing, but there is also some very weird navel gazing that has cropped up. It can be hard to differentiate from the good and the bad.
I am overall a little surprised by the vitriol towards EY in this thread though. While I agree that the guy is somewhat shrill and his ideas are pretty out there, he really doesn't seem like a con artist from my standpoint. It seems to me that he is genuinely extremely concerned about AI alignment/risk, and his actions seem to make sense in that framework without needing to ascribe a bunch of weird and sinister motivations to him.
Not professing agreement or disagreement with him on any particular thing, just my two cents.
Eh, I think the reason for his cult-leader status was for creating the whole community and being its, like, prophet. The sequences were just one of many things in that direction.
I'm not surprised by the vitriol, though. He has got to be one of the most arrogant people alive, and people arrogantly writing as though they are world experts on everything when they are, like, dumb and wrong... is extremely irritating.
I understand your point, more or less, I just find it funny that people calling themselves "rationalists"/"objectivists"/whatever other funny name just keeps stacking up a bunch of "what if"s and freaking out about it.
As somewhat of a cynical person, I also find it funny that
> While I agree that the guy is somewhat shrill and his ideas are pretty out there, he really doesn't seem like a con artist from my standpoint.
I've heard this exact same sentiment about another popular Effective Altruist in the past year. I will let the reader guess which one.
> I've heard this exact same sentiment about another popular Effective Altruist in the past year. I will let the reader guess which one.
I may be missing something, as I don't really follow this "community" besides reading the occasional Slate Star Codex post or whatever.
But I don't see much ground for comparison here. SBF (who I assume you're referring to) was in much different position with regards to potential harm as a result of con man antics than EY is. (As far as I can tell).
SBF was managing billions of dollars. EY is not. If SBF turns out to be a scammer, real people lose money. If EY is a scammer... then so what? He publishes a bunch of blog posts crying doom about AI that suddenly look less sincere? To what end? I can't see a good reason at this time for assuming his reasons are anything less than what he claims they are.
The Objectivists I’ve encountered refuse to engage with lots of things, but still use hypotheticals themselves. They mostly phrase them as generalities with “When…” rather than with “If…”, but they are semantically equivalent.
>If scientific knowledge were hidden in ancient vaults (rather than hidden in inconvenient pay-for-access journals), at least then people would try to get into the vaults. They'd be desperate to learn science. Especially when they saw the power that Eighth Level Physicists could wield, and were told that they weren't allowed to know the explanation.
> [...]
>If the Great Secret of Natural Selection, passed down from Darwin Who Is Not Forgotten, was only ever imparted to you after you paid $2000 and went through a ceremony involving torches and robes and masks and sacrificing an ox, then when you were shown the fossils, and shown the optic cable going through the retina under a microscope, and finally told the Truth, you would say "That's the most brilliant thing ever!" and be satisfied. After that, if some other cult tried to tell you it was actually a bearded man in the sky 6000 years ago, you'd laugh like hell.
>And you know, it might actually be more fun to do things that way. Especially if the initiation required you to put together some of the evidence for yourself—together, or with classmates—before you could tell your Science Sensei you were ready to advance to the next level. It wouldn't be efficient, sure, but it would be fun.
>[...]
>And no, I'm not seriously proposing that we try to reverse the last five hundred years of openness and classify all the science secret.
He’s an excellent writer and has spent a long time trying to come up with high level easily understandable ways to try to measure alignment abstractly through pure logic/outside of any given system.
Essentially him and his crowd have been trying to create a rationally grounded purely logical non contradictory moral framework for action from axiomatic first principles (he’s not the first or the last to do this), and they believe such a system is essential if AGI comes to be.
I personally think AGI is impossible, his purely logical moral system is impossible, and think what he’s working on is not applicable to current AI (I think he knows this, but his conclusion seems to be that the path to AGI is out of the bag/was made incredibly foolishly and should have been constructed differently with more ability to add guardrails)
But I do think the high level abstract way he thinks is fruitful/very valuable, and a good moderating counterbalance to the frenzied “harder better faster stronger” ML stampede that’s not really thinking about the ramifications of what they’re doing with any sort of discipline.
That’s a very weird statement to make. Do you think you have a soul or something? Otherwise how is your own Intelligence something a computer can’t replicate?
>> Do you think you have a soul or something? Otherwise how is your own
Intelligence something a computer can’t replicate?
Strictly speaking, it is for you to say why you think that a digital computer,
which is nothing like a human, can be intelligent like a human.
Btw, I see this quip about souls often on HN, in response to comments that AGI
is impossible and it is invariably introduced in the conversation by the
person arguing that AGI is possible. I have never, ever seen anyone argue that
"digital computers can't be intelligent because they don't have a soul". I
don't even know where that "soul" bit comes from, probably from something
someone said centuries ago? In any case it's irrelevant to most modern
conversations where most people don't believe in the supernatural, and have
other reasons to think that AGI may be impossible.
And why wouldn't they? For the time being, we don't have anything anyone is prepared to call "AGI".
We also don't have a crystal ball to see into the future and know whether it
is possible, or not. For all we know, it might be a theoretical possibility,
but a practical impossibility, like stable wormholes, or time travel, or the
Alcubierre Drive. We will know when we know, not before.
Until then, invoking "souls", or any other canned reply, only seems to serve
to shut down curious conversation and should be avoided.
To say it in very plain English: you don't know if AGI is possible, you only
assume it to be, so let's hear the other person say what they assume, too. Their opinion is just as interesting as
yours, and just as an opinion.
One answer to this would be that humans aren't general intelligences either, but existing in the real world/human society keeps you honest enough you have to try and act like one.
I think it’s impossible because I don’t think you can leapfrog evolution, and there’s almost definitely hidden context essential to our intelligence and perception that you can only get through a very very long history.
However I do think it’s quite possible and likely that we’ll create mimics that convince most people they’re intelligent. But I think that’s a weird type of reflection/anything we can make is limited to mirroring our observable past and existing collective thought and can never be truly introspective, because true introspection requires evolutionary context and hidden motivation you can’t transfer.
Evolution is a process that works without an intelligent steward. In a way it's a brute force technique. Plus, nothing is optimizing for intelligence in evolution, it is merely a happy accident that humans ended up with the brains that we did. A different environment could yield drastically different evolutionary history
It doesn't seem very logical to think that because evolution took so long to get us to where we are now, we consequently won't be able to design an intelligent AGI system.
It’d take a while to justify this argument to the extent I think it’s justified, but I think we’re in an inescapable frame set by evolution, and our adaptation to our environment probably goes a lot deeper than what we can see. I don’t think the visible context is the full context, and think true intelligence probably requires an implicit understanding of context which is invisible to us.
I don’t think true intelligence is propositional/logic based. I also think the observable universe is inherently limited by our perceptual framework, and there will always be something outside it.
Imagine an amoeba that can only detect light and dark. Its observable universe exists on a simple gradient. But the cells that are used in its perceptual system cannot be described on a gradient. I think that probably scales up indefinitely.
We confuse what we see for all that exists because of our recent history and scientific advances. The triumph of science comes from focusing solely on what we can see and reason about and improving our vision because thats what we have control over, but it doesn’t mean we see everything. I think true intelligence originates from something we can’t see. You can call it a soul, or call it a scaled up amoeba cell, but I personally think the origin of true intelligence and qualia is from some weird very very old evolutionarily created thing we can’t see (and I think other animals have similar invisible perceptual machinery somewhere, I don’t think it’s just a human thing).
Current AI as exemplified by LLMs like ChatGPT is not propositional/logic based, though, so I'm unsure what your objection is. Artificial intelligence can be just as messy and alogical as natural intelligence. It's not because it runs on a computer that we fully see or understand what it does.
I suppose there is a discussion to be had about what "true intelligence" is and whether some or most human beings have it in the first place.
It’s still propositional and logic based. Every instruction a computer can interpret is propositional and logic based. AI models are chaotic, but deterministic. All computers are deterministic. The distinction I’m making between propositional systems and intelligence is like the distinction between being able to generate a transcendental number like pi (which is propositionally generated/you can calculate it without “understanding” the sequence/being able to reduce it into a smaller pattern) and being able to know whether your procedure to generate pi is correct or incorrect. That latter ability is what I mean by true intelligence, and this transformers approach doesn’t solve that. I believe this is called the “grounding” problem.
Gödel proved it’s impossible for any deterministic system to determine its own correctness close to a hundred years ago. Whatever we’re doing when we say an output A is “right” and output B is “wrong” is fundamentally different than what a propositional deterministic system does, regardless of how large or chaotic or well fitted to expected output it is.
What qualifies as “true intelligence” is the core issue here, yes, and transformers don’t qualify. That doesn’t mean they aren’t valuable or can’t give correct results to things much faster and better than humans, but I think anything we could deliberately design inevitably ends up being deterministic and subject to the introspective limitations of any such system, no matter how big or all encompassing.
I think we’re going to create better and more comprehensive “checkpoint”/“savepoints” for the results of collective intelligence, but that’s different than the intelligence itself.
Our understanding of computer instructions is propositional and logic-based. The hardware itself is a material thing: it's a bunch of atoms and electrons. So is the human brain. Whatever distinction we can make between them cannot be purely conceptual, it has to be physical at some level. The fact that we "logically" designed and made computers, but not brains, is not a relevant distinction. It's not part of the physical reality of either thing, as a physical thing.
> Gödel proved it’s impossible for any deterministic system to determine its own correctness close to a hundred years ago.
I don't think non-deterministic systems fare much better. At the core, completeness is a quantitative problem, not a qualitative one. In order to fully understand and analyze a system of a certain size, there is no way around the simple fact that you need a bigger system. There are more than ten things to know about ten things, so if you only have the ten things to play with, you will never be able to use them to represent a fact that can only be represented using eleven things. Gödel's theorem pushes this to infinity, which is something that we are notoriously poor at conceptualizing. I think this just obfuscates how obvious this limitation is when you only consider finite systems.
Which brings me to the core disagreement that I think we have, which is that you appear to believe that humans can do this, whereas I believe they blatantly cannot. You speak of the grounding problem as if the human brain solves it. It doesn't. Our reasoning is not, has never been, and will never be grounded. We are just pretty damn good at lying to ourselves.
I think that our own capacity for introspection is deeply flawed and that the beliefs we have about ourselves are unreliable. The vast array of contradictory beliefs that are routinely and strongly held by people indicates that the brain reasons paraconsistently: incoherence, contradiction and untruth are not major problems and do not impede the brain's functioning significantly. And so I think we have a bit of a double standard here: why do we care about Gödel and what deterministic systems can or cannot do, in a world where more than half of the population, but let's be honest, it's probably all of us, is rife with unhinged beliefs? Look around us! Where do you see grounded beliefs? Where do you see reliable introspection?
But I'll tell you what. Evolution is mainly concerned with navigating and manipulating the physical world in a reliable and robust way: it is about designing flexible bodies that can heal themselves, immune systems, low-power adaptable neural networks, and so on. And that, strangely enough, is not something AI does well. Why? My theory is that human intelligence and introspection are relatively trivial: they are high impact, for sure, but as far as actual complexity goes, they are orders of magnitude simpler than everything else that we have in common with other animals. Machines will introspect before they can cut carrots, because regardless of what we like to think about ourselves, introspection is a simpler task than cutting carrots.
I think this explains the current trajectory of AI rather well: we are quickly building the top of the pyramid, except that we are laying it directly on the ground. Its failure will not be intelligence, introspection or creativity, it will be mechatronics and the basic understanding of reality that we share with the rest of the animal kingdom. AKA the stuff that is actually complicated.
Our perception of the physical world is itself an evolved construct. The wider thing that perception is fitting is something I don’t think we can fit a machine to. I think we can only fit it to the construct.
I get where you’re coming from/appreciate the argument, and you may be right, but I‘ve come to appreciate the ubiquity of hidden context more and more/lean that way.
To be generous, I did not know this was a thing, and if you'd told me it was a thing, I would have said "gosh, you don't say" and not realized the implications. Not everybody is embedded in this world, so we need explainers.
The implications aren't quite common knowledge yet: a lot of companies are still operating with the "we'll beat the competition by building a better AI than them" mindset.
I don't think the broader market has caught up with the idea that investing in AI gives you no long-term competitive advantage. (eg StabilityAI hasn't gone bankrupt yet)
Thanks for clearing this up. This seemed like much ado about nothing, but I didn't want to assume that this guy was trying to appear more knowledgeable than he may be.
I think the most impressive feat of GPT/LLMs thus far has been all the ML experts its caused to materialize from thin air.
my possibly incomplete guess: "articulate person opines on a topic that they have no expertise in, and uses articulation to hide their lack of expertise"
Precisely. Also "enthusiastic and opinionated on the matter, without actual involvement in the field except as dilettante sharing ideas irrelevant to actual practicioners"
This has been true of machine learning APIs from the early renaissance of neural networks. Computer vision APIs have been used for years to grow datasets and fine-tune models on the data.
As a matter of fact, the API/SaaS business for machine learning 'direct' model results have suffered from this since the early days. Now, what may make more sense and survive longer are full data-storage-oriented platforms, and niche products that encapsulate all underlying model calls, including to publicly available models. Sometimes glue code or even devops becomes the true solid business case of AI/ML apps ;)
While cloning kind of works, I wouldn't expect the results to be quite as good as the original.
Right now we're in this weird phase where we have no accurate metrics for how good an LM is, so people just try them with a handful of queries and are subjectively impressed or not. Maybe clones can pass that test, but as the market gets more sophisticated, people will prefer superior models.
It may not be exactly as good, but it's also a 7B parameter model vs 175B parameters for GPT-3.5 (i.e., text-davinci-003). People are running the new model on their phones, laptops, and Raspberry PIs.
People's mouths were watering over the commercial implications of the recent 90% drop in cost for the new ChatGPT model. Now imagine if you can get similar performance on a model that requires <5% of the parameters.
The conditions for AI profitability he suggests as a “new idiom” for AI have fairly exact parallels in the world of software, but where the secret sauce has expanded to include inputs and outputs, not just the software itself:
> The AI companies that make profits will be ones that either have a competitive moat not based on the capabilities of their model,
This is the “open source” case, where profitable firm provide ancillary services and support, rather than directly selling the capabilities of the software, because the software is not at all exclusive.
> OR those which don’t expose the underlying inputs and outputs of their model to customers
This is the “internal software” case, where the profitable firm does not provide access to the software at all, but uses it internally in a way which is opaque from the outside to support other products or services it provides, maintaining exclusivity because no one outside actually gets access to the software itself. (Arguably, non-source-available closed source software might fit in here, though its often possible to reverse engineer software that you have access to without the source, from the executable, so I'd argue that even without source available its more the next category.)
> OR can successfully sue any competitor that engages in shoggoth mask cloning.
This is the “proprietary software” case, where the profitable firm protects its exclusivity through legal constraints on people who have access to the software, while giving customers access to it.
Given that the big firms in AI largely either are or are closely associated with big firms in software that have successfully used all three models for different parts of their portfolios, I don’t think that them figuring out AI profitability is particularly challenged by this “new idiom”.
Another way to put it is : current crop "AI companies" don't really have a moat. Transformers have existed for many years, and a bit of RL on top of it is not enough to get ahead. It's all about who has the most money to spend training larger and larger, but what happens when they all run out of money for that? LLMs will become cheaper than web hosting
Web hosting will always be less demanding, unless something weird happens and GPU/APU/TPU type hardware (along with HBM and related ancillaries) somehow becomes less expensive than (or subsumes) general purpose CPUs and RAM.
There will be behaviors smaller models won't be able to mimic, though. So sure, fine-tuning advantages will be rapidly copied, but huge models that require large data centers won't. Economic incentives are still aligned with companies going through the path of larger and larger models.
> Perhaps we will provide feedback to open source Llama using ChatGPT. The cost to adjust the model is presumably what's hard?
What I am so impressed by is that the state of the art is so democratic now. All the top AI posts on HN are things that I was able to get to myself days before the were posted here. This must mean either that the true SOTA is way ahead of what we see or that anyone can hit the current SOTA. Both are very exciting developments!
Completely missed the point, as usual. Consider the amount of world knowledge present in a 175B model vs a 7B model. You can elicit instruction following behavior with a relatively small number of samples , but you're never going to get that coverage of memorized facts. The Alpaca blog post gives the example that Alpaca doesn't know the capital of Tanzania, for example. The most is that depth of knowledge and understanding, not the surface level instruction following behavior.
You don't need to store it in weights. It would be too expensive to constantly update model with new data. It wouldn't be able to contain everything anyways. LLaMa 30B with access to 100TB of storage(you can store whole libgen plus arxiv and much much more in there) and trained to use it should be ahead of LLaMa 65B with no such access in terms of world knowledge. And it would cost less.
You can retrieve small amounts of specific information this way. But suppose the model was asked to write code in a new programming language that it doesn't have existing knowledge of. Could it do that given unguided access to the docs?
We've already seen zero-shot learning of novel syntaxes (that is, chatting with an LLM to interactively devise a new language that is then used, without any actual retraining of the model).
The point is that you can split the work in two: the one you mentioned, creating and training the model, and a separate step of shaping it with reinforcement learning. Which is quite expensive, because it uses human operators in one of its steps. But apparently you can duplicate easily enough this with an API of an already educated engine, which makes the second step not be a moat anymore.
But that's not a new point?! The second, expensive step (RLHF) was already automated by Anthropic with their Constitutional AI approach several months ago.
Cutting-edge LLMs are commoditized anyway. There are very few tech moats here, so his point doesn't change the economics much. Data will sometimes be a moat, but companies with proprietary data will also expose their models through GUIs and therefore have other means of preserving that moat. OpenAI is moving toward economies of scale with their API pricing, which is smart.
As usual, Yudkowsky has valuable insights, but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be.
TL;DR: If your customers can get text input-output samples from your super-secret LLM, whether via web interface or API, then your customers can easily and cheaply train their own LLM to learn to act like your super-secret one.
For example, if you're OpenAI, your customers can spend ~$100 to get input-output samples from text-davinci-003, plus another ~$500 to fine-tune LLaMA on those samples, and -- voila! -- they get Alpaca.
It’s not accidental at all that EY writes this way. He is writing out of L. Ron Hubbard’s style guide. He doesn’t want to communicate to outsiders, he is writing to members of his Cthulhu cult.
I’ve noticed a similar writing style in a lot of others from the same group (AI Doomsdayers/“Rationalists”). Most of the time I stumble across something from that corner of the web it feels like the author was thinking “Why convey an idea in just one sentence when I can convey it in ten?”
Reminds me a of a quote by Chomsky about post-modernism (not what EY is, but similar to what you're talking about)
> Some of what appears in it sort of actually makes sense, but when you reproduce it in monosyllables, it turns out to be truisms. It’s perfectly true that when you look at scientists in the West, they’re mostly men, it’s perfectly true that women have had a hard time breaking into the scientific fields, and it’s perfectly true that there are institutional factors determining how science proceeds that reflect power structures. All of this can be described literally in monosyllables, and it turns out to be truisms. On the other hand, you don’t get to be a respected intellectual by presenting truisms in monosyllables.
It's like Amway, people get into the EY cult and it takes over their life, they are trained to talk like EY to drive all the "normies" away, isolate them, and make them easier to control.
Using a lot of words is often a response to getting repeatedly misinterpreted or misunderstood by others. It results in putting up ten explanatory, context, definitional, or guardrail sentences for each one sentence of novel content.
I think you're talking about something else: using words extremely precisely, even to the point of the being annoying. I recall reading the Unabomber's manifesto and being annoyed by the almost paranoid preciseness.
In contrast, this comment chain was more about EY using large words pretentiously, to suggest expertise or intellectual depth that he doesn't actually have.
Making up lots of vocabulary that divides insiders from outsiders.
I was interested in Scientology as a microcosm of society around the time alt.religion.scientology appeared in the early 1990s. Scientology documented everything about how it runs, even management is scriptural, and only Hubbard could write scripture so it is like a specimen that's been fixed to a microscope slide.
Since then renegade "Free Zone" Scientologists have put all of his writings online so it is very accessible.
Dianetics is the Sequences of Scientology in that it is a long, windy, and nonsensical book that selects for people who are perseverant readers who are impervious to critical thinking. (Like one of those Nigerian scam emails.)
Hubbard's hypnotic language is demonstrated in his Philadelphia Doctorate Lectures where he not only mentions the hypnotist Milton Erickson but uses Erickson's techniques extensively. (There is no record of Hubbard meeting Erickson but Hubbard and Erickson were both highly active at the same time in Arizona and I think Hubbard must have sat in on Erickson’s lectures.)
(That is, that weird ramblely kind of writing is by no means innocent but it is misparsed by many people's minds and helps create an altered state of consciousness.)
Hubbard's most satanic (in opposition to conventional Eastern & Western religious ideas) works are Introduction to Scientology Ethics and A New Slant on Life. The first one describes an elaborate system of punishments that are applied to anyone who violates group norms and a system of evaluation that guarantees you will violate those norms by accident. The list of "ethics conditions" is folded so the positive conditions come first and a person who starts reading at the beginning and doesn't go all the way through will miss the batshit crazy stuff. The second describes how a Scientologist is supposed to disconnect themselves from anyone who disagrees with their spiritual journey.
Something Scientology shares with the EY cult (as well as the LaRouche organization and the third international) is the proliferation of front groups. Scientology has Narconon, Criminon, The Way to Happiness Foundation, and many others. The EY cult has "effective altruism", "rationalism", "longtermism", "AI safety" and probably those parties Aella used to run.
It's a typical path that a coercive group makes multiple 360° turns. "Effective Altruism" seems designed to attract morally weak people but seems rational enough at first. I mean, it seemed to make sense when Bill Gates insisted on funding only international NGOs that could prove the value of what they were doing.
Once, however, you are funding "longtermist" organizations that are concerned with imagined problems on the way between here and the glorious future that we've turned all the planetary mass of the Milky Way into Dyson spheres there are no results to measure!
It is like the technique a stage hypnotist uses of putting volunteers through a number of screens that rapidly selects the most suggestible and/or likely to go along subjects. Both in the short term and long term this process expels the trouble makers and leave behind the potential Manchurian Candidates.
It seems EY is now less fashionable, so criticizing him as a crackpot is more acceptable. But I remember a few years ago when he was in full swing, he had a lot of traction even here on HN, and of course in many nerdy circles. Also see that debacle with Roko's Basilisk and all those silly antics. The LessWrong crowd is insufferable.
I can sort of understand why EY had so much pull with internet nerds. He talked the talk, he was confident, and he was a nerd. But I'm also relieved that more people seem to understand now that he is a crackpot, and that claiming one is an expert on AI doesn't make it so, and that writing long essays mixing Bayes, Harry Potter and "rationalism" doesn't really make anyone an expert on anything.
Do you have any argument besides "he writes long-windedly with insular terminology" and "there are multiple organizations affiliated to him"? E.g.
>It's a typical path that a coercive group makes multiple 360° turns.
What is the evidence that Eliezer is running a coercive group? Whom are they coercing into what?
I fully get that LessWrong et al. are obnoxious, use their own terminology, have far-out ideas about the future of AI that are probably wrong, weird moral systems (though any moral system is weird when you look at it hard enough imho)... But you are making the charge that Eliezer is running a cult and doing so on purpose, which is more specific than just running an annoying community around some non-mainstream ideas.
I bring up Hubbard because Hubbard documented his organization so well that he's a good target. There is a long list of other organizations that I could also find analogies with.
It’s true Hubbard divided the world into ‘public’ who would have their bank accounts empty and ‘staff’ who, penniless, would be turned into worker drones.
EY is not building that sort of machine. But Hubbard was operating at peak egalitarianism in the US and EY is operating at a time (and places) that have peak inequality. A Scientology whale gives a few million to the organization, the ideal EY whale is Sam Bankman Fried. Since it is money that is fungible and ‘makes the world go round’ he doesn’t need an army of worked drones when just one rich kid at Stanford or Cambridge can be quietly bled.
Other groups to look at with apocalyptic ideology are the Symbionese Liberation Army (Why make worker drones when you can brainwash Patty Hearst?), People's Temple (...go to San Francisco), Heaven’s Gate, Aum Shinrikyo, etc. It amazes me that one of them hasn’t tried to assassinate an A.I. researcher yet.
This is not a response to Kinrany's argument, in fact your last line is doing exactly what you're accused of again.
>Other groups to look at with apocalyptic ideology are the Symbionese Liberation Army (Why make worker drones when you can brainwash Patty Hearst?), People's Temple (...go to San Francisco), Heaven’s Gate, Aum Shinrikyo, etc. It amazes me that one of them hasn’t tried to assassinate an A.I. researcher yet.
Not obviously bad: have apocalyptic ideology (might actually be a little bit bad, admittedly)
Obviously bad but Eliezer et al. haven't done it: try to assassinate a researcher
Makes sense to me. It's an enormous inhuman blob trained on all of human knowledge that can manifest a personality out of nothing based on some inoffensive corporate prompt.
Is it a good insight though? The training data for many of these premier models is already available, and that's even better than some input-output pairs.
He’s talking about RLHF - reinforcement learning with human feedback (the process that trained ChatGPT), the training data for which is not publicly available.
And the point is that you don’t need to RLHF as long as you have access to another model that has been trained with RLHF that you can blackbox.
> but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be
Why does he use 2 different niche analogies to explain the thing that is still quite densely written at the end? Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence…
> Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence…
“Making profit on AI is basically the same as making profit on other software, except that the sensitive bits include the direct inputs and outputs to the core model, not just the software itself.”
Yes, it could. But then, the attempt to sell the idea this was some kind of radical phase transition that the players in the AI industry (largely, very experienced players in the software industry) simply were fundamentally not prepared for would be a lot harder.
That's only because these models are all pretty much the same (?) in terms of architecture, and basically do the same thing. New features, like memory, medium-term planning, and iterating towards a design goal, won't be imitatable in the same way.
I can't help but feel like there is an element of sour grapes here after Yudkowsky's attempts to build an AI failed (assuming that's what he was really trying to do, and not just to lure young and enthusiastic grad students into his lair...)
all I see are more arguments for my opinion that digital assets + capitalist marketplace = human stupidity.
> you're giving away your business crown jewels to competitors that can then nearly-clone your model without all the hard work you did to build up your own fine-tuning dataset.
you're loosing the capacity to continue to earn money for work that is already done. This just means they're not gonna be able to charge rent on the done work for too long.
the work is the training of the model. I chose to reject this to way to frame the issue; i.e. I reject that it is problem that they're giving away that hard work. It's only a problem due to social (market) ideologies of ownership which are necessary only by the logic of trade (or commerce).
As I see it, Eliezer is pointing out hat people who use the "product API"/ "chaptGPT service" can use this in such a way that they can clone it so to stop being a customer.
I see some kind of divine comedy in this (due to my own ideological beliefs).
> OR can successfully sue any competitor that engages in shoggoth mask cloning.
this is just the legal, modern day equivalent of old school "break their knees unless they pay you"
> this is just the legal, modern day equivalent of old school "break their knees unless they pay you"
I don't EY is advocating for this, so much as saying "unless AI companies can find a way to establish such a racket, they won't be profitable". That's a factual claim, not a value judgment.
I've recently started to believe that known current legitimate governments are (and have always been) the legacy of people in history doing rackets like these in the now historical past.
We have a very simple view of the product right now, and it's also new, so the different available options don't have much history to inform preference. That will change, and the stakes are very high, since this technology is capable of removing alot of jobs, so the company that f*cks up the least is going to become preferential. This effect will be magnified as regulation / beaurocracy come in more. Also, the post seems to assume that a single model will be published and then not change for a long time, enough time that a competetor can copy it and sell their copy; but this is also not true, models will be constantly updated and as soon as regulation catches up it'll be a legal requirement that they are constantly updated. And due to the power of the models, the state will quickly want to pick favorites, and that will also impact the difficulty of displacing a model provider. So, forget it, there's alot more involved than just having a copy of the result of some algorithms.
EY is not rare, but he is famous by proximity to powerful/wealthy actors. There are many philosophers, likely more than ever in history. Aside from professional philosophers, there’s a large pipeline of PhDs “doctor of philosophy” who work at a high enough level of abstraction in their own field to be considered a philosopher.
Hey, I don't think any real decision theorists would take something like this seriously. And if any would, then let's change the definition of a real decision theorist to exclude them.
What he's describing is common knowledge. This technique is called knowledge distillation (there are some small differences, but the idea is the same). A large model (here, openai's davinci) is used as a teacher model to train a smaller student model (llama). This is used practically everywhere in industry to improve inference times.