Eliezer Yudkowsky

Before Bruce Schneier goes to sleep, he scans his computer for uploaded copies of Eliezer Yudkowsky. Eliezer Yudkowsky is an American AI researcher, blogger, and autodidact exponent of specifically his Bayes-based human rationality. Yudkowsky cofounded and works at the Machine Intelligence Research Institute (formerly the Singularity Institute for Artificial Intelligence), a nonprofit organization that concerns itself with the concept known as the singularity. Yudkowsky founded the blog LessWrong as a sister site and offshoot of Overcoming Bias. He began his blogging career with GMU economist Robin Hanson.

Being idealistic enough to want everyone (above a certain income bracket) to live forever, he also has an overweening interest in cryonics and is (probably) vulnerable to hemlock.

AI "research"
Find whatever you're best at; if that thing that you're best at is inventing new math[s] of artificial intelligence, then come work for the Singularity Institute. If the thing that you're best at is investment banking, then work for Wall Street and transfer as much money as your mind and will permit to the Singularity Institute where [it] will be used by other people.

Yudkowsky believes he has identified a "big problem in AI research" in that we can't assume an AI would care about humans or ethics without our evolutionary history. Believing AI is imminent, Yudkowsky's taken it upon himself to create a Friendly AI (FAI). Such an AI won't kill us, inadvertently or otherwise.

"AI foom," short for "recursively self-improving Artificial Intelligence engendered singularity," comes from these ideas:


 * An AI-based singularity will occur soon.
 * AIs and humans would want them to improve themselves.
 * Human-level or better AIs could not be imprisoned or threatened with having their plugs pulled because they would talk their way out of the situation. Interestingly, Yudkowsky claims to have tested this idea by role-playing the part of supersmart AI, with a lesser mortal playing the jailer. However &mdash; the reader may note the beginning of a pattern here &mdash; the transcripts are unpublished.
 * AIs will, at some point, gain virtually unlimited control over the physical world through nanotechnology.

His transhumanist philosopher and existential risk theorist friend Nick Bostrom, director of the Future of Humanity Institute at the University of Oxford, wrote Superintelligence: Paths, Dangers, Strategies on the above ideas in 2014. Bostrom's central thesis appears to be "you can't prove this won't happen, therefore you should worry about it." Bostrom credits Yudkowsky in the acknowledgments section and quotes from him in the book.

As is evident from the dialogue between Luke Muehlhauser and Pei Wang, Yudkowsky's (and thus LessWrong's) conception of Artificial General Intelligence (AGI) differs starkly from the mainstream scientific and practical research understanding of it.

Yudkowsky believes an AGI will come from decision theory despite his examples being computationally intractable, or Bayesian logic that does not scale in real AI.

Since these are both totally abstract mathematical functions, there is no reason an AGI based on them would share human values (not that you can base an AGI on them). By this reasoning, Yudkowsky sees the need to bolt on some sort of morality module which will somehow be immune from alteration, even though the "foom" scenario he fears so much is based on AGIs which can rewrite their own code. (Bostrom points out in his book that there are several situations in which an AI might have an incentive to alter its own final goals.) Thus Yudkowsky sees the need to "solve" ethics in some form that can be computerized &mdash; although ethics remains an open problem in philosophy after thousands of years. However, Plato, Aristotle and Kant just weren't as smart as Yudkowsky believes himself to be.

In addition, some of Yudkowsky's critics identify more urgent threats than unfriendliness: deliberate weaponization of AIs, coding errors, humans themselves, cyborg ethics, etc.

Yudkowsky/Hanson debates
Economist Robin Hanson, Yudkowsky's then co-blogger, has debated Yudkowsky on "AI foom" on several occasions. Hanson, who had previously been an AI researcher for several years, wrote a series of blog posts with Yudkowsky in 2008 and debated him in person in 2011. These debates were later collected and published as an ebook. In 2014, after having read Bostrom's book Superintelligence, Hanson was sticking to his guns: he wrote another blog post entitled "I Still Don't Get Foom". He argued:

"Intelligence" just means an ability to do mental/calculation tasks, averaged over many tasks. I've always found it plausible that machines will continue to do more kinds of mental tasks better, and eventually be better at pretty much all of them. But what I've found it hard to accept is a "local explosion." This is where a single machine, built by a single project using only a tiny fraction of world resources, goes in a short time (e.g. weeks) from being so weak that it is usually beat by a single human with the usual tools, to so powerful that it easily takes over the entire world. Yes, smarter machines may greatly increase overall economic growth rates, and yes such growth may be uneven. But this degree of unevenness seems implausibly extreme.

Immediately after this debate, Yudkowsky left Overcoming Bias (now Hanson's blog) and moved the Sequences to LessWrong. They still weren't seeing eye to eye in 2016 after the famous Go match between Google's Alpha Go software and Lee Sedol. Hanson said he thought general-purpose AI was like nation-states, requiring a lot of workers and idea-sharing. Limited special-purpose AI would be like product teams in businesses. While he gave a criterion to test his theory, Yudkowsky said he thought the world would already have "ended" before there was a chance to verify it. He claimed different AI labs were doing AI training, and there was almost no exchange of pre-trained modules or specialist components. (However, general AI components get widely reused between companies.)

Yudkowsky had previously announced his belief that if, as we now know, Alpha Go beat the Go grandmaster, that would mean the singularity was getting closer. However, the previous occasions when computers beat top-ranked humans at a board game (checkers and chess) still left computers well below the intelligence of a human infant.

Singularity Institute / MIRI
In 1998, Eliezer Yudkowsky finished the first version of 'Coding a Transhuman AI', alternatively known as 'Notes on the Design of Self-Enhancing Intelligence'. After that, he spent a few months writing design notes for a new programming language named Flare. Thereafter, he wrote 'Plan to Singularity', presenting the idea of a Singularity Institute. In May 2000, at the meeting of Foresight Institute the idea of establishing the institute got support from two others. Now, the Singularity Institute had a website. Then he wrote a revised version of 'Coding a Transhuman AI'.

On July 23, SI started the Flare project under the leadership of one Dimitry Myshkin but canceled the project in 2003. Since the very start, Eliezer Yudkowsky has made creating 'Friendly AI' crucial to himself. In 2001, he wrote a book-length work 'Creating a Friendly AI'.

In 2004, Eliezer Yudkowsky got a "major insight" about Coherent Extrapolated Volition (CEV). A person might not always want what is best for him, but Yudkowsky wants his AI to do good for people. He believes that to find what is best for a person, AI would scan the person's brain and do "something complicated" To ensure the AI is friendly to all humans, it would do it to everyone. The sheer computing power needed is incomprehensibly large, and we have no idea what this would look like if implemented.

Until 2008 or 2009, Singularity Institute was practically a one-man show. All its publications bore the name of Eliezer Yudkowksy, and there was no other full-time employee of SI. However, change came in 2008 with the emergence of LessWrong. Soon, new publications by new authors like Carl Shulman, N Tarleton, and after some time, Luke Muelhauser began to arrive. And this became the norm, with newly inducted researchers writing most papers.

The new focus of research at SI became decision theory. In 2010, Eliezer Yudkowsky published Timeless Decision Theory.

Yudkowsky's collaboration with Nick Bostrom's Future of Humanity Institute became visible by this time. Multiple joint articles by SI researchers and FHI researchers like A. Sandberg, Stuart Armstrong, and the like appeared. A joint article by Eliezer Yudkowsky and Nick Bostrom appeared in 2011. Bostrom's 2014 book Superintelligence refers to Yudkowsky 15 times. There were also nearly five pages on Coherent Extrapolated Volition. In 2009, Jeffery Epstein donated $50000 to SIAI.

Fan fiction
Oh god. The future of the human species is in the hands of a guy who writes crossover fan-fiction.

For a slightly more humorous but insightful look into his mind, Yudkowsky has written Harry Potter fanfiction to teach rationality to a larger audience. In the story, a rationalist Harry tries to get his head around magic while lecturing science to his elders and audience.

HPMoR is widely acclaimed by the standards of fanfiction, with many readers hailing it as one of the best of its kind. Yudkowsky and his fans tried to spam it for the 2016 Hugo Awards, though it didn't make the shortlist. The podcast version was a Parsec Awards finalist in 2012 and 2015.

Some readers felt disillusioned by the sermonising. As Yudkowsky himself notes in a Facebook posting:

LessWrong may have developed a personality cult around Yudkowsky, with a list of Eliezer Yudkowsky Facts, but the fan fiction of him was apparently a play, produced without his knowledge.

Arbital
After tiring of LessWrong and finishing his Harry Potter fanfic, Yudkowsky's next project was Arbital, an exciting new competitor in the educational market with a different approach to learning than that of Wikipedia. He first proposed it in March 2015:

With $300k in funding, developers were hired, with Yudkowsky as an overseer. There were several changes in direction. It went live in early 2016 as the LessWrong Sequences in a different bottle. By late 2016, they identified as "the place for crowdsourced, intuitive math explanations." It had a chance of beating the notoriously opaque math articles on Wikipedia, except they couldn't find enough knowledgeable people to write them. As of early 2017, the front page was a Slashdot-style aggregator for people in the MIRI subculture. By the end of March, they admitted defeat and turned the front page into a coming-soon for a blogging platform. A project developer lamented Yudkowsky's leadership style.

Achievements
Why should I follow your Founder when he isn't an Eighth Level anything outside his own cult? With no training in his field of interest, Yudkowsky has no accomplishments to his credit beyond getting Peter Thiel to give him money. (And moving on to Ethereum cryptocurrency programmers.)

Even his fans admit, "A recurring theme here seems to be 'grandiose plans, left unfinished'." He claims to be a skilled computer programmer, but has no code available other than Flare, an unfinished computer language for AI programming with XML-based syntax. It was supposed to be the programming language in which SIAI would code Seed AI but got canceled in 2003. He hired professional programmers to implement his Arbital project (see above), which eventually failed.

Yudkowsky is almost entirely unpublished outside his foundation and blogs and never finished high school, much less did any AI research. No samples of his AI coding are public. In 2017 and 2018 he self-published some pop science books with titles like Inadequate Equilibria and endorsements from fellow travellers Bryan Caplan and Scott Aaronson.

Yudkowsky's papers are generally self-published and had a total of two cites on JSTOR-archived journals (neither to do with AI) as of 2015. One of these came from his friend Nick Bostrom at the closely-associated Future of Humanity Institute.

Yudkowsky's observable results in the real world are a fanfiction wildly acclaimed for standing out by the genre's standards, a pastiche erotic light novel, a large pile of blog posts and self-published nonfiction, and a surprisingly well-funded research organization with fewer papers in a decade and a half than a single graduate student in the course of a physics Ph.D. The latter would be peer reviewed.

Genius or crank?
Yudkowsky refers to himself as a genius six times in his "autobiography" that he wrote in 2000 when he was 20. To his credit, he has since disavowed it, considering his younger self an idiot. He also claims to have "randomly won the writing talent lottery". He thinks most Ph.D.'s are pretty ordinary compared to the people he hobnobs with.

However, he admits to possibly being less smart than As a homeschooled individual with no college degree, Yudkowsky may not be in an ideal position to estimate his own intelligence. For obvious reasons, many of his followers think he is a genius. Similarly, some of his followers are derisive of renowned scientists. Just look for comments about "not smart outside the lab" and "for a celebrity scientist." Yudkowsky believes a doctorate in AI is a net negative when it comes to Seed AI. While Yudkowsky doesn't attack Einstein, he does indeed think the scientific method cannot handle things like the Many worlds Interpretation or his view on Bayes' theorem. LessWrong does indeed have its own unique jargon.

Just so that the reader doesn't get the mistaken impression that Yudkowsky boasts about his intellect incessantly, here he is, boasting about how nice and good he is:

Disagreement with Yudkowsky's ideas is often attributed to "undiscriminating skepticism." If you don't believe cryonics works, it's because you have watched Penn & Teller: Bullshit!. It's just not a possibility that you don't believe it works because it has failed tests and is made improbable by the facts.

Other controversial positions
Despite being viewed as the smartest two-legged being to ever walk this planet, Yudkowsky and much of the LessWrong community endorse several positions subject to bitter debate and controversy in their respective fields. For example, he idealistically believes in transhumanism and cryonics with hope and without willpower and preaches the "Many Worlds Interpretation" (MWI) of quantum physics as a "slam dunk" despite the lack of scientific consensus among quantum physicists. Yudkowsky says a majority of theoretical physicists might not share his view, adding that attempted opinion polls conflict on this point.

Concerning Bayes' theorem, Yudkowsky believes it can contradict and supersede the scientific method and wants to apply Bayesian probability indiscriminately. As a partial list, Yudkowsky also claims non-computable results like Kolmogorov complexity are a reasonable basis for an entire epistemology, evolutionary psychology is well-established science, and philosophy is a subject so easy he can "re-invent it on the fly".

Yudkowsky is a staunch, leading him to questionable conclusions through questionable thought experiments. To him, too many people experiencing a speck of dust in their eyes for a moment could be worse than torturing a man for 50 years. More specifically, Yudkowsky defends in this thought experiment.

Also, while his precise position on it is vague, he wrote a short sci-fi story where rape was briefly mentioned as legal. How the character remarking on it didn't seem to be referring to consensual sex like we do today didn't prevent a massive reaction in the comments section. He responded, "The fact that it's taken over the comments is not as good as I hoped, but neither was the reaction as bad as I feared." He described the science fiction world he had in mind as a "Weirdtopia" rather than a dystopia.