Bronze-level article


From RationalWiki
Jump to navigation Jump to search
The Star of the House of Fëanor NATO compass rose LessWrong logo
Carefully, correctly
Icon lesswrong.svg
Singularity blues
Yeah, I was on LessWrong for quite a while, in a very low-key way. My period of time there basically went “These are people talking about interesting stuff. Admittedly they have a few odd beliefs like the cryonic thing, but interesting people.” “... apart from this virulent racist who keeps talking about IQ...” “... and all these people who keep talking about being ‘Pick-Up Artists’...” “my God, this place needs to be burned down and the earth salted!”
—Andrew Hickey[1]

LessWrong is a community blog focused on "refining the art of human rationality." To this end, it focuses on identifying and overcoming bias, improving judgment and problem-solving, and speculating about the future. The blog is based on the ideas of Eliezer Yudkowsky, a research fellow for the Machine Intelligence Research Institute (MIRI; previously known as the Singularity Institute for Artificial Intelligence, and then the Singularity Institute). Many members of LessWrong share Yudkowsky's interests in transhumanism, artificial intelligence, the Singularity, and cryonics.


Cover image of Harry Potter and the Methods of Rationality

In July of 2000, Eliezer Yudkowsky founded the nonprofit Singularity Institute for Artificial Intelligence (SIAI) to "create a friendly, self-improving artificial intelligence."[2] In 2006, Yudkowksy began contributing to Overcoming Bias[3] along with GMU economist Robin Hanson. After several years and increasing popularity, Yudkowksy had amassed a cult following and started a collaborative blog/community to focus on some topics of particular interest to himself and SIAI, such as rationality, philosophy, AI, and transhumanism.[4] Overcoming Bias remains as a "sister blog" to LessWrong, where Hanson and others discuss how human beings can compensate for natural biases (and ideas stemming from Hanson's speculations on economics).

MIRI, where Yudkowsky remains a research fellow, hosts and maintains LessWrong[5] to provide "an introduction to issues of cognitive biases and rationality relevant for careful thinking about optimal philanthropy and many of the problems that must be solved in advance of the creation of provably human-friendly powerful artificial intelligence."[6] Yudkowsky considers LessWrong useful insofar as it advances SIAI's work,[7] and the site is a key venue for SIAI recruitment[8] and fundraising.[9] The most popular post of all time on LessWrong, for example, is an assessment of SIAI by charity evaluator GiveWell.[10]

LessWrong originally attracted the bulk of its userbase from communities interested in transhumanism. In addition to Overcoming Bias, these communities include the SL4 mailing list[11] and the Extropians mailing lists[12] (dating back to the 1990s). Accordingly, LessWrong has long been an essentially transhumanist community, emphasizing a focus on rationality per se in order to attract those who might otherwise be skeptical of apocalyptic AI.[13]

Since 2010, many of the newcomers to LessWrong have been introduced to the site from "Harry Potter and the Methods of Rationality",[14][15][16] a popular work of Harry Potter fanfiction written by Yudkowksy. The future focus of LessWrong is unclear, but discussions about rationality have led to the 2012 formation of the Center For Applied Rationality (CFAR).[17][18] CFAR is devoted to researching methods of teaching rationalism, and holding retreats and summer camps to pass these methods on to others.[19]

After 2013, Yudkowsky himself stopped participating much in the site because it isn't fun anymore (though he still comments occasionally). Many others have followed suit. Yudkowsky and many other current and ex-LessWrongers now form what is colloquially referred to as the LessWrong Diaspora, in their own blogs, on Tumblr[20] and on LessWrong user Scott Alexander's blog Slate Star Codex.[21] Alexander refers to the 2015 participants as "some kind of pack of unquiet spirits who have moved in to haunt it after it got abandoned by the founding community members."[22]


The good[edit]

At their best, LessWrong's articles really do articulate important aspects of human cognition. The core of LessWrong are its many parables, metaphors, and explanations of concepts in psychology and philosophy. The popularity of such essays as Yudkowsky's "The Genetic Fallacy," which clearly explains the genetic fallacy and works out some of its potential complications,[23] helped attract the growing community — even luring in those who might not be otherwise interested in transhumanism. While some critics have implied that this is perhaps some sort of deception, the peculiarly focused interests of LessWrong's most prominent members has no reflection on the usefulness of some of its great resources (even if they're unacknowledged idea-for-idea popularisations of Kahneman).

A key part of the LessWrong approach to human rationality is to avoid "fallacies of compression" and mistaking the map for the territory, which is the result of humans trying to fit a vastly huge universe into a relatively small and squishy piece of meat located between their ears. According to Yudkowsky, beliefs should constrain our expectations and those that are true no matter what we see are what constitutes blind faith — for example, two people arguing "whether a tree falling in a forest, with no one around to hear it, makes a sound" might argue yes and no based on different definitions of "sound" but wouldn't actually expect anything different. Precision, therefore, is the order of the day and is achieved through expectations and sensory anticipation, rather than merely saying something is true and arguing through clever wordplay. Particularly useful ideas include:

  • The "ugh field"[24] — the instinctive distaste our minds have for difficult decisions
  • Arguments as soldiers — the instinct to defend any argument you've put forward, no matter how stupid
  • The "affective death spiral"[25] — in which praise for an entity turns into an endless cycle of greater praise
  • The rationalist taboo, which requires speakers to specify precisely what they mean instead of using more concise terms whose meanings might be vague
  • "Crocker's Rules", in which a participant in a debate invites frank responses and declares that they will not take offense. (In practice this is used by people who want to be free to give offense and reliably hit the roof when they're on the receiving end.) Lee Daniel Crocker credits Yudkowsky with actually coining the rule.[26]

All of these efforts that are aimed at promoting better thinking and better decisions, and are accordingly commendable.

In other words: just because Eliezer Yudkowsky wants to live as a simulation in a computer forever doesn't mean that his explanation[27] of Bayes' Theorem isn't interesting and well-written. (Bayesianism, which uses that theorem to assess probabilities when making decisions, has become one of the hallmarks of LessWrong and its most frequent buzzword.)

The meh[edit]

LessWrong has an extensive jargon[28] that mostly comes from the much-revered "Sequences",[29] a long series of essays by Yudkowsky considered essential reading by members of the community. Their extraordinary length can be prohibitive, however (surpassing J.R.R. Tolkien's Lord of the Rings).[citation needed]

Such time might be better spent reading some of the books written by the actual researchers behind the Sequences' concepts, such as The Fabric of reality by David Deutsch, Thinking, Fast and Slow by Daniel Kahneman, The Black Swan by Nassim Nicholas Taleb, or Thinking and Deciding by Jonathan Baron — all of which are easily readable. This does not negate the value of the Sequences, particularly if you are interested in the transhumanist and physical realist ends to which many of them turn.

The "required reading" status of the Sequences might be partially prompted by the same cognitive dissonance that helps perpetuate exclusive clubs: "it was hard to read them all, but I did it, therefore they must be good."Wikipedia

The bad[edit]

The good bits are not original and the original bits are not good. The well-written explanations of cognitive biases are taken idea-for-idea from Kahneman. Take everything with a grain of salt, and read the comments, which frequently carry detailed refutations (often downvoted).

It also contains ill-defined concepts like "The Tao of Physics" (which may or may not be exactly the same as Hume's principle of the uniformity of nature) and science fiction stories where the Yudkowsky stand-in debates with strawmen (a staple of most of his writing). An uncritical reader is likely to either get even more confused about the subject matter or, worse, swallow Yudkowsky's personal, fringe views alongside mainstream science, forming a bond between the two in their worldview.


Lack of application[edit]

Pretend there is a website of trans-accountants who have never had an accounting job nor had any education in accounting.
—Poe News[30]

LessWrong is mainly concerned — or claims to be concerned — with achieving accurate beliefs about the world, rather than achieving goals. The refusal to delve into contemporary politics or policy is held up as laudable, because it is seen as a way to preserve objective rationality.[31] One of the most-cited and most popular phrases is "politics is the mind-killer," derived from the Yudkowsky essay of the same name,[32] which argues that real discussion never occurs in a political context, because "winning" the discussion for your "side" becomes paramount, rather than reaching an optimal decision.[33] While logical to the extent that this is an accurate criticism of most of political discourse, it's also essentially a declaration of surrender: "It's hard to stay rational in politics, so we'll just give up."

If members of LessWrong truly are less biased in their thinking than the general public, as they've argued,[34] then the more they succeed in drawing people into the fold, the more they may cede the field to the irrational. This also leads to a preponderance of (a) Silicon Valley libertarianism (the default view of the participants) as the assumed neutral "not-politics" and (b) open slather for neoreactionaries, who largely incubated on OB/LW.


LessWrong's culture resembles, in most other respects, the standard set of predominately male, middle-class internet-libertarians[35] so familiar in other places — including cringe-inducing discussions of the merits of racism, which the neoreactionaries took as a welcoming signal.[36] Notably, though, members of LessWrong are unusually concerned and active in charitable giving.[37] They are also laudable for prizing accurate thinking over their personal viewpoints: it is not uncommon to witness someone actually change their mind when confronted with a good argument, a rarer phenomenon than one might think.

The site has been accused of being a personality cult of Eliezer Yudkowsky, and does not reflect the other essayists who have become almost as influential. Cultishness is heavily discussed on the site, both by Yudkowsky and others.[38] Amusingly enough, this led to some search engines suggesting "cult" as a related term to "Less Wrong" … in response to which, some users started using the code-word "phyg"[note 1] to mean "cult".[39]

While the appearance of a cult has faded, the like-mindedness that led to the criticism has not. LessWrong has a very deep but narrow set of demographics that have only slightly improved over the years[40] — the same problem common to academic psychology and known as being "WEIRD": "Western, Educated, Industrialized, Rich, and Democratic" or also "White, Educated, Intelligent, Rich, and Democratic."[41] Indeed, the site was strongly concerned about the impacts of growth.[42]


Another problem of LessWrong is that its isolationism represents a self-made problem (unlike demographics). Despite intense philosophical speculation, the users tend towards a proud contempt of mainstream and ancient philosophy[43] and this then leads to them having to re-invent the wheel. When this tendency is coupled with the metaphors and parables that are central to LessWrong's attraction, it explains why they invent new terms for already existing concepts.[44] Yudkowsky takes the position of "requiredism" on free will/determinism which he believes to be different from the mainstream compatibilism.[note 2] on LessWrong, for example, and the continuum fallacy is relabeled "the fallacy of gray."[46] The end result is a Seinfeldesque series of superfluous neologisms.

Although most posters don't consider Less Wrong to be "mainstream" philosophy, it has been compared to Wittgenstein who seems to best represent Yudkowsky and company's views on how language limits the ability for rationalists to communicate, and Quine,[47] whose approach to naturalism and science reflects the empiricism and reductionism of LW. Gary Drescher'sWikipedia excellent-but-dense Good and Real: Demystifying Paradoxes from Physics to Ethics covers a lot of the same ground as the Sequences and came out around when the Sequences started;[48] Yudkowsky had not read it before finishing them, but approves of the book.

Roko's Basilisk[edit]

See the main article on this topic: Roko's basilisk

The most prominent criticism to be made of LessWrong involves the incident of Roko's Basilisk. The absurdities involved beggar belief.

Yudkowsky has long been interested in the idea that you should act as if your decisions were able to determine the behavior of causally separated simulations of you:[49] if you can plausibly forecast a past or future agent simulating you, and then take actions in the present because of this prediction, then you "determined" the agent's prediction of you, in some sense.

The idea is that your decision, the decision of a simulation of you, and any prediction of your decision, have the same cause: An abstract computation that is being carried out. Just like a calculator, and any copy of it, can be predicted to output the same answer, given the same input. The calculator's output, and the output of its copy, are indirectly linked by this abstract computation. Timeless Decision Theory says that, rather than acting like you are determining your individual decision, you should act like you are determining the output of that abstract computation.

This sort of thinking gets odd when you imagine superintelligences, because of all the extremes involved: their predictions of human behaviour may be near-perfect, as in Newcomb's paradoxWikipedia, their power may be near-infinite, and the consequences could be near-eternal. Yudkowsky has also advocated utilitarianism, saying that it would be justified to torture one person for 50 years to prevent dust specks in the eyes of sufficiently large numbers of people (a ridiculously huge number that cannot be completely written in the universe).[50]

In July of 2010, Roko (a top contributor at the time)[citation needed] wondered if a future Friendly AI would punish people who didn't do everything in their power to further the AI research from which this AI originated, by at the very least donating all they have to it. He reasoned that every day without AI, bad things happen (150,000+ people die every day, war is fought, millions go hungry) and a future Friendly AI would want to prevent this, so it might punish those who understood the importance of donating but didn't donate all they could. He then wondered if future AIs would be more likely to punish those who had wondered if future AIs would punish them. That final thought proved too much for some LessWrong readers, who then had nightmares about being tortured for not donating enough to SIAI.

Yudkowsky replied to Roko's post calling him names, claiming that posting such things on an Internet forum "potentially gives superintelligences an increased motive to do extremely evil things in an attempt to blackmail us", and that users had told him of nightmares prompted by the post. Four hours later, he deleted Roko's post,[51] including all comments. Roko left LessWrong, deleting his thousands of posts and comments.[52] (He later briefly returned[53] and posted among other things that "I agree that the post in question should not appear in public"[54] and "I wish I had never learned about any of these ideas".[55])

What you know: When Roko posted about the Basilisk, I very foolishly yelled at him, called him an idiot, and then deleted the post.

Why I did that is not something you have direct access to, and thus you should be careful about Making Stuff Up, especially when there are Internet trolls who are happy to tell you in a loud authoritative voice what I was thinking, despite having never passed anything even close to an Ideological Turing Test on Eliezer Yudkowsky.

Why I yelled at Roko: Because I was caught flatfooted in surprise, because I was indignant to the point of genuine emotional shock, at the concept that somebody who thought they'd invented a brilliant idea that would cause future AIs to torture people who had the thought, had promptly posted it to the public Internet. In the course of yelling at Roko to explain why ths was a bad thing, I made the further error---keeping in mind that I had absolutely no idea that any of this would ever blow up the way it did, if I had I would obviously have kept my fingers quiescent---of not making it absolutely clear using lengthy disclaimers that my yelling did not mean that I believed Roko was right about CEV-based agents[56] torturing people who had heard about Roko's idea. It was obvious to me that no CEV-based agent would ever do that and equally obvious to me that the part about CEV was just a red herring; I more or less automatically pruned it from my processing of the suggestion and automatically generalized it to cover the entire class of similar scenarios and variants, variants which I considered obvious despite significant divergences (I forgot that other people were not professionals in the field). This class of all possible variants did strike me as potentially dangerous as a collective group, even though it did not occur to me that Roko's original scenario might be right---that was obviously wrong, so my brain automatically generalized it.

At this point we start to deal with a massive divergence between what I, and several other people on LessWrong, considered to be obvious common sense, and what other people did not consider to be obvious common sense, and the malicious interference of the Internet trolls at RationalWiki.

What I considered to be obvious common sense was that you did not spread potential information hazards because it would be a crappy thing to do to someone. The problem wasn't Roko's post itself, about CEV, being correct.[57]

Yudkowsky later claimed the basilisk would not in fact work the way Roko had posited; but rather than simply explaining how such a reaction was inappropriate or how the ideas underlying it were flawed, he instead attempted to censor all discussion of the topic. The matter is now the occasional subject of contorted LW posts, as people try to discuss the issue without talking about what they're talking about,[58][59] and is a reliable space-filler for journalists covering LW-related stories.[60]

The basilisk kerfuffle has also alienated fellow cryonicists.[61][62]

How Yudkowsky deals with critics[edit]

One knowledgeable critic of LessWrong had started out buying into their claims, then was seriously put off by the response to Roko's Basilisk and established a popular reference site on the topic, had various LW members following him around the net to harass him, calling him a liar wherever they found him.[citation needed] He posted to LessWrong saying that the ongoing harassment had been affecting his health, and asking how it could stop.[63]

Yudkowsky responded with the following, which is no doubt an entirely reasonable thing to ask someone to do:[64]

You can update by posting a header to all of your blog posts saying, "I wrote this blog during a dark period of my life. I now realize that Eliezer Yudkowsky is a decent and honest person with no ill intent, and that anybody can be made to look terrible by selectively collecting all of his quotes one-sidedly as I did. I regret this page, and leave it here as an archive to that regret." If that is how you feel and that is what you do, I will treat with you starting from scratch in any future endeavors. I've been stupid too, in my life. (If you then revert to pattern, you do not get a second second chance.)

Note that most of these blog posts don't even mention Yudkowsky, much less quote mine him. But still, he wants his own header on top of every single blog post.

This was eventually resolved by removing most of the dispute-related content, placing disclaimers where relevant, and letting the matter go.


LessWrong is run under the umbrella of MIRI. Notable sponsors include PayPal founder and Donald Trump delegate Peter Thiel, who contributed five-figure sums for several years. Many donations of late come from cryptocurrency winners; Ethereum founder Vitalik Buterin was the largest single donor in 2017.[65] Smaller contributions are also solicited and accepted.[66] A $118,000 loss due to theft was reported in 2009, and has not been recovered.[67] MIRI uses its funds to recompense its officers (Yudkowsky makes about $80,000 per annum) and fund research, although no articles were placed in any peer reviewed journals until 2012. In that year, two articles were placed in the "International Journal of Machine Consciousness"[68] and one in the "Journal of Consciousness Studies",[69] both of which compare unfavorably to more prestigious journals in the field, such as "Artificial Intelligence."[70] Requests for funding are regularly made on the LessWrong website and at conferences, under the justification that MIRI is, uniquely, working to save the world from a threat to its very existence. Yudkowsky is happy to solicit financial contributions even from non-wealthy individuals, in order to "save" them from the speculative scenario of runaway Artificial Intelligence.

GiveWell assessment[edit]

GiveWell's 2012 evaluation of SIAI as a charity[10] sets out a number of problems with the organisation, the site and its aims as compared to its verifiable results:

  • SIAI's argument for its work is poorly advanced
  • Its arguments involve huge consequences of small probabilities (Pascal's muggingWikipedia)
  • The artificial intelligence propositions advanced do not engage mainstream research and are not endorsed by active researchers in the field
  • The group shies away from putting its claims to tests
  • Apparent poorly grounded belief in the group's superior general rationality (the point of LessWrong)
  • Overall disconnect between SI's goals and its activities

GiveWell's recommendation was that at this stage (2012), donating to SIAI would work against the organisation's stated goals, which is approximately the worst thing you could say about a charity.

Holden Karnofsky of GiveWell has since walked back his 2012 opinion, and the effective altruism subculture is very into Lesswrong these days — because they've been driving the remaining sensible people out.[71]

In popular culture[edit]

  • Zendegi by Greg Egan (2010) features Nate Caplan, who wants to be uploaded. "My IQ is one hundred and sixty ... You can always reach me through my blog, Overpowering Falsehood dot com, the number one site for rational thinking about the future —" The novel also features the Benign Superintelligence Bootstrap Project, which persuades a billionaire to donate his fortune, hoping that the "being of truly God-like powers" will grant him immortality come the Singularity. He dies disappointed, and the Project "turn[s] five billion dollars into nothing but padded salaries and empty verbiage."[72]
  • Elementary S3E4, "Bella" (2014), features an artificial intelligence and the theory that a member of the Existential Threat Research Association killed its creator to increase the credibility of their anti-AI message. Also includes discourse on "the terror of mortality" and some I-know-that-you-know-that-I-know discussions between Sherlock and his suspect.[73]
  • The Basilisk Murders: A Sarah Turner Mystery by Andrew Hickey. A satirical murder mystery set in the world of transhumanists and Singularitarians.
  • Ex Machina, a movie about a reclusive AI genius who resembles Yudkowsky physically, but not in his ability to actually build stuff.

See also[edit]

External links[edit]


  1. The simple Caesar cipher widely known since the days of usenet as ROT13 transforms "cult" to "phyg" and back again.
  2. Eliezer Yudkowsky, Thou Art Physics:

    “Compatibilism” is the philosophical position that “free will” can be intuitively and satisfyingly defined in such a way as to be compatible with deterministic physics. “Incompatibilism” is the position that free will and determinism are incompatible.
    My position might perhaps be called “Requiredism.” When agency, choice, control, and moral responsibility are cashed out in a sensible way, they require determinism—at least some patches of determinism within the universe.[45]


  1. Not a review of neoreaction a basilisk Retrieved 18th Dec 2018
  2. SIAI's year 2000 990-EZ
  4. "About", Overcoming Bias
  5. "2012 Winter Fundraiser for the Singularity Institute", LessWrong
  6. SIAI's year 2009 990
  7. "important and interesting in proportion to how much it helps construct a Friendly AI"
  10. 10.0 10.1 "Thoughts on the Singularity Institute," LessWrong
  13. Nitasha Tiku, Faith, Hope, and Singularity: Entering the Matrix with New York’s Futurist Set. Archived from, 25 July 2012.

    Michael Vassar, the former president of Singularity Institute, who stepped down in January to pursue his idea for a personalized medicine startup–later bringing on Mr. Mowshowitz and Ms. Vance–admitted the nonprofit had learned to hide some of its more radical ideas, emphasizing rationality instead. As Mr. Yudkowsky put it, “There are plenty of people out there who would be interested in cognitive science-based thinking skills who wouldn’t necessarily buy into the whole 'save humanity' thing."

  14. HPMOR
  15. "Survey Results," LessWrong
  16. "2011 Survey Results," LessWrong
  17. Center For Applied Rationality: Developing clear thinking for the sake of humanity’s future.
  18. Welcome to CFAR! Archived from the Center for Applied Rationality, 21 July 2012.
  19. What We Do. Archived from the Center for Applied Rationality, 7 July 2013.
  20. Rationalist Masterlist (yxoque, 9 June 2015)
  23. "The Genetic Fallacy", LessWrong
  24. "Ugh Fields", LessWrong
  25. "Affective Death Spirals", LessWrong
  28. "Jargon", Lesswrongwiki
  30. I fucking hate these people so fucking much. Archievd from, 22 July 2013.
  31. Is it OK to talk about politics? LessWrong wiki.
  32. "Politics Is the Mind-Killer", LessWrong
  33. "A Fable of Science and Politics", LessWrong
  34. "Participation in the LW Community Associated with Less Bias", LessWrong
  35. "2012 Survey Results", LessWrong
  36. Comments on "Rationality Quotes, November 2012", LessWrong
  37. "Giving What We Can: 80,000 Hours and Metacharity", LessWrong
  38. Every Cause Wants To Be A Cult
  39. Our Phyg Is Not Exclusive Enough
  40. "Survey Results", LessWrong
  43. lukeprog, Train Philosophers with Pearl and Kahneman, Not Plato and Kant, LessWrong, 5 December 2012.
  44. lukeprog, Eliezer's Sequences and Mainstream Academia. LessWrong, 14 September 2012.
  45. Eliezer Yudkowsky, Thou Art Physics. LessWrong, 6 June 2008.
  46. Eliezer Yudkowsky, The Fallacy of Gray. LessWrong, 7 January 2008.
  47. lukeprog, Rationality and Mainstream Philosophy. LessWrong, 20 March 2011.
  48. Gary Drescher, Good and Real: Demystifying Paradoxes from Physics to Ethics. MIT Press, 5 May 2006. ISBN 9780262042338
  51. Solutions to the Altruist's burden: the Quantum Billionaire Trick. Archived from a snapshot of Google's cache, 23 July 2010. Please do not read if you are prone to nightmares or OCD and/or are worried about causing incalculable harm to yourself.
  56. Coherent Extrapolated Volition. LessWrong.

    Coherent Extrapolated Volition was a term developed by Eliezer Yudkowsky while discussing Friendly AI development. It’s meant as an argument that it would not be sufficient to explicitly program what we think our desires and motivations are into an AI, instead, we should find a way to program it in a way that it would act in our best interests – what we want it to do and not what we tell it to.

  61. Mark Plus. A question about Less Wrong's Basilisk . The Life of Man Qua Man on Earth, by a Senior Cryonicist. 2012 October 31.
  62. Mark Plus. Re: A surprisingly positive response. New Cryonet. Yahoo! Groups. 2012 November 2.
  66. I just donated to the SIAI
  67. BrandonReinhart (2 May 2011). "SIAI - An Examination"
  71. Dylan Matthews, I spent a weekend at Google talking with nerds about charity. I came away … worried. Vox, 10 August 2015.