| It's not totally wrong, it's
LessWrong is a community blog focused on "refining the art of human rationality." To this end, it focuses on identifying and overcoming bias, improving judgment and problem-solving, and speculating about the future. The blog has long been dominated by the ideas of Eliezer Yudkowsky, a research fellow for the Machine Intelligence Research Institute (until recently known as the Singularity Institute for Artificial Intelligence); many members of LessWrong share Yudkowsky's interests in transhumanism, artificial intelligence, the Singularity, and cryonics.
The content of LessWrong is sometimes articulate, innovative, and thoughtful. However, the community's focused demographic and narrow interests have also produced an insular culture that is heavy with its own peculiar jargon and established ideas. Sometimes these ideas would benefit from a better grounding in reality.
In July of 2000, Eliezer Yudkowsky founded the nonprofit Singularity Institute for Artificial Intelligence (SIAI) to "create a friendly, self-improving artificial intelligence." In 2006, Yudkowksy began contributing to Overcoming Bias along with GMU economist Robin Hanson. After several years and increasing popularity, Yudkowksy started a collaborative blog/community to focus on some topics of particular interest to himself and SIAI, such as rationality, philosophy, AI, and transhumanism. Overcoming Bias remains as a "sister blog" to LessWrong, where Hanson and others discuss ideas stemming from Hanson's speculations on economics.
SIAI, where Yudkowsky remains a research fellow, hosts and maintains LessWrong to provide "an introduction to issues of cognitive biases and rationality relevant for careful thinking about optimal philanthropy and many of the problems that must be solved in advance of the creation of provably human-friendly powerful artificial intelligence." Yudkowsky considers LessWrong useful insofar as it advances SIAI's work, and the site is a key venue for SIAI recruitment and fundraising. The most popular post of all time on LessWrong, for example, is an assessment of SIAI by charity evaluator GiveWell. and the Extropians mailing lists (dating back to the 1990s). Accordingly, and because of the influence and ownership of the SIAI, LessWrong has essentially been a transhumanist community, emphasizing a focus on rationality per se in order to attract those who might otherwise be skeptical of apocalyptic AI. Increasingly since 2010, however, many of the newcomers to LessWrong have been introduced to the site from "Harry Potter and the Methods of Rationality", a popular work of Harry Potter fanfiction written by Yudkowksy. Community opinion on the future focus of LessWrong is uncertain, but discussions about rationality have led to the 2012 formation of the Center For Applied Rationality (CFAR). CFAR is devoted to researching methods of teaching rationalism, and holding retreats and summer camps to pass these methods on to others.
The core of LessWrong are its many parables, metaphors, and explanations of concepts in psychology and philosophy. The popularity of such essays as Yudkowsky's "The Genetic Fallacy," which explains the eponymous concept clearly and works out some of its potential complications, helped attract the growing community — even luring in those who might not be otherwise interested in transhumanism. While some critics have implied that this is perhaps some sort of deception, the peculiarly focused interests of LessWrong's most prominent members has no reflection on the usefulness of some of its great resources. In other words: just because Eliezer Yudkowsky wants to be a robot doesn't mean that his explanation of Bayes' Theorem isn't interesting and well-written.
Bayesianism, which uses that theorem to assess probabilities when making decisions, has become one of the hallmarks of LessWrong and its most frequent buzzword. Most of this jargon comes from the much-revered "Sequences," a long series of essays by Yudkowsky that are considered essential reading by members of the community. Their extraordinary length can be prohibitive, however (surpassing J.R.R. Tolkien's Lord of the Rings). Such time would probably be better spent reading some of the books written by the actual researchers behind the Sequences' concepts, such as Thinking, Fast and Slow by Daniel Kahneman, The Black Swan by Nassim Taleb, or Thinking and Deciding by Jonathan Baron — all of which are just as easily readable if not more so. While this does not negate the value of the Sequences, particularly if you are interested in the transhumanist ends to which many of them turn (neither Kahneman, Taleb, nor Baron address the potential yet highly speculative future problems of godlike AI) it does mean that the "required reading" status of the Sequences might be partially prompted by the same cognitive dissonance that helps perpetuate exclusive clubs: it was hard to read them all, but I did it, therefore they must be good.[wp]
Furthermore, when reading the Sequences, care should be taken to distinguish between fact and personal opinion of Yudkowsky himself. For example, the quantum physics sequence is a mishmash of a genuinely valuable intuitive explanation of the basic concepts of quantum physics, support for many-worlds (a mainstream, but minority interpretation of quantum mechanics) as if it was proven fact, wild speculation about the identity of consciousness and ill-defined concepts like "the Tao of Physics" (these are at least marked as speculative), and science fiction stories where the Yudkowsky stand-in debates with strawmen (a staple of most of his writing). An uncritical reader is likely to either get even more confused about quantum physics, despite Yudkowsky's promise to explain it in non-confusing terms, or worse, swallow Yudkowsky's personal, fringe views alongside mainstream science, forming a bond between the two in their worldview.
LessWrong's culture resembles, in most other respects, the standard set of male, middle-class Internet-libertarians so familiar in other places — including cringe-inducing discussions of the merits of racism. Notably, though, members of LessWrong are unusually concerned and active in charitable giving. They are also laudable for prizing accurate thinking over their personal viewpoints: it is not uncommon to witness someone actually change their mind when confronted with a good argument, a rarer phenomenon than one might think.
The system on which LessWrong operates is based on the "karma"-based system of Reddit. A post or comment that is deemed insightful will be promoted and highly visible, whereas too many downvotes will hide it. Because of this, LessWrong is almost completely troll-free. While this constant evaluation can be intimidating, members of the community generally take pride in voting for good thinking: wary of groupthink, they will usually endorse even the harshest of criticism, as long as it is intelligent.
A key part of the Less Wrong approach to human rationality is to avoid "fallacies of compression" and mistaking the map for the territory, which is the result of humans trying to fit a vastly huge universe into a relatively small and squishy piece of meat located between their ears. According to Yudkowsky, beliefs should constrain our expectations and those that are true no matter what we see are what constitutes blind faith — for example, two people arguing "whether a tree falling in a forest, with no one around to hear it, makes a sound" might argue yes and no based on different definitions of "sound" but wouldn't actually expect anything different. Precision, therefore, is the order of the day and is achieved through expectations and sensory anticipation, rather than merely saying something is true and arguing through clever wordplay.
At its best, LessWrong's articles really do articulate important aspects of human cognition. Such ideas as the "ugh field" — the instinctive distaste our minds have for difficult decisions — and the "affective death spiral" — in which praise for an entity turns into an endless cycle of greater praise — are valuable insights, and anyone aspiring to make better decisions should read them. Similarly useful are such techniques as the rationalist taboo, which requires speakers to discuss actual meaning rather than semantics, or "Crocker's Rules," which declare that the argument is on a touchy subject and that anyone who wishes to participate cannot claim offense. All of these efforts that are aimed at promoting better thinking and better decisions, and are accordingly commendable.
LessWrong is run under the umbrella of MIRI (formerly the Singularity Institute for Artificial Intelligence). Notable sponsors includes PayPal founder Peter Thiel, who has contributed 5 figure sums for several years. Smaller contributions are also solicited and accepted. A $118,000 loss due to theft was reported in 2009, and has not been recovered. MIRI uses its funds to recompense its officers (Yudkowsky makes about $80,000 per annum) and fund research, although no articles were placed in any peer reviewed journals until 2012. In that year, two articles were placed in the "International Journal of Machine Consciousness" and one in the "Journal of Consciousness Studies", both of which compare unfavorably to more prestigious journals in the field, such as "Artificial Intelligence." Requests for funding are regularly made on the LessWrong website and at Singularity Summit conferences, under the justification that MIRI is, uniquely, working to save the world from a threat to its very existence. Yudkowsky is happy to solicit financial contributions from non-wealthy individuals in order to save them from the speculative scenario of runaway Artificial Intelligence. This is nothing at all like televangelism, which aims to solicit financial contributions from non-wealthy individuals in order to save them from the speculative scenario of Hell.
LessWrong is mainly concerned with achieving accurate beliefs about the world, rather than achieving goals. The refusal to delve into contemporary politics or policy is held up as laudable, because it is seen as a way to preserve objective rationality. One of the most-cited and most popular phrases is "politics is the mind-killer," derived from the Yudkowsky essay of the same name, which argues that real discussion never occurs in a political context, because "winning" the discussion for your "side" becomes paramount, rather than reaching an optimal decision. While logical to the extent that this is an accurate criticism of political discourse, it's also essentially a declaration of surrender: "It's hard to stay rational in politics, so we'll just give up." If members of LessWrong truly are less biased in their thinking that the general public, as they've argued, then the more they succeed in drawing people into the fold, the more they will cede the field to the irrational.
The site has been accused of being a personality cult of Eliezer Yudkowsky, and does not reflect the other essayists who have become almost as influential. This has been enough of a concern that a code-word for "cult" was developed, "phyg," to opaquely discuss the issue.
While the appearance of a cult has faded, the like-mindedness that led to the criticism has not. LessWrong has a very deep but narrow set of demographics that have only slightly improved over the years - the same problem common to academic psychology and known as being "WEIRD": "Western, Educated, Industrialized, Rich, and Democratic" or also "White, Educated, Intelligent, Rich, and Democratic."
Another problem of LessWrong, is that its isolationism represents a self-made problem (unlike demographics). Despite intense philosophical speculation, the users tend towards a proud contempt of mainstream and ancient philosophy and this then leads to them having to re-invent the wheel. When this tendency is coupled with the metaphors and parables that are central to LessWrong's attraction it explains why they invent new terms for already existing concepts. The compabilitism position on free will/determinism is called "requiredism" on LessWrong, for example, and the continuum fallacy is relabeled "the fallacy of gray." The end result is a Seinfeldesque series of superfluous neologisms.
 Roko's Basilisk
The most prominent criticism to be made of LessWrong involves the incident of Roko's Basilisk. The absurdities involved beggar belief.
Yudkowsky has long been interested in the notion of future events "causing" past events: if you can plausibly forecast a future event, and then take actions in the present because of this prediction, then the future event "caused" your action, in some sense. This sort of thinking gets odd when you imagine superintelligences, because of all the extremes involved: their predictions of human behaviour may be near-perfect, as in Newcomb's paradox[wp], their power may be near-infinite, and the consequences could be near-eternal. Yudkowsky has also advocated utilitarianism, under which it would be justified to torture one person for 50 years to prevent dust specks in the eyes of sufficiently large numbers of people.
In July of 2010, Roko (a top contributor at the time) wondered if a future Friendly AI would punish people who didn't do everything in their power to further the AI research from which this AI originated, by at the very least donating all they have to it. He reasoned that every day without AI, bad things happen (150,000+ people die every day, war is fought, millions go hungry) and a future Friendly AI would want to prevent this, so it might punish those who understood the importance of donating but didn't donate all they could. He then wondered if future AIs would be more likely to punish those who had wondered if future AIs would punish them. That final thought proved too much for some LessWrong readers, who then had nightmares about being tortured for not donating enough to SIAI.
Yudkowsky replied to Roko's post calling him names and claiming that posting such things on an Internet forum could cause incalculable harm to the people who read it. Four hours later, he deleted Roko's post, including all comments. Roko left LessWrong, deleting his thousands of posts and comments. (He later briefly returned  and posted among other things that "I agree that the post in question should not appear in public" and "I wish I had never learned about any of these ideas".)
Yudkowsky later claimed the basilisk would not in fact work the way Roko had posited; but rather than simply explaining how such a reaction was inappropriate or how the ideas underlying it were flawed, he instead attempted to censor all discussion of the topic. The matter is now the occasional subject of contorted LW posts, as people try to discuss the issue without talking about what they're talking about, and is a reliable space-filler for journalists covering LW-related stories.
 See also
- AI-box experiment
- Game theory
- High IQ society
- The Problem With Rational Wiki
- Greg Egan disses SIAI
- PoE News on Lesswrong. In particular, "Pretend there is a website of trans-accountants who have never had an accounting job nor had any education in accounting."
- Alexander Kruel A blogger with some past association with LessWrong and current association with the transhumanist community. Often critical of LessWrong claims.
- ↑ SIAI's year 2000 990-EZ
- ↑ "About", Overcoming Bias
- ↑ "2012 Winter Fundraiser for the Singularity Institute", LessWrong
- ↑ SIAI's year 2009 990
- ↑ "important and interesting in proportion to how much it helps construct a Friendly AI"
- ↑ http://web.archive.org/web/20110621192259/http://singinst.org/achievements
- ↑ http://lesswrong.com/lw/3gy/tallinnevans_125000_singularity_challenge/
- ↑ "Thoughts on the Singularity Institute," LessWrong
- ↑ http://sl4.org/
- ↑ http://www.extropy.org/emaillists.htm
- ↑ Michael Vassar, the former president of Singularity Institute, who stepped down in January to pursue his idea for a personalized medicine startup–later bringing on Mr. Mowshowitz and Ms. Vance–admitted the nonprofit had learned to hide some of its more radical ideas, emphasizing rationality instead. As Mr. Yudkowsky put it, “There are plenty of people out there who would be interested in cognitive science-based thinking skills who wouldn’t necessarily buy into the whole ‘save humanity’ thing.”
- ↑ "Survey Results," LessWrong
- ↑ "2011 Survey Results," LessWrong
- ↑ "What We Do," CFAR
- ↑ "The Genetic Fallacy", LessWrong
- ↑ "Jargon", Lesswrongwiki
- ↑ "2012 Survey Results", LessWrong
- ↑ Comments on "Rationality Quotes, November 2012", LessWrong
- ↑ "Giving What We Can: 80,000 Hours and Metacharity", LessWrong
- ↑ "Ugh Fields", LessWrong
- ↑ "Affective Death Spirals", LessWrong
- ↑ I just donated to the SIAI
- ↑ 
- ↑ http://www.scimagojr.com/journalsearch.php?q=19900192593&tip=sid&clean=0
- ↑ http://www.scimagojr.com/journalsearch.php?q=15435&tip=sid&clean=0
- ↑ http://www.scimagojr.com/journalsearch.php?q=23675&tip=sid&clean=0
- ↑ FAQ
- ↑ "Politics Is the Mind-Killer", LessWrong
- ↑ "A Fable of Science and Politics", LessWrong
- ↑ "Participation in the LW Community Associated with Less Bias", LessWrong
- ↑ Our Phyg Is Not Exclusive Enough
- ↑ "Survey Results", LessWrong
- ↑ http://lesswrong.com/lw/17x/beware_of_weird_psychological_samples/
- ↑ "Train Philosophers with Pearl and Kahneman, Not Plato and Kant", LessWrong
- ↑ "Eliezer's Sequences and Mainstream Academia", LessWrong
- ↑ http://lesswrong.com/lw/r0/thou_art_physics/
- ↑ http://wiki.lesswrong.com/wiki/Timeless_decision_theory
- ↑ http://lesswrong.com/lw/kn/torture_vs_dust_specks/uf7
- ↑ Though a google cache of it survived - please do not read if you are prone to nightmares or OCD and/or are worried about causing incalculable harm to yourself.
- ↑ http://lesswrong.com/user/Roko
- ↑ http://lesswrong.com/user/formallyknownasroko
- ↑ http://lesswrong.com/lw/38u/best_career_models_for_doing_research/33w7
- ↑ http://lesswrong.com/lw/38u/best_career_models_for_doing_research/344l
- ↑ http://lesswrong.com/lw/39z/should_lw_have_a_public_censorship_policy/img
- ↑ http://lesswrong.com/lw/ds4/article_about_lw_faith_hope_and_singularity/73ao
- ↑ http://betabeat.com/2012/07/singularity-institute-less-wrong-peter-thiel-eliezer-yudkowsky-ray-kurzweil-harry-potter-methods-of-rationality/?show=all
- ↑ Mark Plus. A question about Less Wrong's Basilisk . The Life of Man Qua Man on Earth, by a Senior Cryonicist. 2012 October 31.
- ↑ Mark Plus. Re: A surprisingly positive response. New Cryonet. Yahoo! Groups. 2012 November 2.