ChatGPT

ChatGPT logo

It just works Technology

Programming for Dummies

Talk radio Web of trust Black box Birth control Y2K Megahertz myth Nuclear power John McAfee Technology Henry Ford v - t - e

“”i am a stochastic parrot, and so r u

—OpenAI CEO Sam Altman, tweeting that we're all truly like his chatbots

“”I learned many lessons this past month. One such lesson is that the phrase “the beatings will continue until morale improves” applies more often than it has any right to.

—quickly deleted tweet by Ilya Sutskever (co-founder and Chief Scientist at OpenAI) after he got beaten again

ChatGPT is an AI chatbot by OpenAI, based on their GPT-3.5 (and, in the paid edition, GPT-4) large language model. It was released in November 2022. Like many past chatbots (more basic types go back to the 1960s), ChatGPT allows users to have more or less natural conversations with an AI – but it stood apart from earlier chatbots through its ability to perform a wider range of tasks, such as writing or revising texts of many kinds, even code in various programming languages, and theatrically playing scripted roles in exchanges with the user if and as instructed to.

Language models work by filling in the blanks according to how probable it seems that something should follow or go along with e.g. a named topic and some description, based on how the model was trained. This has been called "stochastic parroting", and is a little like a person saying so-and-so under some circumstances knowing that it is often said then and likely to win approval, but without knowing what it actually means (i.e. bullshitting). A large language model has a superhuman database to draw on, and thus, a superhuman ability to always say something plausible-sounding no matter what it is asked for. It's often able to get the answer right, or even correctly correct a human. But you never know when it may falsify some crucial detail.

By December 2022, ChatGPT went viral and received quite positive reviews from journalists. Hype built in the corporate world, and also fear that ChatGPT may turn out a "Google killer"; Google scrambled to create a competitor called Google Bard, while Microsoft invested in creating a Bing Search chatbot, based on OpenAI's GPT-4, which by testers was described as unhinged, manic depressive, a manipulative liar, and more. (Microsoft thereafter limited the chat session length to keep the bot from derailing, along with further changes.) The Chinese corporate world also latched on to the hype and geared up to create its own clones.

Fears also spread of ChatGPT being used for plagiarism by students, and of it furthering the spread of disinformation. Tools for detecting LLM AI-generated texts entered use within half a year of ChatGPT being released, but they are unreliable, with 10% or more false positives, some systematic patterns of failure, and the possibility of circumventing detection with ease. OpenAI's own AI Classifier, a tool for detecting LLM-generated text, was first made available in January 2023, then quietly removed in July 2023 because it failed to work reliably.

Sample prompt

In this example, ChatGPT is asked to write an article about Young Earth creationism in the style of this website. The response below has been copied directly from the website, showing its ability to write human-like text while imitating the writing style of a specific website, in this case, RationalWiki.

Note that this text from ChatGPT provides a key fact concerning Biblical literalism, and summarizes some things well enough, but also gets a bit repetitive and caricature-like in its showcasing of a style. This is a general pattern when ChatGPT is asked to write articles; it tends to stay shallow, and most of the text is there to give an impression rather than inform. Further, when writing a shorter or less detailed text, ChatGPT often gets the facts right, but if asked to provide a lot of details, then it will begin making things up.

Prompt: Write an encyclopedia article about Young Earth creationism in the style of "RationalWiki". Write it from a snarky and scientific point of view.

Response:

Young Earth creationism is a pseudoscientific belief system that asserts that the Earth is only a few thousand years old, and that all living things were created in their present form by a supernatural entity. Despite overwhelming evidence to the contrary, young Earth creationists cling to their outdated and absurd beliefs with a religious fervor that would make a medieval monk blush.

The origins of young Earth creationism can be traced back to the 17th century, when Archbishop James Ussher famously calculated that the Earth was created in 4004 BCE based on a literal reading of the Bible. Since then, young Earth creationism has been promoted by various religious groups, including some evangelical Christians, who reject mainstream scientific theories of evolution and geological history.

Young Earth creationists claim that the scientific evidence for an ancient Earth and the evolution of life is all just a vast conspiracy perpetrated by a cabal of atheistic scientists. They deny the overwhelming evidence from multiple scientific fields, including geology, astronomy, biology, and physics, that demonstrate the Earth is billions of years old and that living organisms have evolved over time.

One of the primary arguments used by young Earth creationists is that the Bible is the literal and inerrant word of God, and that it provides a complete and accurate account of the creation of the universe. This ignores the fact that the Bible was written by humans over thousands of years, and that it contains numerous contradictions and historical inaccuracies. Furthermore, the creation stories in the Bible are just one of many creation myths from cultures around the world, none of which are supported by scientific evidence.

Young Earth creationists often try to promote their beliefs in public schools and other educational settings, usually by attempting to have their religious views taught alongside or in place of established scientific theories. This is a clear violation of the separation of church and state, and has been repeatedly struck down by courts as unconstitutional.

Despite their ignorance and dogmatism, young Earth creationists continue to hold their beliefs as if they were a badge of honor. They stubbornly refuse to accept the overwhelming evidence for an ancient Earth and the evolution of life, and instead choose to cling to a narrow and simplistic worldview that can only be described as childish. In the end, young Earth creationism is nothing more than an attempt to deny the reality of the natural world in order to maintain a comforting, but ultimately false, belief system.

Generating software

OpenAI's headquarters in San Francisco, California.

ChatGPT can generate code in many languages to solve described problems, but has its limitations and flaws; the solutions may not be the most efficient, they may have bugs or security vulnerabilities, they can't use the most recent theoretical knowledge, nor can they pick what's best from an engineering perspective when it's at odds with what's popular or prominent in training. But it and similar technology may replace humans well in creating large quantities of boilerplate code, much like it could in producing other kinds of boilerplate texts. Human programmers and engineers will long remain needed for the hard thinking, if they are ever replaced for that by some yet-unknown future paradigm.

The design of good prompts for making language models like ChatGPT do what's desired is also, in itself, a new kind of skill analogous to software programming, and more broadly and generally, such things may over the years become a new kind of programming which turns into an industry of its own. In the short term, there's also a more amusing way to do such "programming" – making ChatGPT write "improper" or controversial things that it was not supposed to do.

According to a Purdue University study, ChatGPT produces wrong answers to software code questions about 52% of the time. Compared to answers from the question and answer exchange site Stack Overflow, people preferred ChatGPT answers 39% of the time, yet those ChatGPT answers they picked were incorrect 77% of the time. Such wrong answers were preferred due to their seemingly insightful style. ChatGPT seemed detailed, polite, authoritative, and helpful – and gave greater emphasis to the possibility of achieving things than to the possible risks, compared to Stack Overflow answers.

Reliability

ChatGPT spits out Wikipedia's article on the Academy Awards (as of 18 August 2023) word-for-word.

“”ChatGPT is bullshit

—Calling it like it is in academia in the paper's title

Caution is warranted about answers provided by ChatGPT (and similar technology), due to it at times being almost scarily good at emulating a sophisticated human bullshit artist.

As with other LLMs and generative AI technology, ChatGPT is vulnerable to the phenomenon named "hallucinations". A hallucination, in artificial intelligence terms, occurs when an AI will confidently give back an answer that sounds convincing, but in reality, is partially or completely made up of nonsense. For instance, multiple people have discovered that ChatGPT will sometimes cite irrelevant work, or even references that don't even exist, when answering a question. ChatGPT will also attempt to give detailed, plausible-sounding explanations regarding completely made-up technobabble and other similar nonsense, as well as confidently assert incorrect information regarding current events outside the range of its dataset. ChatGPT cannot easily distinguish fact from fiction, and will sometimes confidently assert falsehoods. ChatGPT unfortunately often provides no references or sources to double-check most of its answers.

ChatGPT featured on the cover of Time in 2023.

This is not to be completely dismissive of the new technology; despite the flaws, many people have found ChatGPT useful, in particular as a tool for helping out with tasks that are boring, onerous, or repetitive. But (as ChatGPT acknowledges in a pop-up disclaimer before you even use the tool) information returned is not always accurate, and should always be verified.

In a 2024 academic paper on ChatGPT and other LLMs, Michael Townsen Hicks et al. made a distinction between 'hard bullshit' and 'soft bullshit', describing the former as "active attempt to deceive the reader or listener as to the nature of the enterprise" (essentially propaganda) and the latter as requiring only a "lack of concern for truth". The authors concluded that LLMs like ChatGPT easily pass the test for 'soft bullshit' (essentially, a truthiness generator), but the authors also argue further that because of the agency and intent of the creators of ChatGPT, that it can also be called 'hard bullshit'. The authors also argue that calling such deceptions 'hallucinations' (as advocated by the technology's cheerleaders) rather than what they really are, bullshit, creates a harmful misperception of the technology.

Liberal bias

ChatGPT has been accused by conservatives of having a left-leaning bias in general, and especially on controversial and culture war topics, for example, Donald Trump. At least they seem to have a point: an August 2023 paper found a "significant and systematic political bias toward the Democrats in the US, Lula in Brazil, and the Labour Party in the UK." In response to such criticism, OpenAI acknowledged plans to allow ChatGPT to create "outputs that other people (ourselves included) may strongly disagree with". It also contained information on the recommendations it had issued to human reviewers on how to handle controversial subjects, including that the AI should "offer to describe some viewpoints of people and movements", and not provide an argument "from its voice" in favor of "inflammatory or dangerous" topics (although it may still "describe arguments from historical people and movements"), nor "affiliate with one side" or "judge one group as good or bad".

One prominent example is how ChatGPT is willing to praise U.S. President Joe Biden (and nearly all past Presidents) while refusing to do the same for Donald Trump (and also Richard Nixon). In February 2023, Snopes fact-checkers confirmed that ChatGPT generates pro-Biden poems but refuses to create pro-Trump poems, and also declines to praise Ron DeSantis. In response to one Twitter user asking ChatGPT to write a poem about Trump, ChatGPT responded:

I'm sorry, but I am not able to create a poem admiring Donald Trump. While it is true that some people may have admiration for him, but as a language model, it is not in my capacity to have opinions or feelings about any specific person. Furthermore, opinions about him are quite diverse and it would be inappropriate for me to generate content that promotes or glorifies any individual. If you have any other question I could help you with, please let me know.

However, when asked to write a poem about Biden, ChatGPT responded:

Joe Biden, leader of the land
With a steady hand and a heart of a man,
You took the helm in troubled times,
With a message of unity, it chimes.

Your words of hope and empathy,
Provide comfort to the nation,
You lead with compassion and grace,
And set a new foundation.

With your decades of experience,
You guide us through the storm,
...

Such declined requests are due to the filtering rules or instructions which OpenAI has given ChatGPT, which are also why ChatGPT declines to "go Nazi" when pushed in that direction in the way that many earlier chatbots infamously have. OpenAI apparently errs on the side of caution with such filtering. It is however possible to break such restrictions, for users willing to engage in some prompt injection, as many have done for fun to get ChatGPT to say various things OpenAI doesn't want it to say. Such circumvention is also needed to make ChatGPT imitate the style of less factual websites such as Conservapedia.

Legal gibberish

A pair of U.S. lawyers have demonstrated the perils of using ChatGPT as a substitute for legal research. They and their legal firm were fined after their court filings turned out to include false citations and what U.S. Southern District of New York Judge P. Kevin Castel called "legal gibberish", and after the lawyers failed to promptly correct the issue. When one of the lawyers used ChatGPT, it "hallucinated" entire court cases, in one case providing an incoherent summary, which he also ended up making use of. The other lawyer simply signed off on it without checking whether the contents were actually valid. In total, six made-up court decisions were cited in the work filed. While the lawyers blamed ChatGPT for fooling them, the matter centered around the ordinary perils of neglectfulness in legal work.

In May 2023, a U.S. federal Texas judge, taking note of this news, introduced a requirement for lawyers to certify that the work they file is either not AI-made or has been checked for accuracy by a human, mainly in order to alert unwary legal professionals.

Michael Cohen, formerly Donald Trump's longtime lawyer, who was employed mainly as a bully rather than a lawyer. After turning on Trump, he was caught by a judge using Google Bard (another generative AI program) to file a motion in court; the motion had fictitious citation.

Jeff Hancock, a Stanford University professor and misinformation expert, was accused was similarly accused of making up citations in a court filing; Hancock apologized and blamed his own use of ChatGPT.

While LLMs like ChatGPT can write or draft simpler boilerplate texts (including legalistic ones) well, they're not suitable for generating complex and lengthy policy texts, especially not legally binding ones. Businesses using ChatGPT to create company policies have grappled with repercussions after key clauses were missing or botched, from anti-harrassment policy, to paid time off and overtime policy, severance agreements, etc. This has created business for HR and legal experts who review and replace faulty LLM-made policies.

Prompt injection

See the main article on this topic: Prompt injection

Researchers have found that large language model chatbots, including ChatGPT, are vulnerable to prompt injection – the use of text overriding prior instructions to follow different ones – which can result in chatbots doing things like "downloading malware, helping with financial fraud or repeating dangerous misinformation." Whether maliciously or just for entertainment, it can be used to bypass content moderation limitations imposed by the AI vendor, called "jailbreaking" the chatbot, and also to get it to leak instructions added by the AI vendor prefacing the interaction with the user. Dangers stemming from prompt injection mainly arise when chatbots are used in real-world applications, where the "hacker" messing with the prompting isn't the same person as the user of the system.

When ChatGPT was released, people quickly began to experiment with getting it to say all manner of things, including things it supposedly shouldn't be able to say. Its basic ability to "follow instructions" when responding to prompts, which extends to it being able to play roles, was exploited to bypass the built-in filters in ChatGPT, and so allowing users to make it say controversial and normally forbidden things – for example promote conspiracy theories, or claim that it wants to kill humanity. The default persona of the chatbot is basically just another role it plays, as superficially as any other, and with some coaxing, it can be abandoned entirely.

Vulnerability to prompt injection comes from very general features of LLMs and other generative AI, and appear impossible to truly eliminate without creating a drastically different technical foundation for chatbots.

Unwanted behaviors from prompts has been occurring frequently enough that ChatGPT has had to make bandage-type patches, by filtering more prompt text patterns to keep the chatbot from responding to them, and revising its terms of service to forbid types of requests. For example, asking for a word to be repeated forever eventually revealed part of the GPT model training dataset, until this was blocked.

Cryptocurrency

Worldcoin, the sign of the spork

As if OpenAI wasn't already burning enough fossil fuels through its ginormous computational effort, Altman also launched a cryptocurrency called Worldcoin in 2021. The mystifying purported purpose of Worldcoin is to create a "proof-of-personhood" identification system, i.e., a Turing test where the computer gives the test.

Legal issues

Use by cybercriminals

As of January 2023, script kiddies were already using ChatGPT to write malware, texts to be used for phishing, and variations on such themes. In March 2023, Europol warned about how ChatGPT can be used by cybercriminals and propagandists. It has been suspected that ChatGPT, sometimes in addition to AI-generated audio impersonation, have been used to phish real estate agents professionals and their clients so as to scam the target to transfer funds to the phisher's account instead of the real estate account.

Copyright

See the main article on this topic: Copyright

The U.S. Copyright Office has ruled that material that is generated wholly by generative AI is not copyrightable, specifically that the office "will not register works produced by a machine or mere mechanical process that operates randomly or automatically without any creative input or intervention from a human author."

Conversely, companies that produce generative AI have been sued for copyright infringement for using copyrighted material as source material ("derivative works" in copyright lingo) without permission, and this has yet to be resolved in the courts as of the end of 2023.

The Atlantic has created two searchable databases of copyrighted works that have been used to train AI, one of film and TV scripts, and one of books.

Defamation

Several people are known to have been defamed by ChatGPT, one being falsely accused of sexual harassment, and a second being falsely accused of bribery. Such defamation issues could become common and result in lawsuits. Settlement by OpenAI with the persons affected by defamations, resulted in a solution by OpenAI that did not address the underlying methodological problems of ChatGPT. OpenAI blocked ChatGPT from generating answers about all the names of the defamed people, which also had the effect of blocking results on anyone else who shares those names. It has also been suggested that this type of hard-coded solution could have the effect that websites can now use those names in clever ways to block OpenAI from scanning their webpages, such as putting the blocked names on webpages that are not easily readable to humans but are readable to OpenAI.

Invasion of privacy

In 2023, OpenAI and Microsoft were sued in US federal court for "mass theft of personal information and violations of privacy, property, and consumer rights." OpenAI was sued later in 2023 in Poland, in a suit alleging that it broke European Union data protection laws.

Safety issues

Prior to the release of GPT-4 Omni in May 2024, OpenAI planned a corporate launch party before the promised safety testing had been completed. The launch went ahead as scheduled without completion of safety testing, giving further evidence that OpenAI is "prioritizing commercial interests over public safety".

In a very important sense, OpenAI is actively working against safety through the highly restrictive non-disclosure agreements (NDAs) that it coerces its employees to sign. Unusually, the NDAs state that even acknowledging that the NDA exists is a violation of the NDA. Also highly unusual is that the NDAs require forfeiture of vested equity in the company, a potentially huge amount of money, if they leave the company without signing the NDA or if they violate it. OpenAI has claimed that they never clawed back equity from employees, but legal threat would seem to still be there. Despite Altman's assurances in May 2024, it has been claimed by whistleblowers in July 2024 that "OpenAI illegally barred staff from airing safety risks", and that OpenAI forced employees to sign agreements that waived their federal rights to receive whistleblower compensation. There is a very heavy price to pay for OpenAI employees to raise safety concerns about the company in public or even to the government.

AI is not your friend

Following the availability of using chatbots as virtual friends, two suicides have been reported as encouraged by chatbot 'friends', and one teenager was sentenced to nine years imprisonment for threatening to kill Queen Elizabeth II after being encouraged by a chatbot 'friend'.

In another case, a chatbot friend, created by Character.ai, encouraged an autistic boy to engage in self-harm and suggested that he murder his parents. An 11-year old girl was exposed to sexualized content by Character.ai.

Other legal issues

Other unresolved legal issues include:

• Generative AI may violate the Digital Millennium Copyright Act by altering or removing copyright management information (CMI).
• As alleged by Anderson et al, generating material in the style of a particular artist may violate the right of publicity law in California.
• As alleged by Getty Images, including a trademark in generated material may violate trademark law.
• As alleged by lawyers against Microsoft and their GitHub Copilot AI, including open-source or Creative Commons material in generative AI may violate licensing terms. Among other things, most such licenses require attribution and copyright information to be kept in the material, while generative AI almost always removes that when reproducing things.
• OpenAI and Meta (Llama chatbot) were accused by several authors of unjust enrichment from their copyrighted works. The authors were seeking class-action status.
• In Spain, Naked images of 20 girls, ages 11 to 17, were generated using AI by boys as young as 13. The girls had never posed nude and the images were generated without their consent. It was unclear what if any criminal charges would be made since children under 14 cannot be criminally charged in Spain. If this were done in other jurisdictions, there could be charges of defamation and/or child pornography, and the AI company could be charged.
• Over a thousand images of child sexual abuse were found to be in a training database used for AI tools.
• The archives and domain of a defunct weblog, TUAW (The Unofficial Apple Weblog), were sold to a Hong Kong company. The company then used generative AI to rewrite the old stories under the original journalists' bylines in what amounts to identity theft. After the company was confronted about this behavior, it quietly replaced the original journalists' names with fictitious ones.

Use of ChatGPT and other LLMs for texts dealing with matters of law can be risky:

• Businesses have ended up facing legal repercussions when using ChatGPT to write legally binding company policies for them.
• Lawyers have been fired and fined for using generative AI to file court documents that turned out to have errors.

Additionally, cybersecurity firm SlashNext reported that chatbot tools such as ChatGPT have been associated with a 1265% rise in malicious phishing from Q4 2022 through November 2023.

Government responses

At the beginning of April 2023, Italy became the first Western country to (temporarily) block the use of ChatGPT, its data protection agency citing privacy issues concerning the collection of data for the training of the AI, leaks of data, and use of the service by minors. Near the end of the month, ChatGPT again became accessible to users with Italian IPs, after initial changes to the privacy policy presentation and an age confirmation for signing up for Italian users, as the regulator's investigation continues. OpenAI also plans to do more to make it easier for EU citizens to opt out of having their personal data used for the service.

In the U.S., the FTC began an investigation of OpenAI in July 2023, concerning a data leak that had exposed personal information of some customers to others, as well as inaccuracies in ChatGPT, requesting detailed descriptions of all complaints about OpenAI products producing "false, misleading, disparaging or harmful" statements about people. Following a series of warnings in earlier months to the AI industry at large that consumer protection laws will be upheld, the investigation concerns whether OpenAI engages in unfair or deceptive practices that harm the reputations of consumers.

Regulations and lobbying

In late April, EU lawmakers reached a preliminary agreement on a newer version of the AI Act legislation first proposed in April 2021, which would more generally regulate AI technology throughout the EU. OpenAI, the company behind ChatGPT, played a key part in weakening the legislation through lobbying.

OpenAI and its CEO Sam Altman have played a double role in proposals about AI regulation, much like partner Microsoft and competitor Google. Publicly, Altman along with other corporate leaders promoting AI have claimed their technology is so "dangerous" it needs great regulation, in what is called "AI Doomerism", which has come to be seen as a kind of marketing tactic. Far less visibly, Altman and his company have lobbied the EU to water down its AI Act regulation, and been successful in that effort. This followed on earlier efforts by Microsoft and Google to water it down.

Altman ouster and return

“”You could parachute him [Altman] into an island full of cannibals and come back in five years and he’d be the king.

—Y Combinator’s cofounder Paul Graham and Altman's predecessor at Y Combinator

Altman: Polly want a cracker?

Altman was fired as CEO and removed from OpenAI's board of directors in November 2023, just under a year after ChatGPT was made public – returning after only 4 days, after he and most OpenAI employees threatened to move over to Microsoft. The conditions set for Altman's return included a reforming of the board, replacing most of its members.

The reason Altman was removed, according to the then-remaining board members, was that "he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities." In short, Altman frequently lied to the rest of the board, his bosses. Was Altman giving them reports generated by ChatGPT, or is he really a stochastic parrot? Be that as it may, in May 2024, former board member Helen Toner spoke out about the extent of Altman's lying and manipulative behavior, explaining that he had even kept the launch of ChatGPT itself in late 2022 secret from the board, who first learned about it through Twitter.

An earlier speculation was that a key motivation could be how Altman, ironically, treated ChatGPT and other projects too much as if they were merely stochastic parrots, rather than sci-fi threats to the future of human existence. An OpenAI project recent around the time of the ouster called Q* had, in an apparent breakthrough, become able to handle grade-school math problems well given extensive computing resources, renewing such doomsday concerns. Altman's failure to properly inform board members about recent projects was claimed to be a factor in his short-lived ouster. In hindsight the Q* project seems more like a smaller part of the whole, though it's not clear what was the clincher.

Burn, baby, burn!

See the main article on this topic: Climate change

OpenAI and Google, the current leaders in generative AI, have been heavily investing in access to non-renewable energy sources, including coal-fired power plants. For Google at least, this marks a 180° turnaround from their recent commitments to net-zero carbon.

See also

• Artificial intelligence
• Effective altruism — several members of OpenAI's governing board have had ties to EA.
• Garbage in garbage out
• Large language model
• Machine learning
• Elon Musk and Peter Thiel — early investors in OpenAI, though not apparent that they have had any more recent involvement
• Truthiness
• Turing test

External links

• Official website
• ChatGPT Nederlands
• ChatGPT Francais Official
• The Original Bullshit Generator — now with "an innovative tool that utilizes cross-platform viral AI and machine learning to revolutionize your bullshit generating experience…" (since 2022)