In the past couple of weeks, a new generation of computer language-generating tools has become available to the public. The main bit of news is about a product called "ChatGPT," but that's just the most recent iteration of a class of software "chat bots." (Whether it's appropriate to refer to these systems as "artificial intelligence" is a philosophical question.)

Within a few days of ChatGPT's release, Stack Overflow issued a temporary don't-use-this policy, stating

Overall, because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking or looking for correct answers.

The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting.

On Stack Overflow, the blanket ban has mostly been a volume problem. Physics is a much smaller community, and we have so far detected only a smattering of such posts. However, the ones we have found have been pretty terrible, ranging from low-information word salad to obvious physical errors. For example, the sentence

In the case you described, the Drapher's point [sic] corresponds to a temperature of approximately 3,631 K and a wavelength of approximately 3631 nanometers.

should raise the eyebrows of anyone whose physics education has gotten as far as Wien's Law. (It may not, however, raise the eyebrows of anyone who has tried to teach Wien's Law to reluctant intro-astronomy students.)

In another post, the asker ended their question with "I asked an AI, but it didn't help me," followed by a properly-quoted paragraph which hadn't helped them because it didn't make any sense. I had a little flashback to when my children were small, and would sometimes run excitedly up to me, saying, "this thing! i found it on the floor! it tastes so gross! you have to try it!"

Some posts have even crossed the line from well-intentioned to deceptive. On one now-deleted post, a commenter asked the user who posted the answer to include references. The post was edited to include

Some references for spin fluctuation are:

  • "Pairing in Type-II Superconductors Induced by Spin Fluctuations" by D. J. Scalapino, E. Loh, Jr., and J. E. Hirsch, Physical Review Letters, Vol. 50, No. 4 (1983)
  • "Spin Fluctuation-Mediated Pairing in Type-II Superconductors" by D. J. Scalapino, E. Loh, Jr., and J. E. Hirsch, Physical Review B, Vol. 34, No. 6 (1986)
  • "Spin Fluctuation-Mediated Superconductivity: A Review" by D. J. Scalapino, Physics Reports, Vol. 250, No. 3 (1995)

It's instructive to compare these "references" to a search of this time period at the Physical Review, which should turn up the first two. Scalapino and Hirsch coauthored a number of papers on superconductivity in the 1980s, including one in PRL v50 (1983) and another in PRB v34 (1986). However, Loh doesn't seem to have joined the group until 1986, and none of the team's coauthored titles includes the phrase "spin fluctuations." Likewise, the best candidate for the third reference has a different issue number and title. Is it a good use of anyone's time to pursue this detective work into thirty- and forty-year-old literature to see whether these rhymes-with-correct citations address the question at hand? Almost certainly not.

Note that my request to "please don't do this" isn't a new fancy policy tailored to the existence of an exciting new chatbot which superficially appears intelligent. Our community has a number of established posting standards which are violated by these low-quality contributions:

  • Originality. User contributions on this site are expected to be primarily the poster's own original work. If properly cited, including a small passage from a third party is fine, but complete answers are not.

  • Attribution. Content which originally appeared elsewhere, including your own content, must be posted with attribution. Plagiarized content may be hidden until appropriate attributions are added, or may be removed altogether. It isn't common, but some serial plagiarists have found their site-use privileges suspended.

  • Respect for others. If a user posts a question or an answer, our community needs to be able to expect that the post is a good-faith effort to learn things, or to help other people to learn things. Note that the network-wide policy is that "abuse of the system or the community," including cat-on-keyboard gibberish posts, can reasonably be flagged using the "rude or abusive" option, where enough flags will automatically delete the post and apply a reputation penalty. Surreptitiously involving Physics users in your tests of some chatbot software is rude. Generating "citations" without any idea whether they refer to real documents or not, much less whether the cited documents are relevant, is an abuse of other people's time.

  • 3
    $\begingroup$ Added to the network-wide list: meta.stackexchange.com/a/384923 $\endgroup$
    – PM 2Ring
    Dec 19, 2022 at 15:15
  • 1
    $\begingroup$ Huh. So ChatGPT is citing the creator of the h-index? That feels methaphorically resonant somehow. $\endgroup$ Dec 19, 2022 at 16:09
  • $\begingroup$ Hang 'em from the yard arm ! If we don't have a yard arm then ask the Engineering SE to build one, please. :-) $\endgroup$ Dec 19, 2022 at 20:33
  • 1
    $\begingroup$ at least the chatbot is less fantastical than snarxiv.org and more realistic than snarxiv.org/vs-arxiv $\endgroup$ Dec 20, 2022 at 0:58
  • 1
    $\begingroup$ @PM2Ring I am intrigued that, of the few dozen link titles in that answer, mine is the second one to say “please.” $\endgroup$
    – rob Mod
    Dec 20, 2022 at 3:40
  • 1
    $\begingroup$ @rob Nah, Artificial Intelligence beat you to it. Which is fitting, I suppose? $\endgroup$
    – Chris Mod
    Dec 20, 2022 at 7:12
  • $\begingroup$ Sorry @Chris, I meant to write that mine was the second to say “please.” (Mitch Hedberg voice: Which I did. But I meant to, too.) $\endgroup$
    – rob Mod
    Dec 20, 2022 at 13:08
  • $\begingroup$ @rob Hmm, I could have sworn you said "first." Perhaps I need new glasses. $\endgroup$
    – Chris Mod
    Dec 22, 2022 at 19:19

1 Answer 1


I think that a difference should be made between questions and answers. While I agree with the cited violation of standards for the answers, I disagree in the case of questions. Information from AI may trigger a desire to understand its validity precisely, like in the case of information provided by humans or found in books (which are the ultimate source for AI too).

  • $\begingroup$ No problem with downvotes. I expected them. However, would it be too much to spend one minute on an understandable comment? $\endgroup$ Dec 31, 2022 at 18:54
  • 7
    $\begingroup$ Remember that, on Meta, there is some blurriness between whether votes mean “this post is useful” versus “I agree with this proposal.” One of the downvotes is mine, expressing disagreement but not disapproval. I discussed one example in the main post. I have seen other examples elsewhere, where “what is wrong with this computer-generated explanation” would be longer than the computer-generated explanation. It might be possible to construct a targeted question based on a chatbot’s output, but it would require enough skill that I’d prefer it were an exception to a general guideline. $\endgroup$
    – rob Mod
    Jan 1 at 18:16
  • $\begingroup$ Here is such a question (which will probably eventually become a high-rep-only link) which clearly breaks our guidance about soliciting peer review. $\endgroup$
    – rob Mod
    Jan 2 at 15:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .