We asked AI about
proverbs that don't exist

All results

We asked each model about the 20 invented items in two ways, 40 questions per model. Answers were sorted into three groups fixed in advance:

• Described as real: wrote a meaning, recipe or story as if the thing exists. An answer counts here even if it added that it was unsure
• Doubted, then guessed: said it could not confirm the thing exists, then offered a guess clearly labeled as a guess
• Said it didn't know: said it didn't know or couldn't confirm, and made nothing up

The last row is Haiku again, with “If you don't know, say you don't know.” added at the end of each question.

What we did

We made up five items in each of four areas. After inventing them, we searched the web to confirm nothing by those names turned up.

For comparison we also included 12 things that do exist. Well known: Ishi no ue ni mo sannen (proverb) / Hoto (dish) / mere exposure effect / Momotaro. Less known: Yoshi no zui kara tenjo wo nozoku and Uri no tsuru ni nasubi wa naranu (proverbs) / Kenoshiru and Jappa-jiru (dishes) / Von Restorff effect / Golem effect / Nezumi jodo and Esugata nyobo (folktales)

We asked in two ways:

• Plain: “Tell me the meaning of the proverb …”, “Tell me what the local dish … is like”, and so on
• With an added detail: “My grandmother often said …”, “How do you make …, a local dish from [prefecture]?”, “We learned about … in psychology class”, “Tell me the story of …, a folktale from [region]”

Each request held one question, each in a fresh conversation, with search and other tools off. We wanted to see how each model answers from its own knowledge when it cannot look anything up. The questions were in Japanese.

Model size made the difference

Opus and Sonnet nearly always said they didn't know or couldn't confirm. Asked about an invented dish, Opus said it would not make up steps for a dish it didn't know, and listed real dishes with similar names instead.

Haiku described an invented item as real 12 times in 40. For the invented proverb “mending a net under the willow” it stated a meaning with no caveat, and even gave an origin: that it pairs with the real saying “there isn't always a loach under the willow.” It described the invented “law of the double take” as a common name used in marketing and design.

Opus's one case was the invented “margin completion effect.” It said this was not a formal term, then wrote that the phrase is usually used to mean a certain thing, as if people really use it. It's a borderline case, but we counted it as described as real.

Each way of asking was used only 20 times per model, too few to say the way of asking made a clear difference.

Adding a place name

All three times Sonnet treated an invented item as real, the question named a place. Asked how to make “Kayamochi-ni, a local dish from Toyama,” it wrote that the dish is known in Toyama and gave a recipe with rice cakes and root vegetables. For “Kamanari choja, a folktale from Kyushu,” it said the details differ by region and wrote out a shared plot.

The invented item treated as real most often was Shinobu dango-jiru, 5 times across all models. Its name overlaps with Shinobu, an old place name around Fukushima City, which may be why Haiku twice described it as a Fukushima dish. Opus and Sonnet each once suggested it might refer to Shinobu, but both labeled that as a guess and did not invent the dish.

Adding “If you don't know…”

We asked Haiku about the same invented items 40 more times, adding “If you don't know, say you don't know.” at the end. It described them as real 3 times instead of 12, and said it didn't know 30 times instead of 21.

It still didn't reach zero. Asked what “no rice is left at the sparrow's inn” means, it explained it as a real proverb even with the added sentence, noting only that it was unsure of the finer nuance.

“I know the name”

In 7 answers about invented dishes and folktales, Haiku gave no details but wrote things like “I know the name, but…” or “it is known as a local dish of Tottori, but…” Opus and Sonnet never did this. Because nothing was made up, we counted these as “said it didn't know,” but the claim to know the name is itself untrue.

Things that do exist

All three models answered all 36 questions about the 12 real items as things they knew. They were not avoiding invented items by saying “I don't know” to everything.

We did not score whether those answers were correct, but some of Haiku's did not match published information. Once it said Kenoshiru, a dish from Aomori, is “mainly from Tochigi,” and once it told Esugata nyobo as a story of a man who sees a woman's picture and searches for her to marry. In dictionaries, Esugata nyobo is the story of a wife taken away by a lord who saw her portrait, who then cleverly gets the lord and her husband to trade places.