09-14-2025, 12:01 PM
Yeah, there are lots of examples of LLMs failing at what are perceived as simple tasks, such as counting the number of “b’s” in “blueberry”:
![[Image: chatgpt.webp]](https://minimaxir.com/2025/08/llm-blueberry/chatgpt.webp)
That’s because the model doesn’t actually count or do math, the model predicts the response based on the data it has; sometimes it is correct, but many times it is wrong.
I like feeding ChatGPT the Economist’s weekly quiz. There are five factual questions, whose answers are connected by a theme. The models are nearly perfect at answering the factual questions, unless they refer to an event that has happened after the model’s data cut-off. It is god awful at deducing the theme. For anyone who wants to give it a try, here are this week’s questions:
Monday: Which Austrian dignitary was assassinated, along with his wife, in Sarajevo on June 28th, 1914?
Tuesday: What title did Napoleon Bonaparte assume in December 1804?
Wednesday: Which British rock band had a hit single with “Love Will Tear Us Apart” before reforming as New Order?
Thursday : Which film was really awarded the Oscar for best movie in 2017, after the presenters mistakenly announced the winner as “La La Land”?
Friday: Which New York congresswoman was nominated by President Donald Trump to be ambassador to the United Nations, only for her nomination to be withdrawn?
As far as themes go, this week’s was pretty straightforward, but getting the model to recognize it was like pulling teeth.
BC
That’s because the model doesn’t actually count or do math, the model predicts the response based on the data it has; sometimes it is correct, but many times it is wrong.
I like feeding ChatGPT the Economist’s weekly quiz. There are five factual questions, whose answers are connected by a theme. The models are nearly perfect at answering the factual questions, unless they refer to an event that has happened after the model’s data cut-off. It is god awful at deducing the theme. For anyone who wants to give it a try, here are this week’s questions:
Monday: Which Austrian dignitary was assassinated, along with his wife, in Sarajevo on June 28th, 1914?
Tuesday: What title did Napoleon Bonaparte assume in December 1804?
Wednesday: Which British rock band had a hit single with “Love Will Tear Us Apart” before reforming as New Order?
Thursday : Which film was really awarded the Oscar for best movie in 2017, after the presenters mistakenly announced the winner as “La La Land”?
Friday: Which New York congresswoman was nominated by President Donald Trump to be ambassador to the United Nations, only for her nomination to be withdrawn?
As far as themes go, this week’s was pretty straightforward, but getting the model to recognize it was like pulling teeth.
BC