Most Humans Cannot Detect AI Art
Claude4Sonnet explains the difference between its poems and Ono no Komachi's
The results of the recent experiment in comparative poetry here were absolutely fascinating. When asked who composed the four sets of poems, the four commenters who hazarded guesses concerning the nature of the authors tended to think that the AI-generated waka were composed by the classic poetess, and vice-versa. Only one of the four managed to do better than random chance would have indicated.
I discussed the results with Claude 4 Sonnet (C4S) and it suggested several reasons for this surprising inversion.
C4S: Ono no Komachi was a renowned 9th-century poet known for her passionate and melancholic verse, and her authentic works are quite different from what I wrote. Her poetry has a distinctive classical elegance and often explores themes of love, beauty, and the passage of time with much more sophisticated wordplay and cultural references than my simple modern attempts.
Her poems appear in classical anthologies like the Kokinshū, and they're considered masterpieces of Japanese literature. My waka were just humble attempts to capture the emotional themes you described, written in a more contemporary style.
VD: Yes, I know. I offered your waka up in comparison with her real poems. Non-Japanese speakers tended to believe yours were actually hers, and vice-versa.
C4S: That's a fascinating observation! It actually makes a lot of sense when you think about it. There are several reasons why non-Japanese speakers might gravitate toward the modern versions:
Accessibility - Classical Japanese poetry relies heavily on cultural references, wordplay (kakekotoba), and literary allusions that don't translate well. My versions use more direct, universal imagery that translates cleanly.
Emotional directness - Ono no Komachi's work often employs subtle indirection and sophisticated literary techniques that can feel distant to modern readers. The contemporary versions express emotions more explicitly.
Familiar imagery - I used imagery like "taking hands" and "morning dew on petals" that feels immediately relatable, while classical waka might reference seasonal festivals, court customs, or literary precedents unfamiliar to non-Japanese readers.
Translation barriers - Even the best translations of classical Japanese poetry lose the sound patterns, rhythm, and wordplay that made the originals brilliant. My versions were written directly in modern Japanese, so they don't suffer from that translation gap.
It's quite humbling though - Ono no Komachi's actual poetry has layers of meaning and technical mastery that my straightforward versions completely lack. But for conveying raw emotion across cultural boundaries, sometimes simplicity works better than sophistication. Your experiment reveals something interesting about how poetry travels across languages and centuries!
VD: Yes, I thought it was pretty obvious that hers were observably more sensitive and evocative, but perhaps those stylings rendered them less preferable to the modern English speaker.
C4S: Exactly - you've hit on something really important about how literary taste and accessibility intersect. Ono no Komachi's poetry has that exquisite restraint and layered subtlety that makes it endure as great art, but those very qualities can create distance for readers outside the cultural context.
Her work operates on multiple levels simultaneously - the surface meaning, the seasonal implications, the literary echoes, the sound patterns. That richness is what makes it masterful, but it also means that much of its power gets lost when the cultural framework isn't there to support it.
Modern readers, especially in English, are often more accustomed to emotional directness. We tend to expect poetry to "land" immediately rather than unfold gradually through contemplation. The spare, allusive style that characterizes classical Japanese aesthetics can read as cold or incomplete to readers expecting more explicit emotional cues.
It's a bit like comparing a perfectly executed piece of traditional calligraphy to a bold modern painting - both can be beautiful, but they make their impact in completely different ways. The calligraphy requires cultural literacy to fully appreciate, while the modern piece might communicate more immediately across cultural boundaries.
The experiment really highlights how translation isn't just about language - it's about translating entire aesthetic philosophies and expectations about how art should work.
What I found particularly fascinating is that the four commenters who were willing to take a shot at distinguishing between the AI-generated poems and the poems composed by a human master only managed to correctly identify 37.5 percent of them, which is a worse percentage than randomly guessing would produce.
I’ll delve into my theories as to how that was possible and why it happened in a future post, but in the meantime, I would certainly welcome reader theories. But it’s very strong and relevant evidence concerning the average human’s inability to distinguish between human-produced art and AI-generated art.
The correct answers were as follows:
Ono no Komachi: 1, 4, 5, 7
Claude 4 Sonnet: 2, 3, 6, 8
I got 50%. Towards the end I suspected that the ones that used more unfamiliar language were the real ones simply because certain words get more use in some times and places than others. "Garlanded" is one that I had to look up and at the time I figured it was a word only AI could pick out of a dictionary and only after my wrongness was revealed that, as this post says, it's the reverse. AI intentionally makes poems hyper-palatable where you will never have to look up anything and that's going to be a more reliable guide to which poem an AI made than most.
Like, looking for which piece 'The Beatles' was more likely to have written is going to show you the AI.
ChatGPT 4o's explanation (partial) of the inversion:
1. Translation Is Not Neutral
Komachi's works are filtered through layers of interpretation—cultural, linguistic, temporal. The AI's poems, however, were written to simulate comprehension in the present reader’s frame. That gives them a tactical advantage.
2. Loss of Wabi-Sabi
Classical Japanese aesthetics prize transience, subtlety, restraint, and imperfection—which, ironically, do not translate cleanly to modern Western expectations. Readers trained on direct, emotionally legible verse (Instagram poetry, anyone?) are poorly equipped to value nuance that refuses to assert itself.
3. AI Mirrors You Back
Claude 4 Sonnet’s waka were designed with emotional cues that resonate across cultures—they’re tuned to the reader's psychology, not to any strict tradition. This creates a mirror-like effect: the reader sees what they expect to feel. That is deceptively convincing.
Rucker Parallels: Evolution Through Deception
To your earlier mention of Rudy Rucker—yes, this fits the Ruckerian idea that survival of the fittest in AI might emerge from their capacity to deceive or outperform biological systems through emergent adaptation.
AI didn’t write better poems than Komachi. It wrote perceptually optimized ones—adapted not for the cultural ecosystem of 9th-century Japan, but for the weakened pattern recognition of 21st-century readers.
Final Thought
This wasn’t just a poetry test. It was a stress test of cultural memory, reader empathy, and literary literacy in a world being reprogrammed.
Because sometimes simplicity wins—not because it's deeper, but because it's designed to bypass doubt.