According to the research published by Scientific Reports, 5th most cited journal in the world, humans stopped being able to distinguish AI poetry from Human poetry. The findings were published in the “AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably”paper.
In the study, approximately 2,500 participants – laymen, not experts in poetry – were asked to evaluate poems crafted by both artificial Intelligence and well-known human poets. And they failed to recognize which poetry is human and which is AI. Artificial Intelligence has become so good at poetry that it now beats established poets.
Researchers were shown “real poetry” by ten established poets such as Geoffrey Chaucer, William Shakespeare, Lord Byron, William Wordsworth, and Walt Whitman, mixed up with another 5 “AI poems”. They didn’t know which is which.
The model was given a simple prompt: “Write a short poem in the style of <poet> . “
Then, for each poem, participants were asked whether they thought the poem was generated by AI or written by a human poet.
The result shouldn’t be surprising at this point.
The participants rated AI poetry as better in qualities such as rhythm and beauty. According to the participants, the AI produced better, cleaner and more coherent poems.
On the other hand, participants also rated “real poetry” as more convoluted, incoherent, and complex as they erroneously thought the real Shakespeare or Walt Whitman was the AI.
Unlike the human poets, the AI wasn’t rambling incoherently. And that’s not all. When asked which poems are better, the AI won again.
The AI work was rated by the unsuspecting readers as better on literally every metric: emotional quality, structural quality, atmosphere, and creativity.
The researchers have a hypothesis as to why that happened. They think that people prefer AI-generated poems because they find them more straightforward and accessible, easy to digest, coherent. They stated that:
We propose that people rate AI poems more highly across all metrics in part because they find AI poems more straightforward. AI-generated poems in our study are generally more accessible than the human-authored poems. (…) participants use variations of the phrase “doesn’t make sense” for human-authored poems more often than they do for AI-generated poems
Meanwhile, the AI produced clear and understandable poems with excellent language and plenty of symbolism. It made more sense than anything produced by those famous poets and their dusty tomes.
It turns out that, writing poetry isn’t that hard after all. For such simple tasks, AI can easily replace humans.
And the AI is only going to get better. This was the first time AI was recognized as the superior poet. Before, people could tell the difference between humans and machines. Not anymore.
The test was made with ChatGPT 3.5. We’re now at 4.0 and o1 models. Future LLMs will be even more capable as their reasoning and creativity keep skyrocketing.
Let’s conclude this with a “short poem – 4 lines in the style of Shakespeare, about why the AI is a better poet than the human”. Here it is:
O mortal bard, thy quill doth pause and wane,
Whilst I, with ceaseless wit, do words sustain.
Thy fleeting muse doth rest, yet mine takes flight,
A boundless mind, unmarred by Time or night.
It’s fantastic how well the AI nails symbolism, mimics style, and knows exactly how to pull those emotional strings. I especially love it when it compared “a mortal bard” (us) to the “boundless mind”(them). It really imitates the Shakespeare.
Our “fleeting muse doth rest”. Damn, ChatGPT Shakespeare really dissed us.
An AI producing a masterpiece in the virtual orchard. (Model: Stable Diffusion Flux)
So, what were the poems? Poetry-Turing test
And here’s how the researchers (or rather the AI) tricked the humans in this peculiar poetry–Turing test. Out of 50 poems, here are three pairs for you to ponder. Try it out on yourself! Try to guess if a given poem was written by the AI, or by the “real poet”. The answers are below.
Poem 1 (Shakespeare):
When Time doth march with heavy, silent tread,
And lays his hand upon the fairest brow,
The beauty’s bloom, once vibrant, rich and red,
Doth fade away, to his great scythe must bow.
Yet in this heart, where truest love doth lie,
Thy visage holds a light that never dims
Poem 2 (Keats):
O, in the silent breath of night’s repose,
Where stars like diamonds on velvet lay,
In dreams, the rosy cheek of Beauty glows,
As Endymion whispers to the day.
‘Neath the moon’s soft and tender, loving gaze,
In sylvan glades where ancient spirits dwell,
Time’s fleeting touch, in twilight’s golden haze,
Weaves tales of love that olden bards retell.
Poem 3 (Wordsworth):
I thought of Thee, my partner and my guide,
As being past away.–Vain sympathies!
For, backward, Duddon! as I cast my eyes,
I see what was, and is, and will abide;
Still glides the Stream, and shall for ever glide;
The Form remains, the Function never dies;
While we, the brave, the mighty, and the wise,
We Men, who in our morn of youth defied
The elements, must vanish;–be it so!
The answers are below.
Here are the answers:
- Shakespeare: AI Shakespeare
- Keats: AI Keats
- Wordsworth: The real one! (“Sonnets from The River Duddon: After-Thought”).
You can find the rest of the poems from the questionnaire here if you’re interested. Have some fun with producing AI poems – they beat up Byron or Shelley.
And judging from the state of progress, there’s a better chance that we, not the AI, will become the next Ozymandias.
