I find after reading a selection of LLM generated poetry/short fiction, you start notice signs of a generated text. It tends to be a bit too polished, without any idiosyncrasies and with almost too much consistency in the delivery.
EDIT: So I read the short story. To be honest, I am don’t think I would be able to tell whether this was LLM generated or written by a writer. There are some subtle signs, but it’s very much possible that I am seeing these signs because I knew it was LLM generated.
One thing to point out is that this is not really a short story, it’s less than a thousand words with honestly not much going on and there is lots of colourful descriptive text. I was expecting something in the range of 3K to 5K words.
After reading the “short story”, I am not sure I agree with the methodology. Some of their test statements include “I was interested in the struggles of these characters” and “This story deserved to be published in a top literary journal”. The story is not long enough enough to make meaningful conclusions about such test statements.
Their approach to willingness to pay also doesn’t make sense as it’s too short. Here is their graph for the TWP metic:
This seems artificial, no one is going to pay 30 cents in a real world scenario for such a text (irrespective of whether you think it was written by a person or if it was LLM generated). The respondents might rate it at being worth 30 or 40 cents as part of the survey, but that’s not the same thing as actually going through with a purchase. I will note they didn’t simply ask for a value and they did have a system tied to the survey payout; but this almost seems irrelevant.
“Polished” is a generous word for it. I think you might be mistaking generic plastic gloss for polish, though. AI writing is so formulaic, it’s like looking at a repeating texture in a video game. It writes lyrics like a five year old, too - very simple, entirely conventional, no creativity. Nothing new.
Would be curious to read the LLM output.
I find after reading a selection of LLM generated poetry/short fiction, you start notice signs of a generated text. It tends to be a bit too polished, without any idiosyncrasies and with almost too much consistency in the delivery.
EDIT: So I read the short story. To be honest, I am don’t think I would be able to tell whether this was LLM generated or written by a writer. There are some subtle signs, but it’s very much possible that I am seeing these signs because I knew it was LLM generated.
One thing to point out is that this is not really a short story, it’s less than a thousand words with honestly not much going on and there is lots of colourful descriptive text. I was expecting something in the range of 3K to 5K words.
After reading the “short story”, I am not sure I agree with the methodology. Some of their test statements include “I was interested in the struggles of these characters” and “This story deserved to be published in a top literary journal”. The story is not long enough enough to make meaningful conclusions about such test statements.
Their approach to willingness to pay also doesn’t make sense as it’s too short. Here is their graph for the TWP metic:
This seems artificial, no one is going to pay 30 cents in a real world scenario for such a text (irrespective of whether you think it was written by a person or if it was LLM generated). The respondents might rate it at being worth 30 or 40 cents as part of the survey, but that’s not the same thing as actually going through with a purchase. I will note they didn’t simply ask for a value and they did have a system tied to the survey payout; but this almost seems irrelevant.
“Polished” is a generous word for it. I think you might be mistaking generic plastic gloss for polish, though. AI writing is so formulaic, it’s like looking at a repeating texture in a video game. It writes lyrics like a five year old, too - very simple, entirely conventional, no creativity. Nothing new.
I find that LLMs also tend to create very placative, kitschy content. Nuance is beyond them.
By Bender
I stand corrected, this speaks to my robot soul
A polished turd is still polished!
It looks like it’s available in the linked study’s paper (near the end)
Cheers!