I built a GenAI pipeline that pulls six fields out of every customer review, work a retailer would otherwise need a team to read by hand. Six prompt designs were scored against the same fifty reviews before I recommended one.
Customer reviews are the cheapest source of product feedback a retailer has, and the least used. Read at scale, they show which products are failing, what customers like, and which complaints need someone to act today. The brief asked for a system that does that reading automatically: work out the sentiment of each review, identify the product it is about, and flag the high-priority issues.
The retailer wanted six things from each review: the garment type, the sentiment (Positive, Negative or Neutral), how urgently to act (High, Medium or Low), a one-line summary, a reply to the customer, and one suggested action for the product team.
22,640 reviews, after removing duplicates and reviews with no text. Each square is one per cent of them.
755 hours to read by hand at two minutes each: about twenty working weeks, or roughly $24,000 at an assumed $32 an hour. Nobody does it today.
77.1% are four or five stars. A filter that only looks at low ratings skips all of them, including one four-star review, with "would recommend" ticked, that reports holes along the seams.
The review corpus
Two summaries can both be faithful to the same review and share no wording, so the evaluation harness came before any of the prompts.
One fixed sample. Fifty reviews, ten per star rating, so urgency was tested across its whole range. Same reviews, same order, every run.
Six prompt versions on gpt-4o-mini: zero-shot (instructions only), few-shot (plus worked examples) and chain-of-thought (the model reasons before answering), each written plainly (V1) and then improved (V2). The prompt is the only thing that changes.
Reasoning removed. Chain-of-thought prompts return an extra reasoning field. It is removed before scoring, so the judge sees the same six fields from every prompt.
A different model judges. gpt-4o, a larger model, scores each field 0 to 1 against written criteria and names the weakest. Its instructions were fixed before the first run and never edited.
The judge was checked first. When two prompts gave identical answers for a review, the judge's scores still differed by 0.023 on average. Any gap between prompts smaller than that could be the judge, not the prompt.
The per-review evaluation loop
Six versions, fifty reviews, one judge whose instructions never changed. Output tokens, the unit the API bills by, were counted from every response.
The plain zero-shot prompt scored 0.850 out of 1, averaged across the six fields, using 5,223 output tokens. Everything else is compared with it.
Chain-of-thought spent twice the tokens. V1 scored lowest of the six at 0.831. V2 recovered to 0.887 and led on urgency at 0.970.
Few-shot V2 scored 0.906 on 5,825 tokens, 57% of what either chain-of-thought version used, and led on the garment field at 0.980.
Plain few-shot matched the best chain-of-thought. The plain few-shot prompt scored 0.891 and the improved chain-of-thought prompt 0.887. That 0.004 gap is inside the judge's own margin, so they are level, and few-shot used about half the tokens: 5,501 against 10,235.
Judge score against output tokens
One of the six fields is the garment a review is about: a dress, a top, a skirt. Some reviews never say, for example "love it, great quality, runs small". For those the model should answer "unspecified" rather than guess, because a guessed garment files the feedback against the wrong product. Seven of the fifty reviews name no garment.
Three versions carry that instruction, worded almost identically. They differ only in how it is delivered.
Written as a plain instruction, the model answered "unspecified" for 21 of 50 reviews, three times too often. At least 14 of those reviews did name a garment, so their feedback lost its link to a product. The garment field scored 0.820, the lowest of the six.
With reasoning first, in the chain-of-thought prompt, it answered "unspecified" 13 times: 0.940.
Shown once in a worked example, it answered "unspecified" 7 times, the same as the number of reviews that name no garment, and the field scored 0.980.
With no instruction at all, few-shot V1 never answers "unspecified" and scores 0.870: worse than showing the instruction, better than stating it.
"Unspecified" answers and garment-field score
Why it matters: the same instruction scored anywhere from 0.820 to 0.980 depending only on how it was delivered. That spread is twice the gap between the best and worst prompt overall (0.906 against 0.831). When a model has to hold back on unclear cases, one worked example of holding back did more than any extra wording.
Sentiment never moved. All six prompts scored between 0.810 and 0.830, a spread smaller than the judge's own 0.023 inconsistency. The field had to be Positive, Negative or Neutral, and a review that praises the fit and complains about the colour is none of those cleanly.
Predicting "would recommend". In a separate test, the model read each review and guessed whether the customer ticked "would recommend". It was right 74% of the time; always guessing the more common answer is right 54% of the time.
Its mistakes ran one way. Eleven of thirteen were customers who complained and still ticked "would recommend". On three-star reviews it was right only 30% of the time.
Sentiment score by design
The pipeline gives each review one sentiment label, tied to the product it is about. The fix for mixed reviews is not a better prompt: it is scoring sentiment per aspect (fit, fabric, colour, size) instead of once per review, which changes what the model is asked to return.
High-urgency reviews are the priority issues: product faults someone should read the same day. The recommended design uses few-shot V2 and processes new reviews in scheduled runs, so real-time here means same-day: an urgent fault reaches a person the day it is posted. Nothing is sent to a customer automatically: the model drafts the reply and a person decides.
The same pipeline applies to any free-text feedback: support tickets, survey comments, social media posts.