MIT Applied AI & Data Science · Generative AI capstone · 2026
A GenAI customer feedback system.
A pipeline that reads thousands of customer reviews and pulls out what a retailer would otherwise pay a team to read by hand. The question was which prompt to trust, answered by measurement instead of opinion.
0.906
Top overall score, few-shot V2
$5.36
To label all 22,640 reviews
57%
Of chain-of-thought's token spend
74%
Recommend classifier, against a 54% baseline
The problem
Each customer review needed six things pulled out: the garment, sentiment, urgency, a summary, a reply and a merchandising action. Prompt choice usually comes down to whichever version reads best in a demo. I wanted a number for each one.
The method
Six prompt designs ran against one identical, stratified sample of fifty reviews, ten per star rating. A second, larger model scored each of the six fields from 0 to 1 against written criteria.
Before trusting that judge, I measured its own variation on identical answers: 0.023 on average. Every difference between prompts could then be checked against the judge's noise floor.
Results
| Prompt version | Overall score | Output tokens |
|---|---|---|
| Zero-shot V1 (baseline) | 0.850 | 5,223 |
| Zero-shot V2 | 0.867 | 5,276 |
| Few-shot V1 | 0.891 | 5,501 |
| Few-shot V2 (recommended) | 0.906 | 5,825 |
| Chain-of-thought V1 | 0.831 | 10,206 |
| Chain-of-thought V2 | 0.887 | 10,235 |
What it found
- Few-shot V2 scored highest overall and led on the garment field at 0.980. The same rule about unnamed garments scored 0.820 when stated as an instruction; shown once in a worked example, it scored 0.980.
- Chain-of-thought doubled the token bill. Its first version scored lowest of the six, at 0.831; the revised version led on spotting customer urgency.
- The recommend classifier reached 74% against a 54% baseline. Eleven of its thirteen errors were customers who complained and recommended the product anyway.
My role
Solo project: evaluation design, prompt engineering, harness and pipeline build, judge rubric design, cost modelling, analysis and write-up.
The retailer in the brief is fictional; the data is a public corpus of 23,486 clothing reviews. Scores come from a model judge, not from people labelling the same reviews.