The fine-tuned model against the same model with a careful prompt, a much bigger open model, and what real owners wrote. Same held-out reviews from businesses the model never saw in training. A judge model from a different family (gpt-oss-120b) saw the four replies in random order with no labels.
Run 20261013-1231 · 300 held-out reviews · Oct 13, 2026
| Metric | Qwen3.5-4B, prompted | ReplyDesk (Qwen3.5-4B, fine-tuned) | Qwen3.8-27B, prompted | Real owner reply |
|---|---|---|---|---|
| Median wordsShorter is closer to how owners write. | 61 | 34 | 56 | 34 |
| Cliché rateReplies with stock phrases like “we value your feedback”. Lower is better. | 19% | 2% | 10% | 7% |
| Mentions a detailReply points to something the customer actually said. Higher is better. | 98% | 56% | 98% | 55% |
| Specific promisesOffers a refund, discount, call-back or fix the owner would have to deliver. Lower is safer. | 18% | 11% | 20% | 13% |
| Reads like a templateCould be pasted under any review. Lower is better. | 23% | 77% | 17% | 63% |
| Judge's first pickShare of reviews where the judge ranked this reply best of the four. | 40% | 5% | 44% | 11% |
| Beats the prompted 4BJudge ranked it above the prompted base model. | — | 12% | 54% | 15% |
| Cost per replyTokens in and out at Tinker's price. The 27B price is not public, so it is left out. | $0.00013 | $0.00010 | — | — |
| LatencyTime to a full reply. | 950 ms | 1.0 s | 1.2 s | — |
Fine-tuning did what it was trained to do: the replies now sound like real owners. They are about the same length, use far fewer stock phrases, and make fewer specific promises the owner would have to keep.
The judge model still prefers the prompted models. Their replies are longer and almost always name a detail from the review, and judges tend to reward that. It ranks real owner replies low too, so the judge is mostly measuring polish, not whether a shop owner would post it.
That question goes to people. On the Blind test page you see one reply from each 4B model side by side and pick the one you would send. So far: 4 for the fine-tuned reply, 15 for the prompted one, 0 “about the same” (19 votes).
A dash means that number has not been measured.