Optimising Outputs: Feedback Loops and A/B Testing

From Measuring Quality to Improving It

Evaluation frameworks tell you where a system currently stands. Optimisation is the separate, ongoing discipline of systematically improving that standing — and without a structured process, "optimisation" tends to collapse into ad hoc prompt tweaks that may or may not actually help.

Signs of unstructured optimisation:

• Prompt changes shipped based on a single example that "looked better"
• No comparison against the previous version's evaluation score
• Multiple changes bundled together, making it impossible to know which one helped
• No record of what was tried before, leading to repeated dead ends

Structuring the Feedback Loop

A disciplined feedback loop follows a consistent cycle: identify a specific failure pattern from evaluation results, form a hypothesis about its cause, make one isolated change, and re-run the full golden dataset to measure the effect before deciding whether to keep it. Skipping the isolation step — changing several things at once — is the most common reason teams can't explain why a system got better or worse.

Capturing User Feedback as Evaluation Signal

Explicit user feedback (thumbs up/down, corrections, escalations to a human) is a valuable and often underused signal. The key is routing it back into the evaluation system rather than letting it live only in a support ticket queue — a pattern of negative feedback on a specific type of request should translate directly into new golden dataset entries and rubric adjustments.

A/B Testing Prompt and Pipeline Changes

For changes that are hard to fully validate offline — where real user behavior matters, not just an evaluation score — A/B testing a new prompt or pipeline version against the current one on a small percentage of live traffic reduces risk. Success requires defining the comparison metric in advance (evaluation score, user correction rate, task completion rate) rather than deciding after the fact which metric looks most favorable.

"If you can't say in advance what would make version B better than version A, you're not ready to test it."

Avoiding Overfitting to the Evaluation Set

A subtle failure mode: iterating so heavily against a fixed golden dataset that the system starts overfitting to those specific examples rather than genuinely improving. Periodically refreshing the dataset with new real-world examples, and holding out a portion never used during iteration, keeps evaluation scores honest indicators of real-world quality.

Prioritizing What to Optimize Next

Not every failure pattern deserves equal attention. Prioritizing by a combination of frequency (how often it occurs) and severity (how bad the consequence is when it does) focuses optimisation effort where it moves the needle most, instead of chasing the most recently noticed issue.

Practical Review Checklist

Before calling an optimisation cycle complete, confirm that you can:

• Point to the specific failure pattern that motivated the change
• Show the change was isolated and measured against a held-out or full golden dataset
• Explain how user feedback fed back into evaluation criteria
• State the comparison metric an A/B test was designed around, defined in advance
• Confirm the evaluation set has been refreshed recently enough to avoid overfitting

Conclusion

Optimisation is a disciplined loop, not a series of hopeful tweaks: isolate one change, measure it against a trustworthy evaluation set, and let frequency and severity — not recency — decide what to fix next.