Google's Agile Deliberation builds classifiers for fuzzy categories from edge cases first
Victoria Shinder2 min readGoogle Research has published Agile Deliberation, a framework for building image classifiers for subjective categories, where the boundary of the concept is not fixed in advance. The post, dated 4 November and written by Otilia Stretcu and Leijie Wang, starts from a plain observation: labelling assumes the labeller already has a clear and stable definition, which holds for a stop sign but not for healthy food, inspiring artwork or content moderation.
The framework splits the work into two stages. In concept scoping, a broad target is broken into in-scope definitions, the things the classifier should flag, and out-of-scope ones, the false alarms the user wants excluded. In concept iteration, the system retrieves borderline images to expose edge cases, the user gives feedback on them, and an LLM-driven loop refines the prompt that becomes the classifier.
The team evaluated it in 18 live sessions of 1.5 hours each with human participants rather than on a static benchmark, arguing that subjective concepts cannot be captured by fixed test sets. It compared the approach with a zero-shot classifier, an automated agent that decomposes the concept on its own, and manual deliberation in spreadsheets. According to the post, participants using Agile Deliberation labelled more consistently over time and built classifiers that outperformed all three baselines, with the largest gains on the hardest borderline images; manual deliberation often failed to improve accuracy or made it worse.
The study also measured the cost to the people involved. Participants reported significantly lower effort, stress and mental demand than with manual prompt engineering, and every participant preferred the new workflow. Code is open-sourced on GitHub, with a notebook walking through the healthy-food example.
What it means
The useful idea is that the ambiguity sits with the person, not the model. Showing a user the cases that split their own definition is cheaper than asking them to write a perfect prompt up front.