Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness

Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model. We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured…