
Agentic finetuning using a custom harness
The goal was to figure out whether a small model running on my own hardware could do the job of intent classification and structured parameter extraction well enough for one of my use-cases - read a request like âadd 8 spare tent stakes to the camping boxâ and turn it into something the application can act on. I created an agentic fine-tuning and evaluation loop with a custom evaluation harness: an AI agent proposes one change at a time, a custom evaluation harness measures it against a frozen test set, and every run gets appended to a log with its settings, result and next decision. The agent does the all the work of dataset generation, training, evaluation and diagnosis. ...


