The goal was to figure out whether a small model running on my own hardware could do the job of intent classification and structured parameter extraction well enough for one of my use-cases – read a request like “add 8 spare tent stakes to the camping box” and turn it into something the application can act on.
I created an agentic fine-tuning and evaluation loop with a custom evaluation harness: an AI agent proposes one change at a time, a custom evaluation harness measures it against a frozen test set, and every run gets appended to a log with its settings, result and next decision. The agent does the all the work of dataset generation, training, evaluation and diagnosis.
Setup
I built a fixed test set of 45 requests covering nine intents, five cases each, and froze it. Every model, change and training run was measured against those same 45 cases. The output metrics of each run were: intent accuracy, meaning did it pick the right action, and parameter accuracy, meaning did it extract the right details. A third important number is: fallbacks, the count of cases where the model declined to answer and returned “unclear.”
The OpenAI baseline (gpt-5.6-luna) was created and frozen. It scored 93.33% intent and 71.11% parameter accuracy with no fallbacks.
All local training ran on a consumer RTX 3060 with 12 GB of memory, under WSL2 with CUDA. LoRA fine-tuning, rank 16, alpha 32, learning rate 1e-4, batch size 4. A 15-epoch run takes about 20 minutes.
The entire source, including training data, eval harness, scripts and agent skill is on GitHub – needle-tuner.
Part 1 – needle2
The first local comparison with needle2 model from Cactus. It scored 4.44% intent accuracy. The problem turned out to be how I was asking the question. I had given the model one abstract, generic “classify this” tool. I replaced it with eight concrete tools, one for each real action the application supports.
The result went to 62.22% intent and 68.89% parameter accuracy.
With this reasonable starting point, I fine-tuned on a small purpose-built training set and ran a structured loop: change one thing, measure against the frozen test set, write down the result.
| Needle2 configuration | Intent | Parameter |
|---|---|---|
| Base model, redesigned tools, no training | 62.22% | 68.89% |
| Best fine-tuned run (5 epochs, rank 8) | 64.44% | 62.22% |
| Longer training (30 epochs) | 57.78% | 64.44% |
| Heavier defaults | 42.22% | 51.11% |
The best training result gained two points on intent accuracy over no training at all, and lost six on parameters. More training made things worse. The heaviest run also refused to answer far more often, with 15 fallbacks against 9 for the untrained model. The results were noisy enough to make any single comparison suspect. Two runs of the identical best configuration measured 64.44% and 60.00%, so I stopped treating one good run as evidence of anything. Validation loss kept improving while held-out accuracy got worse. Had I watched the training dashboard instead of the frozen test set, I would have shipped a regression. I stopped at this point.
Part 2 – needle3
A new version of the model family had shipped in the mean time, with a different architecture and a different native engine. Untrained, needle3 measured 71.11% intent and 66.67% parameter accuracy with 7 fallbacks. I modified the evaluation harness loop to support the new model and re-ran it on the new model.
A one-epoch smoke test scored 68.89% and 73.33%, which is one run on one epoch and proves nothing on its own. A full 30-epoch run at the wrapper defaults scored 71.11% and 66.67%, identical to the untrained base. However, the per-intent breakdown had shifted underneath, some intents up and some down, while the totals stayed flat. Validation loss bottomed out at epoch 13 and climbed for the remaining 17 epochs while training loss kept falling. Same overfitting pattern needle2 had shown.
The issue
Instead of tuning another hyperparameter, I stopped reading aggregate scores and started reading individual failures.
The training data and the held-out test set agreed on which intent was correct. They disagreed on the convention for writing the parameters down.
The first problem was descriptive adjectives. For delete requests, the test set strips condition words from item names, so “the broken toaster” is expected to yield ‘toaster’. All 12 authored delete rows in my training data did the opposite, keeping ‘cracked mixing bowl’ and ‘expired pain reliever’ intact.
The second problem was a wrong key name. For inventory-viewing requests, the training data used a ‘location’ key for the container slot while every other intent, and the test set, used ‘boxLabel’. It also carried a ‘scope’ key that appears in zero held-out cases.
The model was learning my convention faithfully and being marked wrong for it, on 100% of those cases. That also explains why longer training hurt: more epochs meant learning the wrong answer more thoroughly.
The Fix
I corrected 12 delete rows and 20 view-inventory rows, regenerated the derived datasets, re-pinned the integrity hashes and re-ran the validation suite. No hyperparameter change. Epochs dropped from 30 to 15, taken from the previous run’s validation-loss minimum.
Result: 82.22% intent and 75.56% parameter accuracy, 6 fallbacks.
| Run | Intent | Parameter | Fallbacks |
|---|---|---|---|
| Needle2 base | 62.22% | 68.89% | 9 |
| Needle2 best fine-tune | 64.44% | 62.22% | not recorded |
| Needle3 base | 71.11% | 66.67% | 7 |
| Needle3 fine-tuned, original data, 30 epochs | 71.11% | 66.67% | 6 |
| Needle3 fine-tuned, corrected data, 15 epochs | 82.22% | 75.56% | 6 |
| OpenAI (frozen baseline) | 93.33% | 71.11% | 0 |
That is 11 points of intent accuracy over its own base model and 20 over needle2‘s base. It is also the first time a local model beat the hosted API on parameter accuracy, 75.56% against 71.11%. Intent accuracy is still 11 points behind.
The per-intent movement matches the diagnosis. Delete went from 80%/40% to 100%/80%. Search went from 60%/60% to 100%/100%. Update improved as well, since the bare-noun convention is shared by every intent that names an item.
Validation loss behaved differently too. With the corrected data it declined every one of the 15 epochs, with none of the mid-run turnaround the unfixed run showed. The contradictory labels had been generating the overfitting signal themselves.
Next
Parameter extraction already beats the API. Intent classification is 11 points behind, and most of the remaining gap sits in one diagnosed failure mode: deciding whether a vague request is actionable.
The next lever is contrastive training examples sitting right on that boundary. After that the options get expensive: changes to how the model produces its answers, training that explicitly teaches it what a wrong answer looks like, or a different architecture.
Because the same 45 cases were used to decide which experiment to run next, they are now a model-selection set rather than an unbiased final exam. Before anyone calls this production-ready it has to be tested against a separate set of cases it has never influenced.
The project also produced a reproducible pipeline along the way. A frozen test set, hash-pinned datasets, a documented training process, a log of every attempt including the failures, and a default path that validates without training and without calling paid APIs. That infrastructure is the reason the annotation issue was discoverable at all.
