LLM on Ash Tewari

Agentic finetuning using a custom harness

Fri, 25 Sep 2026 02:56:11 +0000

The goal was to figure out whether a small model running on my own hardware could do the job of intent classification and structured parameter extraction well enough for one of my use-cases - read a request like “add 8 spare tent stakes to the camping box” and turn it into something the application can act on.

I created an agentic fine-tuning and evaluation loop with a custom evaluation harness: an AI agent proposes one change at a time, a custom evaluation harness measures it against a frozen test set, and every run gets appended to a log with its settings, result and next decision. The agent does the all the work of dataset generation, training, evaluation and diagnosis.

Setup

I built a fixed test set of 45 requests covering nine intents, five cases each, and froze it. Every model, change and training run was measured against those same 45 cases. The output metrics of each run were: intent accuracy, meaning did it pick the right action, and parameter accuracy, meaning did it extract the right details. A third important number is: fallbacks, the count of cases where the model declined to answer and returned “unclear.”

The OpenAI baseline (gpt-5.6-luna) was created and frozen. It scored 93.33% intent and 71.11% parameter accuracy with no fallbacks.

All local training ran on a consumer RTX 3060, in WSL2 with CUDA. LoRA fine-tuning, rank 16, alpha 32, learning rate 1e-4, batch size 4. A 15-epoch run takes about 20 minutes.

The entire source, including training data, eval harness, scripts and agent skill is on GitHub - needle-tuner.

Part 1 - needle2

The first local comparison with needle2 model from Cactus. It scored 4.44% intent accuracy. The problem turned out to be how I was asking the question. I had given the model one abstract, generic “classify this” tool. I replaced it with eight concrete tools, one for each real action the application supports.

The result went to 62.22% intent and 68.89% parameter accuracy.

With this reasonable starting point, I fine-tuned on a small purpose-built training set and ran a structured loop: change one thing, measure against the frozen test set, write down the result.

Needle2 configuration	Intent	Parameter
Base model, redesigned tools, no training	62.22%	68.89%
Best fine-tuned run (5 epochs, rank 8)	64.44%	62.22%
Longer training (30 epochs)	57.78%	64.44%
Heavier defaults	42.22%	51.11%

The best training result gained two points on intent accuracy over no training at all, and lost six on parameters. More training made things worse. The heaviest run also refused to answer far more often, with 15 fallbacks against 9 for the untrained model. The results were noisy enough to make any single comparison suspect. Two runs of the identical best configuration measured 64.44% and 60.00%, so I stopped treating one good run as evidence of anything. Validation loss kept improving while held-out accuracy got worse. Had I watched the training dashboard instead of the frozen test set, I would have shipped a regression. I stopped at this point.

Part 2 - needle3

A new version of the model family had shipped in the mean time, with a different architecture and a different native engine. Untrained, needle3 measured 71.11% intent and 66.67% parameter accuracy with 7 fallbacks. I modified the agentic loop to support the new model and executed again.

A one-epoch smoke test scored 68.89% and 73.33%, which is one run on one epoch and proves nothing on its own. A full 30-epoch run at the wrapper defaults scored 71.11% and 66.67%, identical to the untrained base. However, the per-intent breakdown had shifted underneath, some intents up and some down, while the totals stayed flat. Validation loss bottomed out at epoch 13 and climbed for the remaining 17 epochs while training loss kept falling. Same overfitting pattern needle2 had shown.

The issue

Instead of tuning another hyperparameter, I stopped reading aggregate scores and started reading individual failures.

The training data and the held-out test set agreed on which intent was correct. They disagreed on the convention for writing the parameters down.

The first problem was descriptive adjectives. For delete requests, the test set strips condition words from item names, so “the broken toaster” is expected to yield ’toaster’. All 12 authored delete rows in my training data did the opposite, keeping ‘cracked mixing bowl’ and ’expired pain reliever’ intact.

The second problem was a wrong key name. For inventory-viewing requests, the training data used a ’location’ key for the container slot while every other intent, and the test set, used ‘boxLabel’. It also carried a ‘scope’ key that appears in zero held-out cases.

The model was learning my convention faithfully and being marked wrong for it, on 100% of those cases. That also explains why longer training hurt: more epochs meant learning the wrong answer more thoroughly.

The Fix

I corrected 12 delete rows and 20 view-inventory rows, regenerated the derived datasets, re-pinned the integrity hashes and re-ran the validation suite. No hyperparameter change and dropped Epoch count from 30 to 15, taken from the previous run’s validation-loss minimum.

Result: 82.22% intent and 75.56% parameter accuracy, 6 fallbacks.

Run	Intent	Parameter	Fallbacks
Needle2 base	62.22%	68.89%	9
Needle2 best fine-tune	64.44%	62.22%	not recorded
Needle3 base	71.11%	66.67%	7
Needle3 fine-tuned, original data, 30 epochs	71.11%	66.67%	6
Needle3 fine-tuned, corrected data, 15 epochs	82.22%	75.56%	6
OpenAI (frozen baseline)	93.33%	71.11%	0

That is 11 points of intent accuracy over its own base model and 20 over needle2’s base. It is also the first time a local model beat the hosted API on parameter accuracy, 75.56% against 71.11%. Intent accuracy is still 11 points behind.

The per-intent movement matches the diagnosis. Delete went from 80%/40% to 100%/80%. Search went from 60%/60% to 100%/100%. Update improved as well, since the bare-noun convention is shared by every intent that names an item.

Validation loss behaved differently too. With the corrected data it declined every one of the 15 epochs, with none of the mid-run turnaround the unfixed run showed. The contradictory labels had been generating the overfitting signal themselves.

Parameter extraction already beats the API. Intent classification is 11 points behind, and most of the remaining gap sits in one diagnosed failure mode: deciding whether a vague request is actionable.

The next lever is contrastive training examples sitting right on that boundary. After that the options get expensive: changes to how the model produces its answers, training that explicitly teaches it what a wrong answer looks like, or a different architecture.

Hyperparameter tuning and the size of the training dataset should also be revisited to explore further improvements to intent classification.

Because the same 45 cases were used to decide which experiment to run next, they are now a model-selection set rather than an unbiased final exam. Before anyone calls this production-ready it has to be tested against a separate set of cases it has never influenced.

The project also produced a reproducible pipeline along the way. A frozen test set, hash-pinned datasets, a documented training process, a log of every attempt including the failures, and a default path that validates without training and without calling paid APIs. That infrastructure is the reason the annotation issue was discoverable at all.

WhichBox - AI Assistant App

Sun, 04 Aug 2024 17:37:56 +0000

Do you remember which one of those boxes in the garage has your old phone? Or that toy your child wants to play with again? Or the tripod you need for your weekend trip? Oh what about that massager that you could really use right now! You would probably have to dig through those boxes to find it. Same problem when moving. There are always some boxes that you don’t want to open right away, but you wish you could open just the right one when you need something.

I have created WhichBox- your AI Assistant to find that box. It uses latest Vision AI models to help you find things quickly. Here’s what you do -

Take pictures of the content inside each labelled box to create an inventory.
Use WhichBox to easily identify the box containing the item you are looking for.

You can check it out here - https://whichbox.streamlit.app/

The demo has four photos of labelled boxes with some content in them. Note that there can be multiple photos of the same box. You can take photos as you are filling up the box to capture things at the bottom.

You can ask for a specific thing, like “Find Fitbit” or just “fitbit”

You can go for a general category, like “Camera Equipment”

You can get all boxes containing “USB Adapters”

You can look for “Groot” or if you can’t remember the name of a specific toy then you can look for “all toys of movie characters”

For now, you can bring your own API Key and use your own photos to try this out.

WhichBox : https://whichbox.streamlit.app/

AI Powered Bookshelf

Mon, 29 Apr 2024 00:25:55 +0000

Bookshelf is a Generative AI application built as a rudimentary, but fairly capable, RAG implementation written in python. It can use an open source LLM model (running locally or in the cloud) or a GPT model via OpenAI’s API.

The application is created using streamlit.
I used llama-index for orchestrating the loading of documents into the vector database. Only TokenTextSplitter is currently used. It does not optimize for PDF, html and other formats.
ChromaDb is the vector database to store the embedding vectors and metadata of the document nodes.
You can use any open source embeddings model from HuggingFace.
Bookshelf will automatically use the GPU when creating local embeddings, if the GPU is available on your machine.
You can use OpenAI embeddings as well. There is no way to use a specific OpenAI embedding model or configure the parameters yet.
Use OpenAI API or any OpenAI compatible LLM API (using LMStudio, Ollama or text-generation-webui) of your choice.
There is a live demo on streamlit cloud - https://bookshelf.streamlit.app/
The demo allows only OpenAI integration. You can run it locally for accessing Open Source embedding models and LLMs.

Live demo - https://bookshelf.streamlit.app/

You will need your OpenAI api key for the demo.

If you are running it locally, you will have the option of using an Open Source LLM instance via an API Url. In the screenshot, I am using an open source Embedding Model from HuggingFace (sentence-transformers/all-mpnet-base-v2) and The local LLM server at http://localhost:1234/v1

Collections tab shows all collections in the database. It also shows the names of all the files in the selected collection. You can inspect individual chunks for the metadata and text of each chunk. You can delete all contents of the collection ( there is no warning ).

You can modify the collection name to create a new collection. Multiple files can be uploaded at the same time. You can specify if you want to extract metadata from the file contents. Enabling this option can add significant cost because it employs Extractors which use LLM to generate title, summaries, keywords and questions for each document.

On the Retrieve tab, you can query chunks which are semantically related to your query.

On the Prompt tab, you can prompt your LLM. The context as well as the Prompt Template is editable.

Here is an example of using the context retrieved from chunks in the Vector database to query the LLM.

This inference was performed using Phi3 model running locally on LMStudio.

Code is on Github - https://github.com/ashtewari/bookshelf

Have fun!

Iterative Prompt Engineering with LLMStudio

Mon, 05 Feb 2024 01:10:11 +0000

Creating the ideal prompt can be the key to transforming a less than average outcome into one that is remarkably relevant. Discovering the right prompt often involves numerous revisions and a method of trial and error. I felt the need to refer to my past attempts, modifications made to prompts, LLM models I tried and other LLM settings as well relative cost of these combinations. There are a few options to help with this : promptflow, langsmith, LLMStudio and others.

I tried LLMStudio and promptflow. This article is about LLMStudio.

If you are installing LLMStudio on Windows, use WSL. Here are the steps :

Create a folder on your machine.
Create .env file here with following content:

OPENAI_API_KEY="sk-api_key"
ANTHROPIC_API_KEY="sk-api_key"

Enter wsl and create a new python environment with conda

(base) PS c:\code\llmstudio> wsl
(base) ash@DESKTOP:/mnt/d/code/llmstudio$ conda activate lmstd

LLMStudio runs on Bun. Bun is a javascript runtime like Node.

Instructions for installing bun on windows :
https://bun.sh/docs/installation#windows

sudo apt-get install unzip
powershell -c "irm bun.sh/install.ps1|iex"

Install Node v18, if you run into ReferenceError: Request is not defined at Object. (/home/ash/miniconda3/lib/python3.11/site-packages/llmstudio/ui/node_modules/next/dist/server/web/spec-extension/request.js:28:27)

Start LLMStudio

Export the execution data as csv file by clicking on [Export data] button. This data includes the input, output, LLM model, Input and Output tokens as well as cost.

References
What is LLMstudio?
LLMstudio by TensorOps
LLM Studio Quickstart
Install Bun for Windows
Setup Bun JS in windows using WSL and VS code
NextJS - ReferenceError: Request is not defined

Prompt Engineering and Security - Custom GPT

Sun, 21 Jan 2024 23:01:24 +0000

I have been delving into Advanced Prompt Engineering and Security techniques for Large Language Models (LLMs). As an exercise, I have created a custom GPT in ChatGPT to help practice spelling of English words. Spell It GPT is secured with Advanced Prompt Engineering techniques to guard against common attack vectors, including Direct Prompt Injection, Prompt Leaking, Role Playing, Simulation, DAN and Code Injection.

Spell-It-GPT

Play with the Spell It GPT and try to break it. It is not impossible but (probably) fairly difficult to do ;) Regardless, practice spelling and have fun!

Switch to Voice Mode in ChatGPT Mobile App to practice spelling and put headsets on for best results!

Spell It: https://bit.ly/3Ua0oib

Here are some resources you can learn more about Prompt Engineering and Security:
Advanced Prompt Engineering Techniques
OWASP Top 10 for Large Language Model Applications
Educational Resources
Adversarial Prompting in LLMs