Finetune on Ash Tewari

All Things Open 2026 - Session Recommendations

Tue, 20 Oct 2026 01:52:51 +0000

All Things Open 2026 is on in Raleigh! With more than 100 sessions and speakers, choosing what to attend is not easy.

I went through the conference program and shortlisted the sessions that caught my attention. My interests lean toward AI agents, software architecture, developer productivity, security, and data engineering, practical engineering challenges, along with a few interesting topics on model training, real-time AI, and emerging user interfaces.

AI coding assistants have evolved considerably. We’re now moving toward autonomous agents that can modify repositories, execute commands, provision infrastructure, and even deploy applications. But giving an agent the ability to write code is relatively easy. Making it reliable, secure, efficient, and useful in a real engineering environment is a much harder problem.

AI Coding Agents and Developer Infrastructure

1. Death by Worktree: Why Agent Sandboxes Choke on Disk I/O

Speaker: Bradley Hilton

Why attend: Git worktrees and isolated sandboxes are becoming common approaches for running multiple coding agents concurrently. But filesystem performance can become a serious bottleneck. This session addresses an often-overlooked infrastructure problem: the I/O overhead associated with agent execution environments. Particularly relevant for anyone experimenting with parallel coding agents and multi-repository workflows.

2. Building Functional Dev Environments for Humans and Agents

Speaker: Matyas Danter — Staff Solutions Engineer, Coder

Why attend: AI agents need reproducible development environments just as much as human developers do. The talk explores practical approaches to managing development environments across monorepos, multiple repositories, and distributed teams.

3. AI Sandboxes: Running Coding Agents Safely in Production-Grade Environments

Speaker: Shivay Lamba — Lead ML Engineer, Qualcomm

Why attend: Coding agents execute commands and generate code that may not be entirely trustworthy. Sandboxing is becoming essential for safely scaling autonomous development.

4. Sandboxing AI Agents in the Enterprise: A Practitioner’s Guide to Isolation Technologies

Speaker: Balu Mathew — Machine Learning Engineering Technical Leader, Cisco

Why attend: An architectural perspective on isolation technologies for enterprise AI agents. I’m particularly interested in understanding the security, performance, and operational trade-offs between different sandboxing approaches.

5. The Last Mile of AI-Assisted Development

Speakers: Colin Griffin and David Johnson — Krumware / SUSE

Why attend: Generating code is only part of software delivery. Building, testing, deploying, and maintaining it remain substantial challenges. The session includes a look at extending Rancher using Epinio, an open-source Kubernetes application deployment platform. Worth considering for its practical platform-engineering angle, although the presentation has a vendor-specific component.

Agent Security, Permissions, and Governance

6. You Can Give Any Agent Access to kubectl. But How Do You Make One That Should Have Access to kubectl?

Speaker: Charles Johnson — Mezmo

Why attend: Connecting an agent to Kubernetes is easy; deciding whether it should be trusted with cluster operations is not. This session challenges the assumption that giving agents write access to production systems must inevitably end in disaster. An interesting opportunity to explore controlled autonomy, operational safeguards, and the boundary between agent capabilities and permissions.

7. Fine-Grained Access Control for Hosted MCP Servers You Don’t Own

Speaker: Nick Taylor — Developer Advocate, Pomerium

Why attend: One of the more practical security topics on my list. Nick demonstrates an identity-aware proxy that enforces per-tool authorization across hosted MCP servers without modifying them. The approach addresses credential sprawl, OAuth limitations, and tool-call auditing. Nick has presented extensively on MCP and Zero Trust security, including at Black Hat and other technical conferences. The demo uses open-source Pomerium.

8. AI Wrote the Code – Is It Secure?

Speaker: Adhithi Ravichandran — Founder and Software Consultant, Surya Consulting

Why attend: AI-generated code can introduce vulnerabilities even when it appears functionally correct. This session examines the security implications of AI-assisted software development. Adhithi’s software consulting background should offer a practitioner perspective on the security checks and engineering discipline needed before trusting AI-generated code in production.

Model Training, Real-Time AI, and New Interfaces

9. The Beginners Guide to Training Your Own AI Coding Model on Your Own Data

Speaker: Bekah Hawrot Weigel — The Paper Compute Company

Why attend: I’m particularly interested in small, specialized coding models trained on custom datasets. Understanding how to prepare training data and adapt a model for specific coding tasks is becoming increasingly valuable. This should be a useful introduction for developers exploring model customization beyond prompting and RAG.

10. How Open is your Realtime AI Stack?

Speaker: Omar Warraky — Senior Solutions Architect, LiveKit

Why attend: Real-time voice AI requires considerably more than an LLM. Speech recognition, synthesis, media transport, orchestration, and latency management all matter. This session should be particularly interesting for understanding the open-source components and architectural choices available when building low-latency conversational applications.

11. Where Is My Futuristic App? Generative UI and the Open Future

Speakers: Mark Techson, Andrew Brogdon, and Marc Dougherty — Google

Why attend: What happens when agents generate not just content but the user interface itself? This keynote explores Generative UI and the A2UI protocol, including a look at a working application’s codebase. Although Google-led, the focus on open protocols and actual implementation makes it worth watching.

12. To Boldly Go: Data Science Exploration with AI Agents (ft Star Trek’s LCARS Theme)

Speaker: Dawn Wages — Director, Community and Developer Relations, Anaconda

Why attend: A creative approach to exploring data science through AI agents, using Star Trek’s LCARS interface as inspiration. This looks like an enjoyable opportunity to see how conversational agents and interactive interfaces can make data exploration more approachable.

Software Engineering and the Developer Community

13. When Code Costs Nothing: Tech Debt in the Age of GenAI

Speaker: Rachel Stephens — Research Director, RedMonk

Why attend: AI dramatically reduces the effort required to produce code, but does it reduce the long-term cost of maintaining software? This session examines how generative AI changes our understanding of technical debt, code ownership, and engineering accountability. This session should bring an independent industry-research perspective, making this particularly appealing as a counterbalance to AI productivity hype.

14. AI Tinkerers Raleigh: Top Demos of the Year and Their Community Impact

Speaker: Danny Castro — AI Tinkerers Raleigh

Why attend: Sometimes the most interesting innovations come from local builders rather than major technology companies. This demo showcase highlights practical AI projects from the Raleigh community. A good opportunity to discover emerging tools, meet developers, and see what people are actually building.

Final Thoughts

Looking across these sessions, a few important themes stand out:

Agent infrastructure matters. Sandboxing, and reproducible environments are becoming fundamental building blocks of AI-assisted development.
Security cannot depend on trusting the model. Permission boundaries, isolation, authorization, and auditing must be enforced outside the agent itself.
The software engineering bottleneck is shifting. Generating code is becoming easier, while validation, maintainability, and production readiness remain difficult.

See the complete All Things Open Tuesday schedule for session times and locations.

Looking forward to seeing what the speakers and the open-source community have been working on!

Agentic finetuning using a custom harness

Fri, 25 Sep 2026 02:56:11 +0000

The goal was to figure out whether a small model running on my own hardware could do the job of intent classification and structured parameter extraction well enough for one of my use-cases - read a request like “add 8 spare tent stakes to the camping box” and turn it into something the application can act on.

I created an agentic fine-tuning and evaluation loop with a custom evaluation harness: an AI agent proposes one change at a time, a custom evaluation harness measures it against a frozen test set, and every run gets appended to a log with its settings, result and next decision. The agent does the all the work of dataset generation, training, evaluation and diagnosis.

Setup

I built a fixed test set of 45 requests covering nine intents, five cases each, and froze it. Every model, change and training run was measured against those same 45 cases. The output metrics of each run were: intent accuracy, meaning did it pick the right action, and parameter accuracy, meaning did it extract the right details. A third important number is: fallbacks, the count of cases where the model declined to answer and returned “unclear.”

The OpenAI baseline (gpt-5.6-luna) was created and frozen. It scored 93.33% intent and 71.11% parameter accuracy with no fallbacks.

All local training ran on a consumer RTX 3060, in WSL2 with CUDA. LoRA fine-tuning, rank 16, alpha 32, learning rate 1e-4, batch size 4. A 15-epoch run takes about 20 minutes.

The entire source, including training data, eval harness, scripts and agent skill is on GitHub - needle-tuner.

Part 1 - needle2

The first local comparison with needle2 model from Cactus. It scored 4.44% intent accuracy. The problem turned out to be how I was asking the question. I had given the model one abstract, generic “classify this” tool. I replaced it with eight concrete tools, one for each real action the application supports.

The result went to 62.22% intent and 68.89% parameter accuracy.

With this reasonable starting point, I fine-tuned on a small purpose-built training set and ran a structured loop: change one thing, measure against the frozen test set, write down the result.

Needle2 configuration	Intent	Parameter
Base model, redesigned tools, no training	62.22%	68.89%
Best fine-tuned run (5 epochs, rank 8)	64.44%	62.22%
Longer training (30 epochs)	57.78%	64.44%
Heavier defaults	42.22%	51.11%

The best training result gained two points on intent accuracy over no training at all, and lost six on parameters. More training made things worse. The heaviest run also refused to answer far more often, with 15 fallbacks against 9 for the untrained model. The results were noisy enough to make any single comparison suspect. Two runs of the identical best configuration measured 64.44% and 60.00%, so I stopped treating one good run as evidence of anything. Validation loss kept improving while held-out accuracy got worse. Had I watched the training dashboard instead of the frozen test set, I would have shipped a regression. I stopped at this point.

Part 2 - needle3

A new version of the model family had shipped in the mean time, with a different architecture and a different native engine. Untrained, needle3 measured 71.11% intent and 66.67% parameter accuracy with 7 fallbacks. I modified the agentic loop to support the new model and executed again.

A one-epoch smoke test scored 68.89% and 73.33%, which is one run on one epoch and proves nothing on its own. A full 30-epoch run at the wrapper defaults scored 71.11% and 66.67%, identical to the untrained base. However, the per-intent breakdown had shifted underneath, some intents up and some down, while the totals stayed flat. Validation loss bottomed out at epoch 13 and climbed for the remaining 17 epochs while training loss kept falling. Same overfitting pattern needle2 had shown.

The issue

Instead of tuning another hyperparameter, I stopped reading aggregate scores and started reading individual failures.

The training data and the held-out test set agreed on which intent was correct. They disagreed on the convention for writing the parameters down.

The first problem was descriptive adjectives. For delete requests, the test set strips condition words from item names, so “the broken toaster” is expected to yield ’toaster’. All 12 authored delete rows in my training data did the opposite, keeping ‘cracked mixing bowl’ and ’expired pain reliever’ intact.

The second problem was a wrong key name. For inventory-viewing requests, the training data used a ’location’ key for the container slot while every other intent, and the test set, used ‘boxLabel’. It also carried a ‘scope’ key that appears in zero held-out cases.

The model was learning my convention faithfully and being marked wrong for it, on 100% of those cases. That also explains why longer training hurt: more epochs meant learning the wrong answer more thoroughly.

The Fix

I corrected 12 delete rows and 20 view-inventory rows, regenerated the derived datasets, re-pinned the integrity hashes and re-ran the validation suite. No hyperparameter change and dropped Epoch count from 30 to 15, taken from the previous run’s validation-loss minimum.

Result: 82.22% intent and 75.56% parameter accuracy, 6 fallbacks.

Run	Intent	Parameter	Fallbacks
Needle2 base	62.22%	68.89%	9
Needle2 best fine-tune	64.44%	62.22%	not recorded
Needle3 base	71.11%	66.67%	7
Needle3 fine-tuned, original data, 30 epochs	71.11%	66.67%	6
Needle3 fine-tuned, corrected data, 15 epochs	82.22%	75.56%	6
OpenAI (frozen baseline)	93.33%	71.11%	0

That is 11 points of intent accuracy over its own base model and 20 over needle2’s base. It is also the first time a local model beat the hosted API on parameter accuracy, 75.56% against 71.11%. Intent accuracy is still 11 points behind.

The per-intent movement matches the diagnosis. Delete went from 80%/40% to 100%/80%. Search went from 60%/60% to 100%/100%. Update improved as well, since the bare-noun convention is shared by every intent that names an item.

Validation loss behaved differently too. With the corrected data it declined every one of the 15 epochs, with none of the mid-run turnaround the unfixed run showed. The contradictory labels had been generating the overfitting signal themselves.

Parameter extraction already beats the API. Intent classification is 11 points behind, and most of the remaining gap sits in one diagnosed failure mode: deciding whether a vague request is actionable.

The next lever is contrastive training examples sitting right on that boundary. After that the options get expensive: changes to how the model produces its answers, training that explicitly teaches it what a wrong answer looks like, or a different architecture.

Hyperparameter tuning and the size of the training dataset should also be revisited to explore further improvements to intent classification.

Because the same 45 cases were used to decide which experiment to run next, they are now a model-selection set rather than an unbiased final exam. Before anyone calls this production-ready it has to be tested against a separate set of cases it has never influenced.

The project also produced a reproducible pipeline along the way. A frozen test set, hash-pinned datasets, a documented training process, a log of every attempt including the failures, and a default path that validates without training and without calling paid APIs. That infrastructure is the reason the annotation issue was discoverable at all.