Today we're launching LangSmith Fine-Tuning and smithtune, a CLI that helps teams turn LangSmith trajectories into custom fine-tuned models for their agents. It handles the entire fine-tuning process from one CLI: dataset creation and preparation from LangSmith trajectories, training with Fireworks or Baseten, and evaluation with LangSmith. You can run smithtune directly or work with your coding agent to run commands and inspect the results. smithtune is built for post-training models. It currently supports supervised fine-tuning (SFT) which trains a model using examples of good behavior. You give the model inputs/outputs and it learns by updating model weights to imitate that behavior. LangSmith trajectory data is designed to support SFT, and smithtune helps turn these trajectories into useful training data. This gives teams a way to train specialized models without building the data pipeline by hand. A model trained on your examples can often perform as well as or better than a general purpose frontier model on specified tasks, often at lower cost and latency.
One of the biggest drivers of fine-tuning gains is data selection. With the launch of smithtune, it’s easier to connect that loop — from curated production traces in LangSmith to managed training and a served model on Fireworks. Teams can move seamlessly from data to training to deployment without standing up infrastructure along the way. That’s the path we’re excited to be building with LangChain.
– Pranav Jain, Product Lead at Fireworks
smithune's integration with Baseten Loops makes it seamless for teams to go from data collection to running fine-tuning experiments in minutes. Builders are able to continuously collect and curate better data to produce better models over time with the fully managed training infrastructure that Loops provides so they can focus on designing for their biggest customer use-cases.
– Aaron Ellis-Bloor, Applied Researcher at Baseten
Agent trajectories and post-training
Before diving into the capabilities smithtune provides, let’s first discuss agent trajectories. A trajectory is an ordered sequence of messages, tool calls, and tool results that shows how an agent worked through a task. LangSmith assembles this sequence from a trace or thread, brings supported message formats into a common representation, preserves tool definitions, and removes duplicated history. LangSmith trajectory format was designed with post-training in mind. For SFT, the student model needs the exact context the teacher model had when it produced a successful result. In complex long-running agents, tool availability context often changes as the agent works (for example, deferred tool loading), and a naive export of the final message list loses that nuance. LangSmith's trajectory format records precisely what the model saw at every turn, so smithtune can pair each action with its true context. smithtune uses trajectories throughout the workflow. You curate successful example trajectories, prepare them for your chosen model, and train on their recorded responses and tool calls. Trajectories kept out of the training split provide the context and reference actions for evaluation. Step-by-step walkthrough Build your dataset A dataset contains ‘golden’ trajectories that a target model will fit to. This data is the foundation for supervised fine-tuning. There are a few key stages when creating a dataset with smithtune :
- Pull Dataset: smithtune pulls trajectories from a LangSmith tracing project to a local directoryDIR , with optional filters.
- Label Traces: smithtune works with humans (and their agents) to identify characteristics of “good” traces, create a rubric to based on this, and then sends a council of agents to review and filter trajectories that are good candidates for SFT
- Store a persistent dataset: smithtune makes sure that any data used for training can be audited later as a persistent artifact. It uploads the agreed on set of golden trajectories to a LangSmith dataset for training and evaluation.
Along the way smithtune handles details such as:
- making sure trajectories are compatible with a chosen model by filtering traces that are beyond a given sequence length
- splitting data into train/val/tests splits for downstream evaluation Train a model Before committing to training, smithtune plan helps humans review their settings such as the selected model, the number of training examples, and hyperparameters like the learning rate, batch size, and epochs. You can adjust these settings before running smithtune train to start the fine-tuning job. smithtune submits the job to Fireworks managed SFT or Baseten Loops which support LoRA training on your prepared trajectories. There's no GPU provisioning or training infrastructure to manage on your side. During training, smithtune also checks performance on the validation set and selects the saved checkpoint with the lowest validation loss. Evaluate the result After training completes, run smithtune evaluate to compare the selected checkpoint with the base model. smithtune uses a built-in replay evaluation to test the base vs fine-tuned model. Models are evaluated on being able to complete actions from a golden trajectory and a judge scores those predictions against the true recorded examples. The CLI returns a LangSmith comparison link, with results appearing as evaluation progresses. You can compare scores, inspect individual responses and tool choices, and see where fine-tuning helped or introduced regressions. Deploy your model If you’re happy with your evaluation results, use smithtune deploy to serve your tuned model and connect it to your application. If the results aren’t what you were looking for, refine your dataset or adjust the training settings, then train and evaluate again. You can review the comparison in LangSmith with your coding agent to identify which responses or tool choices need more work. Results from running this in practice To assess the quality of our the smithtune flow, we applied it to two highly used agents at LangChain: