Skip to the yard
Waxsealai

Evaluation Driven Development Helps Dosu Track LLM Reliability at Scale

Shipped
Shipper
David Brown
Readers
343

Dosu uses production traces and failure examples to improve its software support assistant while checking that prompt changes don’t cause regressions.

Evaluation Driven Development Helps Dosu Track LLM Reliability at Scale

For an LLM product, a prompt change can fix one kind of request and quietly break another. Dosu’s approach to that problem is evaluation driven development: observe failures in production, turn them into evaluation examples, and use those examples to guide the next change. LangSmith gives the team a way to search the growing stream of activity and feed what it learns back into that cycle.

Why Dosu needs a quality loop

Dosu is an AI assistant for developing, maintaining, and supporting software projects. It grew from CEO Devin Stein’s experience as an open source maintainer, where answering questions and triaging issues left less time for building features. That load can contribute to maintainer burnout, while users may wait days, weeks, or months for a response. Similar work extends beyond open source: one cited study found developers spend up to 85% of their time on tasks outside coding, including ad hoc questions and issue handling.

Dosu aims to help with that work by responding to questions about codebases, troubleshooting errors, and providing information about project features. That can give maintainers more room to develop software and offer community members quicker help with problems that require knowledge of a codebase.

The challenge is that Dosu’s responses come from a system with probabilistic components. A change to a model, prompt, or other part of its logic can have unpredictable effects. A tweak might improve answers in one area while making another worse, so the team needs a repeatable way to check both gains and regressions.

From manual review to evaluation driven development

Dosu launched in late June 2023. At first, request volume was low enough for the team to inspect every response. They used basic command line tools and print statements to review logs, learn how people used Dosu, and spot tasks it handled well or poorly. This close review helped shape the system, but it became harder to sustain as usage grew.

The team adopted evaluation driven development, or EDD, as a way to set measurable targets for changes. Like test driven development, EDD gives engineers a set of examples to check against. These evaluations, often called evals, provide a baseline for assessing changes to prompts, models, and core logic.

Dosu’s process begins with a new behavior and a small set of evals. The team releases the behavior, watches how it performs in production, and looks for failure patterns. It then adds examples of those failures to its offline evaluation datasets, makes improvements against the updated evals, and releases another version. The cycle repeats as new problems emerge.

The method depends on seeing what happens in production. As Dosu expanded to thousands of repositories, ran around the clock, and added dozens of submodules for different scenarios, reviewing individual responses became impractical. The team needed monitoring that could surface problems without pulling prompts and evaluation work out of its existing development practices.

Tracing the work behind each response

Dosu’s requirements reflected how its team builds software. Prompts had to remain in Git and receive the same care as code changes. Traces needed to capture relationships between steps in the application, not just isolated LLM calls. The team also wanted to export data into its existing evaluation tools and choose which metadata to record as its system evolved.

After considering monitoring options, the team chose LangSmith. The SDK was a key part of that decision: Dosu added a @traceable decorator to several LLM related functions and began receiving traces in the LangSmith interface after deploying the changes.

The decorator can capture both function activity and LLM calls in a single trace. That lets engineers inspect raw function inputs, rendered prompt templates, and model outputs together, making it easier to follow how a response was produced. The team can also attach metadata to traces, which adds context for searching and reviewing activity.

Searching for failure patterns

Dosu handles varied requests, from straightforward questions about a codebase to error logs from a library upgrade or questions about a feature’s status. Each kind of input can produce different failure modes. The team watches several signals: direct user feedback, user sentiment in GitHub discussions, internal errors such as oversized inputs or outputs that fail to match a required schema, and response time.

Speed is not the only measure of quality. Some requests call for a quick answer, while others may need more processing to produce a precise response. Tracking delays helps the team understand which cases are slow and why.

LangSmith search lets engineers filter traces by criteria such as feedback, errors, delays, and sentiment. Together with custom metadata, those searches help them find unusual behavior across production activity and gather further examples of a problem.

Some of the failures were not obvious during early development. The team found very slow responses when users included thousands of lines of logs or raw floating point values from an OpenAI embedding. These cases point to a practical limit for systems that accept broad, unstructured inputs: large payloads can create performance problems even when the request itself is valid.

Another unexpected response came when Dosu was asked to label a pull request. Instead of labeling it, the assistant talked about being excited to attend a concert that night. The example is unusual, but it illustrates why reviewing traces matters: a response can be fluent and still miss the requested task entirely.

When the team identifies a failure mode, it searches for related traces, adds examples to its evaluation datasets, and iterates against them before releasing an updated version. That turns production observations into checks that can help catch similar regressions in later changes.

Building datasets from production traffic

Dosu’s next step is to customize LangSmith so engineers can assemble evaluation datasets from production conversations more easily. The team wants to curate examples by topic, user segment, request category, and other characteristics. If that workflow becomes straightforward, engineers can draw on real usage to keep evaluation sets relevant as the product and its users change.

The approach depends on a continuing loop: production traces reveal problems, evaluation examples make those problems easier to test, and subsequent improvements can reduce the support work Dosu handles. The team describes a reciprocal effect with LangChain: as Dosu improves, it can reduce maintenance and support demands on the LangChain team, giving that team more time to develop LangSmith, which Dosu then uses to refine its own system.

For teams building LLM products, the practical lesson is that evaluation depends on more than a fixed test set. It also needs a way to find failures in real use, inspect the steps that produced them, and turn those cases into repeatable checks. Dosu’s experience suggests that tracing and dataset curation become increasingly important as manual review stops scaling.

Source: David Brown · www.langchain.com

Same vesselRelated cargo