JAMIE
A persistent AI development partner built on agent harnessing and context engineering.
[ read case study ]Ask a D&D rules question and get a ruling from the SRD 5.1 that cites the rules it rests on, or browse the SRD in a searchable encyclopedia.
A rules dispute stops a D&D game while someone searches the book. A general chatbot answers faster, but without the rules behind its answer, so the table still has to look them up to trust it.
Lazy Dunmas rules from the SRD 5.1, the System Reference Document: the part of D&D 5th edition's core rules that Wizards of the Coast publishes free under a Creative Commons license. That license is what lets the app search the rules and quote them back in every ruling.
Every question goes through five steps before a ruling comes back.
Each step streams to the screen as it finishes. The models run locally through Ollama, so no API key is needed. The screen is React, built on Loom.
The app is scored on a fixed set of questions with known answers. Each answer is scored twice:
Scoring them apart shows which part failed when an answer is wrong.
Each question comes with its correct ruling and the SRD entries that ruling rests on. Some are contested on purpose: the SRD's text supports two different answers, so the correct ruling gives both and marks the question as ambiguous. An agent drafted each label from the SRD text, and I reviewed every one.
The questions are divided into two sets, so the results aren't flattered by questions the app was built around:
The SRD is cut into entries, and a search returns the entries it ranks highest. Plain search, the base the others build on, was chosen on the development questions:
nomic-embed-text was the best of the three tried, and the fastest.Five ways of searching are then compared on the test set, each handing the model exactly ten entries:
BAAI/bge-reranker-v2-m3, which reads the question and each entry together. Its order is merged with plain search’s using RRF.A search has found a question's rules when every entry the correct ruling needs is among the ten it hands the model.
A ruling is right when its verdict means the same as the correct one. Matching the text can't decide that, since "No, she can't" and "no" are the same answer, so a second model compares them: Claude Sonnet 5.5, used only as this judge. Before its scores counted, I graded twelve of its verdicts by hand, blind, and it agreed with me on all twelve.
Contested questions have no single verdict, so they never go to the judge; they're scored on the flag alone. A reply that doesn't fit the ruling's structure counts as wrong, and is never retried. Two more scores:
I tested every search with two models writing the ruling. The local model is qwen3.5:9b, the one I used. The other is Claude Opus 5.5, through the API, to show the best result to aim for.
The last row is a fake model that answers "no" to every question. It is judged the same way, so it shows the score any model has to beat.
The table has two scores for each model:
Each row is a single run.
| Search | Found, local | Found, Opus | Right, local | Right, Opus |
|---|---|---|---|---|
| Plain search | 54% | 54% | 42 / 55 | 48 / 55 |
| Reranked | 58% | 58% | 45 / 55 | 46 / 55 |
| Named lookup | 61% | 61% | 42 / 55 | 47 / 55 |
| Split first | 55% | 70% | 44 / 55 | 51 / 55 |
| Tool loop | 67% | 91% | 41 / 55 | 53 / 55 |
| Always "no" | — | 27 / 55 | ||
Finding the rules is where the searches differ most. Plain search put every needed entry in front of the model for 54% of questions. Reranking and named lookup each raised that a little, to 58% and 61%, without the model taking part. Split first and the tool loop depend on the model’s own sub-questions and searches: with Opus they reached 70% and 91%, and with the local model 55% and 67%.
For Opus, finding more of the rules led to more right rulings. It went from 48 of 55 with plain search to 53 with the tool loop, gaining five questions and losing none.
The local model shows no such climb. Its rulings ranged from 41 to 45 of 55 across the five searches, in no order that follows how much each search found. In reruns on an earlier version of the question set, its score moved by up to four questions on the same prompts, as much as that whole range, so one run can’t rank its searches against each other.
The second table adds the two scores from "Scoring the ruling" that the first one leaves out: the ambiguity flag and the citations. It covers only split first, because that is the search the app runs.
Fourteen questions should be flagged as ambiguous. Twelve are the contested ones.
The other two are rules-as-intended questions. Read word for word, the SRD gives one answer, but the rule plainly means another, and the correct ruling takes the intended one. In one, a creature is 15 feet from a fireball's center, but around a corner. The general rule says a spell's area spreads only in straight lines, which leaves the creature out. Fireball's own text says the fire spreads around corners, which only means something if it overrides that rule. So the correct ruling is that the creature is caught.
These two have a verdict, so the judge scores them. A good ruling also flags them, because someone reading only the text could reach the other answer.
| Score | Local | Opus |
|---|---|---|
| Should be flagged, and were | 0 / 14 | 5 / 14 |
| Settled, flagged by mistake | 4 / 53 | 0 / 53 |
| Citations naming a real entry | 108 / 121 | 186 / 186 |
The local model flagged none of the fourteen. It also flagged four settled questions by mistake. Opus flagged five of the fourteen, and no settled questions. Every citation Opus gave named a real entry, while 13 of the local model's 121 did not. So the local model's flag tells a player nothing: it never went up when it should have. Opus raised it on about a third of the questions that needed it, and never on one that didn't.
The code and data the measurement rests on.
The shape every model has to return. The model first names the kind of question, and the verdict has to fit it: a yes-or-no question gets yes or no, and a how-many question gets a number. A reply that doesn't fit this shape counts as wrong.
class YesOrNo(BaseModel):
"""An answer to a yes-or-no question."""
kind: Literal["yes-or-no"]
verdict: Literal["yes", "no"]
class HowMany(BaseModel):
"""An answer to a how-many question."""
kind: Literal["how-many"]
# A number answer is written as a bare integer. Forced into a string, the
# verdict to a how-many question came out as braces: "}", "{2}".
verdict: int
# …
# The model names the kind of question first, and a verdict of the wrong
# type for that kind is refused.
Answer = Annotated[YesOrNo | HowMany | What, Field(discriminator="kind")]
class Ruling(BaseModel):
"""The ruling a model returns."""
# Model writes the fields in the order they're declared,
# so it gives its reasoning and the flag before it commits to a verdict.
explanation: str
ambiguous: bool
answer: Answer
cites: list[str]A persistent AI development partner built on agent harnessing and context engineering.
[ read case study ]A design system generator — describe a brand in one file, get the styles and Figma library every project is built from.
[ read case study ]A digital bulletin board for spontaneous event planning — events are posters, your people and calendars are the boards.
[ read case study ]A news digest and the monitor behind it: charts that show which feeds are failing, and how.
[ read case study ]