[ Case Study ]

Lazy Dunmas

Ask a D&D rules question and get a ruling from the SRD 5.1 that cites the rules it rests on, or browse the SRD in a searchable encyclopedia.

  • React
  • Python
  • Evals
  • Ollama
  • RAG
  • Structured Output

Source on GitHub (opens in a new tab)

[ The Problem ]

A rules dispute stops a D&D game while someone searches the book. A general chatbot answers faster, but without the rules behind its answer, so the table still has to look them up to trust it.

[ How It Works ]

Lazy Dunmas rules from the SRD 5.1, the System Reference Document: the part of D&D 5th edition's core rules that Wizards of the Coast publishes free under a Creative Commons license. That license is what lets the app search the rules and quote them back in every ruling.

Every question goes through five steps before a ruling comes back.

  1. Check the question. A first call checks that it’s about D&D rules.
    • If it isn’t, it comes back to you to edit.
  2. Split it into sub-questions. The model names each separate rule the question depends on, and writes a short question for each one.
  3. Search the SRD. The original question and each sub-question are searched on their own, each finding the SRD entries closest to it in meaning.
  4. Merge the results. Those searches are combined into one ranking, and the top ten entries are kept.
  5. Rule. The model rules from those ten entries only, and cites the ones it used.
    • The ruling flags when the rules don’t settle the question.

Each step streams to the screen as it finishes. The models run locally through Ollama, so no API key is needed. The screen is React, built on Loom.

[ Measured ]

The app is scored on a fixed set of questions with known answers. Each answer is scored twice:

  1. The search: did it find the rules the correct ruling needs?
  2. The ruling: is its verdict the correct one?

Scoring them apart shows which part failed when an answer is wrong.

The questions

Each question comes with its correct ruling and the SRD entries that ruling rests on. Some are contested on purpose: the SRD's text supports two different answers, so the correct ruling gives both and marks the question as ambiguous. An agent drafted each label from the SRD text, and I reviewed every one.

The questions are divided into two sets, so the results aren't flattered by questions the app was built around:

  • 98 development questions are the ones every design choice was tried on: how the SRD is cut into entries, which embedding model to use, how the prompts are worded.
  • 67 test questions were held back while the app was built. Everything below is scored on them, once.

The searches

The SRD is cut into entries, and a search returns the entries it ranks highest. Plain search, the base the others build on, was chosen on the development questions:

  • Entries: each SRD section under its own header is one entry, and each record, such as a spell or a monster, is one entry whole. This beat fixed 800-character windows, and the same entries with the names of the sections above them added to their text.
  • Embedding model: an embedding model turns text into numbers, so that passages with similar meaning land close together. nomic-embed-text was the best of the three tried, and the fastest.
  • Keyword search: rules questions rarely use the rules’ own words, so keyword search (BM25) found fewer questions’ rules than search by meaning. Merging the two with reciprocal rank fusion (RRF), which scores each entry by its rank in every list merged, did worse than search by meaning alone.

Five ways of searching are then compared on the test set, each handing the model exactly ten entries:

  • Plain search: the question is embedded, every SRD entry is ranked by how close it is in meaning, and the top ten are kept.
  • Reranked: plain search’s top thirty are rescored by a cross-encoder, BAAI/bge-reranker-v2-m3, which reads the question and each entry together. Its order is merged with plain search’s using RRF.
  • Named lookup: the spells, weapons, conditions, races, class features, magic items and creatures the question names are found by name and put ahead of plain search’s order.
  • Split first: the question and each sub-question are searched on their own, and the rankings are merged with RRF. This is what the app runs.
  • Tool loop: the model gets two tools, a search that returns the top five entries in full and a lookup by name, for up to six calls. The records it looked up go first, then the question’s search and its own searches, merged with RRF.

A search has found a question's rules when every entry the correct ruling needs is among the ten it hands the model.

Scoring the ruling

A ruling is right when its verdict means the same as the correct one. Matching the text can't decide that, since "No, she can't" and "no" are the same answer, so a second model compares them: Claude Sonnet 5.5, used only as this judge. Before its scores counted, I graded twelve of its verdicts by hand, blind, and it agreed with me on all twelve.

Contested questions have no single verdict, so they never go to the judge; they're scored on the flag alone. A reply that doesn't fit the ruling's structure counts as wrong, and is never retried. Two more scores:

  • Every citation has to name a real SRD entry.
  • The ruling's ambiguity flag, from step 5 above, is scored two ways: how many of the questions that should be flagged it marked as ambiguous, and how many settled questions it marked by mistake.

Results

I tested every search with two models writing the ruling. The local model is qwen3.5:9b, the one I used. The other is Claude Opus 5.5, through the API, to show the best result to aim for.

The last row is a fake model that answers "no" to every question. It is judged the same way, so it shows the score any model has to beat.

The table has two scores for each model:

  • Found is the share of the 67 test questions where every entry the correct ruling needs was among the ten handed to the model. For plain search, reranked and named lookup, the model plays no part in the search, so its two columns are the same. For split first and the tool loop, the model writes the sub-questions or runs the searches itself, so the two columns differ.
  • Right is how many of the 55 questions with a single correct answer got a verdict the judge counted as matching it. The contested questions are left out, since they have no single verdict.

Each row is a single run.

SearchFound, localFound, OpusRight, localRight, Opus
Plain search54%54%42 / 5548 / 55
Reranked58%58%45 / 5546 / 55
Named lookup61%61%42 / 5547 / 55
Split first55%70%44 / 5551 / 55
Tool loop67%91%41 / 5553 / 55
Always "no"—27 / 55

Finding the rules is where the searches differ most. Plain search put every needed entry in front of the model for 54% of questions. Reranking and named lookup each raised that a little, to 58% and 61%, without the model taking part. Split first and the tool loop depend on the model’s own sub-questions and searches: with Opus they reached 70% and 91%, and with the local model 55% and 67%.

For Opus, finding more of the rules led to more right rulings. It went from 48 of 55 with plain search to 53 with the tool loop, gaining five questions and losing none.

The local model shows no such climb. Its rulings ranged from 41 to 45 of 55 across the five searches, in no order that follows how much each search found. In reruns on an earlier version of the question set, its score moved by up to four questions on the same prompts, as much as that whole range, so one run can’t rank its searches against each other.

The second table adds the two scores from "Scoring the ruling" that the first one leaves out: the ambiguity flag and the citations. It covers only split first, because that is the search the app runs.

Fourteen questions should be flagged as ambiguous. Twelve are the contested ones.

The other two are rules-as-intended questions. Read word for word, the SRD gives one answer, but the rule plainly means another, and the correct ruling takes the intended one. In one, a creature is 15 feet from a fireball's center, but around a corner. The general rule says a spell's area spreads only in straight lines, which leaves the creature out. Fireball's own text says the fire spreads around corners, which only means something if it overrides that rule. So the correct ruling is that the creature is caught.

These two have a verdict, so the judge scores them. A good ruling also flags them, because someone reading only the text could reach the other answer.

ScoreLocalOpus
Should be flagged, and were0 / 145 / 14
Settled, flagged by mistake4 / 530 / 53
Citations naming a real entry108 / 121186 / 186

The local model flagged none of the fourteen. It also flagged four settled questions by mistake. Opus flagged five of the fourteen, and no settled questions. Every citation Opus gave named a real entry, while 13 of the local model's 121 did not. So the local model's flag tells a player nothing: it never went up when it should have. Opus raised it on about a third of the questions that needed it, and never on one that didn't.

[ Under the Hood ]

The code and data the measurement rests on.

The shape every model has to return. The model first names the kind of question, and the verdict has to fit it: a yes-or-no question gets yes or no, and a how-many question gets a number. A reply that doesn't fit this shape counts as wrong.

> src/rules_attorney/answer.py
class YesOrNo(BaseModel):
    """An answer to a yes-or-no question."""

    kind: Literal["yes-or-no"]
    verdict: Literal["yes", "no"]


class HowMany(BaseModel):
    """An answer to a how-many question."""

    kind: Literal["how-many"]
    # A number answer is written as a bare integer. Forced into a string, the
    # verdict to a how-many question came out as braces: "}", "{2}".
    verdict: int

# …

# The model names the kind of question first, and a verdict of the wrong
# type for that kind is refused.
Answer = Annotated[YesOrNo | HowMany | What, Field(discriminator="kind")]


class Ruling(BaseModel):
    """The ruling a model returns."""

    # Model writes the fields in the order they're declared,
    # so it gives its reasoning and the flag before it commits to a verdict.
    explanation: str
    ambiguous: bool
    answer: Answer
    cites: list[str]

[ More Case Studies ]

JAMIE

A persistent AI development partner built on agent harnessing and context engineering.

[ read case study ]

Loom

A design system generator — describe a brand in one file, get the styles and Figma library every project is built from.

[ read case study ]

Taulu

A digital bulletin board for spontaneous event planning — events are posters, your people and calendars are the boards.

[ read case study ]

Paperboy

A news digest and the monitor behind it: charts that show which feeds are failing, and how.

[ read case study ]