# Agent-ready data marts: documentation and SQL accuracy

An LLM analyst answered 27 of 40 banking questions correctly from a bare schema and 33 from documented
data marts; the allowed values of the category columns alone got it to 31.
A paired test on synthetic data, for analytics engineers and data leaders building marts that agents query.

Open [report.html](report.html) directly in a browser. It works without a server, API key or network
request. Every question expands to show all three original responses, executed rows and expected answers.

## Observed result

On 40 fixed questions, the bare schema answered 27/40 correctly, the schema plus allowed values
answered 31/40 and the full documentation bundle answered 33/40. Full documentation minus bare is
+15.0 percentage points (paired bootstrap interval 0.0 to +30.0). Giving the bare schema only the
distinct values of its category columns closed 4 of those 6 answers (+10.0 points, interval 0.0 to
+22.5). What full documentation adds on top of the allowed values is +5.0 points, with an interval
from −7.5 to +17.5 that includes zero. All three arms scored 0/4 on the clarification questions.

| Question type | Questions | Bare schema | Schema plus allowed values | Full documentation |
|---|---:|---:|---:|---:|
| All questions | 40 | 27 | 31 | 33 |
| Analytical SQL | 36 | 27 | 31 | 33 |
| Clarification | 4 | 0 | 0 | 0 |

| Comparison | Difference, pp | Question-level interval | Twin-clustered interval |
|---|---:|---:|---:|
| Full documentation minus bare | +15.0 | 0.0 to +30.0 | 0.0 to +31.4 |
| Allowed values minus bare | +10.0 | 0.0 to +22.5 | −2.6 to +24.3 |
| Full documentation minus allowed values | +5.0 | −7.5 to +17.5 | −7.5 to +17.5 |

Intervals are paired percentile bootstraps with 10,000 resamples and seed 7. The twin-clustered
interval resamples the CAD and USD versions of a question as one unit (32 clusters); it was added
after the first results were seen. The questions are purposive and related, and each arm is one
model run, so the intervals describe this set and do not establish generalization or variation
across model reruns. Errors, malformed answers, denied queries and failed requests stay failures.
These are draft portfolio results pending founder verification.

Why answers failed: the bare schema guessed category values that do not exist in 6 of its 9 wrong
analytical answers (for example direction = 'inbound' where the values are 'credit' and 'debit').
With the value lists no answer guessed a value, but 3 still used occurred_at where the posting date
was needed. Full documentation, which defines the posting rule, had none. See
[failure-causes.csv](failure-causes.csv).

### Two declared changes after the first scoring (founder rulings D12 and D23)

1. **Validator correction.** The evaluator's SQL validator denied EXISTS as if it were a function,
   although the prompt permits ordinary subqueries. One bare answer (B35, NOT EXISTS) had been scored
   query_denied; in the sandbox it returns the expected accounts. All 80 cached answers were rescored
   with EXISTS treated like AND, OR and NOT ([banking/correction.py](banking/correction.py)), with no
   new model calls. The original scores, 26/40 versus 33/40 (+17.5 points, interval +5.0 to +32.5),
   stay in [results.json](results.json) and still reproduce; the rescored run is
   [results-corrected.json](results-corrected.json).
2. **Third arm: schema plus allowed values.** The bare arm's exact prompt plus one message listing the
   distinct values of every non-identifier text column ([the list](documentation/value-domains.json)),
   and no other documentation. Same model, settings, questions and scoring. Its protocol addendum,
   [evals/values-arm-protocol-addendum.json](evals/values-arm-protocol-addendum.json), was frozen with
   its sha256 before any request. One of its 40 requests failed at the provider (HTTP 503) and is kept
   as an incorrect outcome without a retry.

## Checks behind the numbers

- The 40 questions, their expected answers and the run 1 prompts were frozen before the scored
  collection. The allowed-values arm froze its own protocol by hash before its first request.
- Every expected answer is computed twice: by SQL over the marts and by a separate Python
  calculation over the source records that never reads a mart. The two agree on all 36 analytical
  questions.
- Every daily balance is reconciled against an independent fold of the account's events, and every
  spend row against a separate grouping. Perturbation tests show that pending or void amounts can't
  move a posted mart and that August postings can't move July.
- Money compares exactly in integer minor units, so a one-cent error on a large balance is wrong.
  SQL errors, denied queries, malformed answers and the failed request all count as wrong.

The tests are in [tests/](tests/); `make check` runs them.

## What is real and what is simulated

The accounts, cards, categories and transactions are deterministic fictional records. No customer
or employer data is used. The SQL execution, model responses, token usage, costs and paired scoring
are real. Neither generated fixtures nor corrected model responses count as model answers.

All arms receive the same physical schema, model snapshot, raw-SQL interface, settings, output-token
ceiling and sandbox permissions. The documented arm adds [the documentation bundle](documentation/bundle.md):
the allowed values of the status, event kind, direction and currency columns, metric definitions,
grain, currency, posting time, refunds, transfers and safe joins. The allowed-values arm adds only
the value list. Gold SQL agrees with independent Python calculations over the source events.
Monetary values are compared exactly in integer minor units. There is no FX conversion or
available-balance model.

Run 1 (bare and documented) froze its questions and prompts before collection. Three separate
development questions were collected and replayed first; their responses are excluded from the
score. Collection used gpt-4.1-mini-2025-04-14, temperature 0, seed 7 and an 800-token output
ceiling in every arm.

## Reproduce without credentials

Prerequisites: Linux or WSL with user/network namespaces enabled, bubblewrap available as bwrap,
Python 3.12 and make. The SQL worker requires /usr/bin/python3 to be Python 3.12. DuckDB and sqlglot
are pinned in [requirements.lock](requirements.lock). No live model request occurs during replay.

From this directory:

```sh
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.lock
make check
make reproduce
```

The initial dependency installation needs package access. After installation, the complete
reproduction uses the included caches and dependency archive. `make reproduce` runs, in order:
`replay` (run 1 as originally scored, into results.json), `followup` (the validator correction,
the allowed-values arm and the three-arm tables), `report` (report.html and assets/accuracy.svg)
and `tables` (the run 1 CSVs). An existing result is compared with the reproduced value and
retained; a mismatch fails. There is no unsafe SQL fallback if namespaces are unavailable.

To use another Python executable that has these exact dependencies:

```sh
make PYTHON=/absolute/path/to/python reproduce
```

The evaluator is the unchanged version 0.1.0 archive in vendor/. Every member is verified before
extraction into runtime/. Its included mobility sample is unused by this banking study; the original
notice is retained in [NOTICE.txt](NOTICE.txt). The page styling is viz-design-system release
2.0.4, copied unedited into vendor/viz-design-system/ and pinned by hash in
[vendor/viz-design-system-adoption.json](vendor/viz-design-system-adoption.json). All data marts and
runtime files belong to this project.

## Inspect the outputs

- [Interactive details in the static report](report.html): all three answers to every question.
- [Three-arm results](results-three-arm.json), [summary](summary-three-arm.csv) and
  [120 outcomes](question-outcomes-three-arm.csv), scored with the corrected validator.
- [Failure causes](failure-causes.csv) and the [hand labels](evals/failure-labels.json) behind them.
- [Rescored run 1](results-corrected.json); [original run 1 results](results.json),
  [outcomes](question-outcomes.csv) and [summary](summary.csv), kept unchanged.
- [Accuracy chart](assets/accuracy.svg): correct answers per arm.
- [Run 1 protocol](DESIGN.md), [frozen run manifest](evals/run-manifest.json) and
  [cache manifest](evals/cache-manifest.json); [allowed-values addendum](evals/values-arm-protocol-addendum.json)
  and [its cache manifest](evals/cache-manifest-values.json).

The model SQL is executable untrusted input. It runs in a restricted namespace with only the public
evaluation database, no checkout, no credentials, no external network, read-only data and bounded
CPU, memory, wall time and output. File access, extensions, mutations and multiple statements are
denied. Responses and queries are displayed as escaped text.

## Acquisition, time and limits

Known model cost was about US$0.06 for the 80 run 1 requests and US$0.02 for the 40 allowed-values
requests (exact values in results.json and results-three-arm.json). Run 1 including development
cost about US$0.06; the study cap was US$10 and the allowed-values arm had its own US$0.50 cap.
One allowed-values request has an unknown cost because it failed at the provider.

The paid collectors are `collect.py` (run 1) and `collect_values.py` (allowed-values arm); neither
is imported by replay, and reproduction does not run them. Frozen runs refuse changed prompts,
data, settings or core code. Collection requires verified prices less than 24 hours old and
refuses unresolved reservations or a budget overrun. All cached responses and failures remain intact.

The build began 2026-10-08T09:57:30.003720+00:00; 100.3 elapsed minutes through the first report
build, recorded in [build-info.json](build-info.json). The validator correction, the allowed-values
arm and the restyled report were added on 2026-10-10. This artifact does not establish
production-bank performance. The three metadata
examples illustrate the bundle; apart from the allowed-values arm, no individual-field ablation is
claimed.