# Documentation and SQL answer accuracy

This study measures a documentation bundle on a fictional banking dataset.
It compares paired answers on 40 questions; it does not estimate performance on real customers.

## Comparison fixed before acquisition

The bare arm receives physical table and column names, DuckDB types, the common answer protocol
and one question. The documented arm receives exactly those messages plus documentation/bundle.md.
Both use the same six tables, raw-SQL adapter, model snapshot, temperature, seed, output-token
ceiling and namespace sandbox. Neither receives data samples, gold SQL, gold rows, prior answers
or feedback. No semantic compiler is used. Documentation length and information are intentionally
part of the intervention; no matched-length placebo or individual-field ablation is claimed.

Forty purposively authored final questions cover spend, dimensions, joins, cash flows, balances,
posting boundaries, nonposted events and clarification. Thirty-six require SQL; four require a
specific clarification. The questions are related, not an independent sample of banking tasks.
Development cases and their outcomes stay separate. Final prompts and cache identities are frozen
before final paid collection. Retain every response, refusal, malformed answer and failed request.
No post-outcome edits to questions, gold answers, documentation, settings or selected headline run.
A defect discovered after freezing requires an explicit invalidated-run record and a new run, with
all old outcomes retained. Never rerun only wrong answers to improve the score.

## Synthetic data contract

All rows are generated from seed 20261008. No customer or employer records are used. Time is UTC.
The snapshot is 2026-08-03 at 00:00. Starting balances apply at 2026-07-01 at 00:00.
Four sources contain accounts, cards, categories and transaction events. The fixture includes two
currencies, partial refunds, paired transfers, missing categories, multiple cards, zero activity,
closed accounts, pending/void events and events that cross a month at posting.

Two physical marts expose daily account balances and daily card/category spend. Every balance row
is reconciled against an independent Python event fold; every spend row against a Python grouping.
Source primary keys, foreign keys, currency consistency, timestamps, refund totals and transfer
pairs are checked before a database is built. Perturbing pending or void amounts cannot change
posted marts. Perturbing August postings cannot change July marts. Positive controls compare
individual accounts because equal debit and credit perturbations can cancel in a grand total.

## Gold and scoring

Every analytical question has SQL gold and a separate Python oracle over the source CSV records.
The oracle does not read either mart or execute SQL. Both must agree before freezing. Question
output columns have a fixed order; aliases do not matter. Row order matters only when requested.
Duplicate multiplicity and NULL identity matter. All financial outputs use integer minor units;
there is no currency conversion or implicit combined total across currencies.

Reuse the unchanged version 0.1.0 evaluator archive, including RawSQLAdapter,
DuckDBDatasetAdapter, execute_prepared, rows_equal, cache_key and paired summarizer. Both arms use
the same exact comparison policy: rows_equal with atol=0 and rtol=0 after sandbox execution.
This consumer tightens the shared comparator's optional tolerances because a one-cent difference
must remain wrong even on a large balance. It does not change the dependency source.

A clarification answer must contain precisely the predeclared missing dimension(s), with no SQL.
An analytical answer that requests clarification is incorrect; unsupported certainty on an
ambiguous question is incorrect. Malformed JSON, denied SQL, SQL errors and timeouts remain failures
in the denominator. A broken namespace boundary aborts evaluation rather than grading the model.

Report bare and documented correct counts, the paired difference and a paired percentile bootstrap
with 10000 question resamples and seed 7. Show analytical and clarification strata separately,
as well as the combined rubric score. The interval describes sensitivity within this fixed set;
it does not establish population generalization, model rerun variation or a causal ranking of
metadata fields. Report ties, regressions and improvements with the same presentation.

## Acquisition and reproduction

The budget is US$10 including development, failed requests and retries. A separate acquisition
command reads the API key. Reserve the UTF-8 request byte ceiling plus output-token cap at verified
provider rates before each request. Unknown cost fails closed. Keep unresolved reservations, use
no automatic retries and stop on access/quota failures. Alternate which arm goes first by question.
Record original latency, token usage, prices and actual spend separately from replay time.

Default reproduction reads the complete frozen cache without credentials or provider imports.
Cache identity binds question text, arm, exact model/settings, full prompt, logical dataset/schema
and documentation. Logical identity uses source and model hashes; the built database also has a
physical hash for mutation checks. Record but do not assume byte-identical DuckDB rebuilds.

Model SQL executes only through the shared restricted SELECT validator and namespace worker.
The worker has public data, no checkout, no credentials, no network, bounded CPU/memory/time/output,
and disabled external access and extensions. Its database is read-only and checked unchanged.
Presentation escapes every model answer and query. A hostile-markup test is required before export.

## Three preselected metadata examples

These examples explain the intervention and are chosen before model outcomes:
1. Posting time determines the financial period; occurrence time describes the initiating event.
2. Account balances have account/date grain; joining unaggregated cards can multiply balances.
3. Refunds reduce spend when they post; transfer debit and credit legs are cash movement, not spend.

These are illustrations of the bundle, not estimates of each field's causal contribution.

## Delivery limits

AI-assisted development and the actual build interval will be recorded in the README. There is no
real-bank endorsement, production deployment, fraud model, available-funds calculation or FX model.
Source/copy research is retained privately. All results need complete copy/source mapping,
independent public-export replay and clean-checkout reproduction before local completion.