Diogo TravassosData leadership · Analytics · Decision-making
Enterprise AI adoption: a measurement plan
Link successful tasks, answer quality and repeat use to renewal, then test the links in three account-randomized experiments. The first, a guided first task, is sized for its own rollout rule: roll out only if the 98.33% interval for the activation gain sits above zero.
| Accounts needed360for the activation trial: 180 per arm, 7,200 users, ICC 0.05 | Chance of rollout80%if activation truly rises from 30% to 35%; the earlier rule gave 50% | If correlation is high1,232accounts at ICC 0.30; 542 if account sizes vary (CV 1.0) | Earliest readout6 weeks4 weeks of recruitment, then 14 days to final outcomes |
Main limitation Every enterprise rate, correlation and account count here is an assumption; month one must measure them before this size means anything. | |||
The arrows are a causal story to test. Procurement, account maturity, task mix and contract timing can produce the same correlations without any product effect.
No arrow is a measured effect. Each measure is defined below.
Unit: eligible user. The first qualifying task completes within 7 days of assignment and is not reopened in the next 7 days, so every outcome is final on day 14. Divide activated users by all users on the frozen eligible roster, including nonusers.
Unit: eligible user. A qualifying task succeeds in at least 3 of 4 follow-up weeks. Report this share against the original roster, alongside the distribution of active weeks. Do not condition the primary analysis on activation.
Unit: attempted task. A blinded audit checks correctness, evidence support, permission boundaries and an appropriate refusal when needed. Sample failures and abandoned tasks too. Report audit coverage and correctness separately; a thumbs-up is not a gold label.
Unit: account. Report signed paid-seat expansion and incremental recurring contract value against the baseline account cohort. Remove trial seats and distinguish price changes from seat changes. This is a secondary commercial measure.
Unit: renewal-eligible account. Report paid contract renewal by the prespecified deadline and gross retained recurring value excluding expansion. Keep unresolved renewals visible. Short-term repeat use is an early signal, not a substitute for renewal.
A qualifying task has an approved use case, a unique task ID and a terminal outcome. A successful task produces the requested usable output, passes the use-case rubric and is not reopened within 7 days. Repeated prompts for the same goal share one task ID. Activation applies this same test to the first task: completed by day 7, reopen check through day 14.
Proposed events include account assignment, eligible-roster snapshot, exposure, task start, terminal outcome, reopen, audit result and contract change. Store pseudonymous account and user IDs, event time, task type and versioned rubric IDs. Do not put raw prompts or document text in analytical events. Respect existing consent, access and retention controls. Reconcile task outcomes with source logs and contracts with billing.
For the weekly-use endpoint, observe completions in each of the first 4 weeks and wait another 7 days for reopens. Missing outcomes count as no observed success in the main analysis; investigate instrumentation failure and report sensitivity bounds before interpreting a treatment effect.
Randomize accounts 1:1 within prespecified size, deployment and baseline-use strata. Every member of an account receives the same arm. Freeze eligibility before assignment and analyze intention-to-treat. Exclude no account because it failed to use the feature. Run disjoint account cohorts or sequence the trials to prevent treatment interactions.
E1 · Activation
E2 · Repeat use
E3 · Renewal
These are proposed rules to freeze before launch. The three primary tests share family alpha 0.05 through Bonferroni allocation of 0.05/3 per two-sided test. Report 98.33% primary intervals. This controls the three planned primary comparisons; it does not license selecting a favorable subgroup or metric. Secondary metrics remain exploratory.
For E1 and E2, use the account as the independent unit and retain fixed baseline roster weights for a user-average effect. Cluster-robust intervals need enough accounts; with few accounts use a prespecified account-level randomization test. For E3 use an account-weighted renewal comparison. Final analysis and power must be simulated using actual cluster sizes and the intended estimator before launch.
Proposed noninferiority margins are 2 percentage points for audited correctness, 2 seconds for task-latency p95, 10% for support hours per assigned account and 2% of baseline value for gross retained recurring value per account. Require the appropriate one-sided 95% bound to clear each relevant harm margin. Guardrail sample sizes and missingness must support that precision; contract values are heavy-tailed, so the 2% margin may need a larger sample or a different summary. Any confirmed critical permission breach pauses treatment and starts incident review; zero observed incidents alone does not prove safety.
Observe assignment, actual exposure and shared-workspace membership separately. Consultants with access to several accounts, shared templates and cross-account content can contaminate controls. Group linked accounts before randomization or exclude those links by a predeclared eligibility rule; then recalculate power for the larger clusters. Never remove contaminated accounts after outcomes are known. If containment fails, report dilution and uncertainty and do not present the result as an isolated account effect.
No repeated significance peeking. Safety monitoring can stop a trial for harm; success is assessed only at the frozen endpoint. An imprecise guardrail or an under-recruited primary test is inconclusive, not a pass.
This is a planning calculation, not a claim that these accounts exist or that the final analysis reaches its target. It assumes activation rises from 30% to 35%, a 5 percentage-point gain: the smallest gain worth a rollout and the gain the trial is sized to detect. Target power is 80% at two-sided alpha 0.05/3, with accounts allocated equally, exactly 20 eligible users per account, independent accounts and no attrition.
Total E1 accounts, both arms, by assumed within-account correlation (ICC). Every row keeps a mean of 20 eligible users per account; the last row lets account sizes vary.
| ICC | Size CV | Design effect | Accounts per arm | Total accounts | Total users |
|---|---|---|---|---|---|
| 0.00 | 0.0 | 1.00 | 92 | 184 | 3,680 |
| 0.05 (base) | 0.0 | 1.95 | 180 | 360 | 7,200 |
| 0.10 | 0.0 | 2.90 | 267 | 534 | 10,680 |
| 0.20 | 0.0 | 4.80 | 441 | 882 | 17,640 |
| 0.30 | 0.0 | 6.70 | 616 | 1,232 | 24,640 |
| 0.05 | 1.0 | 2.95 | 271 | 542 | 10,840 |
Source: power-results.json (rows and cluster_size_rows), computed by power.py from assumptions.json.
ICC is the assumed correlation of activation between users of the same account. At ICC 0.05 the design effect is 1.95. Admin enablement acts on whole accounts, so the range runs to 0.30 until month one measures it. Unequal account sizes add to the burden: with the same mean and a size coefficient of variation of 1.0, the base case needs 542 accounts. User counts are observations; account counts govern the number of independent assignments.
| True gain (treatment rate) | This plan: interval above zero | Earlier rule: estimate also 5 points or more |
|---|---|---|
| 0 points (30%) | 0.8% | 0.0% |
| 2.5 points (33%) | 22.5% | 5.1% |
| 5 points (35%) | 80.3% | 50.0% |
| 7.5 points (38%) | 99.3% | 94.6% |
With no true gain the rule rolls out 0.8% of the time, the one-sided share of alpha. The earlier version of this plan also required the estimate itself to reach 5 points, which makes a true 5-point gain a coin flip at any size. Raising that rule's chance to 80% means assuming a larger true gain, and the answer swings with the guess: 610 accounts at 6 points and 186 accounts at 7 points. This plan uses the interval rule instead and keeps 5 points as the gain to detect. Guardrail tests are not sized here; requiring them to pass can only lower these chances.
A fixed public mobility question set illustrates separate scoring of analytical answers and clarification, including failures and uncertainty.
Read the static evaluation snapshotA synthetic marketplace study illustrates why assignment, spillovers and carryover change the contrast being estimated.
Read the static simulation snapshotThese are local, frozen method snapshots. Neither measures enterprise AI outcomes nor supplies any input to the power calculation. Their underlying project results remain draft portfolio evidence, and they contain summary results, not the original interactive replay or executable simulation.
For independent users, use pooled null variance and Bernoulli alternative variance with equal allocation. The baseline variance is 0.30 × 0.70; the treatment variance is 0.35 × 0.65. This normal approximation omits continuity correction and the opposite rejection tail, matching the usual planning convention.
pbar = (p0 + p1) / 2
n = [z(1-alpha/2) * sqrt(2*pbar*(1-pbar))
+ z(power) * sqrt(p0*(1-p0) + p1*(1-p1))]^2 / (p1-p0)^2
DE = 1 + ((1+CV^2)*m-1)*ICC
accounts_per_arm = ceil(n*DE/m)Here n is users per arm before clustering, m is the mean number of eligible users per account, CV is the coefficient of variation of account sizes (0 in the ICC rows) and z is the standard normal quantile. The design effect is the variance of the fixed-weight user average across independent accounts, from adding the variances and covariances of every user pair. Rounding is upward to whole accounts.
Constant ICC, no attrition and the stated size variation are simplifying assumptions. Informative account size, connected accounts, rare outcomes, few clusters and missing data require a revised model. Recruitment and guardrail feasibility can still rule out the experiment. E2, E3 and the guardrails need their own endpoint-specific calculations.
The first version of this plan (8 October 2026) required the 98.33% interval to clear zero and the estimate to reach 5 points, with the same 360 accounts. A true 5-point gain clears its own threshold only half the time, so that rule's chance of a rollout decision was 50%, not the 80% the sizing implied. The rule now uses the interval alone, which is the rule the size formula describes. The calculation also gained the ICC rows above 0.10, the account-size row and the decision table; the base row did not move. The earlier inputs and outputs are kept unchanged: earlier assumptions and earlier calculation output.
No build, service, account or API key is needed. Open this page directly. With Python 3.12.3 and the standard library, python3 -B check.py runs the numerical tests, regenerates the three pages and the calculation output and compares them byte for byte, checks the README's calculation paragraph against the same templates and resolves every local link. The tests include an inverted normal-tail reference, R's power.prop.test documentation value, covariance enumeration for equal and unequal account sizes, the decision table against a direct search, and a seeded simulation (seed 20261010, 2,000 simulated trials of beta-binomial accounts) of both rollout rules. It changes no files.
Declared assumptions · Full calculation output · Calculation code · Tests · Provenance and limitations · R planning reference