Methodology & technical report

Each module is a performance exam: candidates do real tasks in sandboxed simulations (terminal, spreadsheet, file workspace, browser, editor, AI agents) rather than answering trivia. Graders check outcomes — the file exists with the right contents, the formula returns the right value — and accept any valid method. Efficiency (time versus an expert par, shortcut use) adds at most 15% to an item score, so speed never outweighs correctness.

A level requires the cut on all items up to that level and at least 60% on items at that level, so easy items alone cannot earn a high level. Unproctored results are capped at L3 publicly; L4–L5 need a proctored exam. Module-specific cuts from standard setting override the defaults.

Typing (net WPM with accuracy gate)

Candidates with approved motor accommodations can earn an “accuracy and efficiency” typing credential without the speed gate.

• Item score = correctness (0–1, partial credit via checkpoints) × (0.85 + 0.15 × efficiency).
• Scaled score 200–800 with a confidence band from the binomial standard error (e.g. 612 ± 25). IRT calibration replaces this once each item has enough responses.
• Every result stores the exam version, each item's version and the scoring version. Re-scoring an old attempt reproduces the same result.
• Role readiness and the Productivity Index (0–1000) are derived from module results; changing role weights never requires a retest.
• Percentiles compare against a disclosed reference group (module, mode and year) with the sample size shown.

Items move through draft → independent SME review (six-point checklist; authors cannot approve their own items) → pilot (embedded unscored in real forms) → calibration (≥ 5 responses; difficulty, discrimination and exposure checked) → live → retired. Live items are immutable; changes create a new version. Cut scores are set with a modified Angoff panel: subject-matter experts estimate the chance that a borderline candidate answers each item; the panel mean becomes the recommended cut and is revisited annually.

Target reliability ≥ 0.85 per track and typing test–retest ≥ 0.9 (spec §7). Forms are randomised, so split-half on mixed forms is a conservative estimate; small samples are unstable and shown for transparency.

Standardised p-difference matched on rest-score quintiles. Class C items (|DIF| ≥ 0.10) go back to review for possible bias (keyboard layout, language, accommodations).

Written reports, prompt tasks, routing decisions, Markdown and redaction are graded automatically against checklists. A deterministic 20% sample goes to a human auditor who sees the response and rubric but not the candidate. Disagreements beyond ±15 points re-score the item and the result.

Audited so far: 2 · agreement: 50% · mean absolute difference: 0.29.

Standard exams use behavioural signals (focus loss, paste events, keystroke plausibility, timing) to compute a risk score; nothing is decided automatically. Flags go to trained reviewers, every decision is logged in a hash-chained audit log, and candidates can appeal to a different reviewer. Proctored exams add ID verification, webcam and screen capture, kept 90 days. Employers never see raw keystrokes or video.

Content validity comes from SME-approved blueprints mapped to job tasks. Criterion validity studies against employer-rated job performance run with partner organisations and are published here as they complete. Limitations: simulations approximate real applications; results reflect performance on one day; L4–L5 claims rely on proctoring; percentiles depend on who chooses to test.