Technical Evidence

Technical Evidence.
Reproducible. Verifiable.

This page documents the architecture, methodology, test results, and reproducibility of the BuildIT Index certification harness. Every claim here can be independently verified against the repository and the demo data.

1. Architecture

The certification harness is a Python CLI application. The reference platform is a Next.js application. They are separate but related codebases in the same repository.

Certifier (Python)

  • Entry point: scripts/certify-model.ts (TypeScript wrapper calling Python)
  • Core logic: certifier/ package (Python 3.10+)
  • Dependencies: numpy, pandas, scipy (standard scientific stack)
  • Database: None required. Operates on CSV input only.
  • Output: STDOUT verdict + exit code (0=PASS, 1=OVERFIT, 2=MARGINAL, 3=FAIL)

Reference Platform (Next.js)

  • Framework: Next.js 15 (App Router)
  • Runtime: Node 22 (Node 24 breaks Prisma install)
  • Database: PostgreSQL + Prisma ORM (50 models)
  • API routes: 102 routes
  • Pages: 73
  • Lines of code: ~172,000 in src
  • Commits: 452
  • Features: Paper trading only, Yahoo Finance data, single-operator
Figure 1. High-level architecture. The certifier runs independently; the platform provides a reference implementation and UI.
BuildIT Index Architecture Diagram Certifier: CLI → CSV input → Python core → Validation gates → Verdict + exit code. Platform: Next.js app → PostgreSQL/Prisma → API routes → Paper trading UI. Both in same repository. CERTIFIER (Python) CLI Entry Point ▼ CSV Input (predictions) ▼ Python Core (certifier/) ▼ Verdict + Exit Code REFERENCE PLATFORM (Next.js) Next.js 15 App Router ▼ PostgreSQL + Prisma (50 models) ▼ 102 API Routes · 73 Pages ▼ Paper Trading UI

2. Validation Methodology

The harness applies eight validation components in sequence. Each has a defined boundary. Nothing is described as more than it is.

01 Recompute Accuracy Verdict gate · always runs

Accuracy is recomputed from raw predictions. Any claimed-accuracy column in the file is ignored entirely.

Boundary It can only recompute what the file contains. A file with no usable predictions is rejected, not scored.
02 Baselines Verdict gate · 2 on bare CSV

The model is compared against baselines computed from the data supplied — not against a number someone wrote down.

Boundary A bare CSV runs 2 baselines: always-bullish and no-change. All 5 defined baselines need price history. If the history isn’t supplied, the other three simply don’t run — they are never faked.
03 Walk-Forward Consistency Verdict gate · period stability

Accuracy is measured fold by fold, earliest to latest, so decay shows up instead of hiding inside one average.

Boundary This is period stability: how the model holds up across time periods. Purged walk-forward exists as a tested library function, but it is not a DIRECTION verdict gate. The distinction is stated here because buyers should not have to discover it later.
04 Multiple-Testing Correction Verdict gate · Bonferroni

Trying many configurations and reporting the best one is corrected for — the more you tried, the harder it is to pass.

Boundary Bonferroni correction is applied to the number of configurations tried. That count is supplied with the run, so it is only as honest as the input — the harness does not pretend to know what it wasn’t told.
05 Uncertainty Intervals Wilson intervals · always runs

Every accuracy figure carries a Wilson interval, so a small sample can’t dress up as certainty.

Boundary Wilson intervals are computed from the sample supplied. They describe the sample, not the market.
06 Leakage Gate Verdict gate · pass / fail

Checks for leakage — information in the test data that the model shouldn’t have had when it made the prediction.

Boundary A pass means the supplied columns pass the gate. It is not a proof about unseen data.
07 Probability of Backtest Overfitting (PBO) Not a gate · needs candidates file

When a candidates file is supplied, the harness estimates the probability that the chosen model only looks best because many were tried.

Boundary For a single model with no candidates file, PBO is null — and never a gate. It answers one question: among everything you tried, how likely is this winner a fluke?
08 Calibration Verdict gate · always runs

Checks whether the model’s stated confidence matches how often it is actually right.

Boundary Calibration needs a stated confidence. Without one there is nothing to compare.

3. Dated Test Results

All results below are from the verified demo fixture (vendor-model-sample.csv, 600 rows) run against the certifier at commit d06dcaf (2026-11-03).

Commit
d06dcaf (2026-11-03)
Fixture
vendor-model-sample.csv (600 rows)
Command
npx tsx scripts/certify-model.ts examples/vendor-model-sample.csv --configurations 20
Recomputed Accuracy
69.33% (416 / 600 correct)
Claimed Accuracy
78.00% (ignored)
Wilson 95% CI
65.5% – 72.9%
Baselines (bare CSV)
2 run: always-bullish, no-change
Walk-Forward Folds
5 folds: 80.00% → 77.50% → 71.67% → 61.67% → 55.83%
Fall Across Folds
−24.17 pts
Leakage Gate
Passed
Verdict
OVERFIT · exit code 1
PBO
Null (no candidates file supplied)
Calibration
Ran (no confidence column in fixture)

These results are reproducible. Run the command above against the fixture to verify.

4. Runtime Requirements

Minimal dependencies. No database. No external services required for the certifier.

Certifier

Python 3.10+

Standard library + numpy, pandas, scipy. No TensorFlow, PyTorch, or ML frameworks required.

  • OS: Linux, macOS, WSL (no native Windows)
  • Python: 3.10, 3.11, 3.12
  • numpy ≥ 1.24
  • pandas ≥ 2.0
  • scipy ≥ 1.10
Reference Platform

Node 22 + PostgreSQL

Required for the Next.js platform. The certifier does not need this.

  • Node: 22.x (24.x breaks Prisma)
  • PostgreSQL: 15+
  • Prisma: 5.x
  • pnpm or npm

Estimated Setup Time

3–5 engineer-days for a clean environment. The certifier itself runs in seconds.

Running Costs (Platform)

$35–80 / month at hobby scale (Vercel + managed Postgres). Model weights (Chronos-2, ~460 MB) download at first run.

5. Reproducibility Package

Everything needed to independently verify the demo results.

Downloadable Assets

Verification Steps

  1. Download vendor-model-sample.csv
  2. Verify SHA256: 8b3747cd980b388249b92f32d00d81c554e90f4998c158d63558b477c8d3f5a3
  3. Run: npx tsx scripts/certify-model.ts examples/vendor-model-sample.csv --configurations 20
  4. Confirm output matches §3 exactly

Browser vs CLI

The homepage terminal demo runs the same algorithm in-browser (JavaScript port of the Wilson interval and fold logic). The CLI is the authoritative implementation. Both produce identical results for the demo fixture.

6. Dependency & Licensing Summary

Third-party dependencies and their licenses. Verified at commit d06dcaf.

Certifier (Python)

PackageVersionLicense
numpy≥1.24BSD-3-Clause
pandas≥2.0BSD-3-Clause
scipy≥1.10BSD-3-Clause

Reference Platform (Key Dependencies)

PackageVersionLicense
next15.xMIT
react18.xMIT
prisma5.xApache-2.0
@prisma/client5.xApache-2.0
tailwindcss3.xMIT
zod3.xMIT

Note: Full dependency trees with transitive licenses available in the repository (pnpm licenses list / pip-licenses).

7. Known Limitations

Honest boundaries. These are not bugs — they are the defined scope.

01

Bare CSV = 2 baselines only

Price history required for the other 3 baselines. The demo fixture has no price history.

02

PBO is null without candidates file

The demo does not supply a candidates file. PBO is not computed, not "passed".

03

Calibration needs confidence column

The demo fixture has no confidence column. Calibration runs but has nothing to compare.

04

No live broker execution

Platform is paper trading only. No licensed market data feed included.

05

Single-threaded certifier

No parallel fold evaluation. Large fixtures take proportionally longer.

06

MIT history (1–22 Oct 2026)

Repository was public under MIT. Anyone who copied then holds a perpetual MIT licence to that snapshot.

07

AI-assisted development

Copyright in purely AI-generated expression is unsettled. Disclosed in writing to buyers.

The number on the page is the number the tool returns.

Request the technical proof pack See the eight checks