Is your backtest real?
TokIO tells you whether a strategy's profit is distinguishable from luck, once you allow for what a Sharpe ratio hides: P&L that isn't independent from bar to bar, and every variant you tried before this one. Every test it runs has a measured false-positive rate.
pip install tokio-ai
import tokio_ai
print(tokio_ai.check_backtest(pf)) # a vectorbt Portfolio, a whole grid included
print(tokio_ai.check_backtest(stats)) # backtesting.py's Backtest(...).run()
print(tokio_ai.check_backtest(tokio_ai.read_tradingview("List of trades.csv")))The best of 19 SMA crossovers on SPY
Tested alone, the best one is significant: p = 0.038. Tested with the other eighteen it was picked from: p = 0.12, haircut Sharpe 0.21, and a probability of backtest overfitting of 0.90: the in-sample winner did worse than a random pick out of sample in 90% of splits.
What every result reports
- A test of the mean that allows for serial dependence (HAC, adaptive window), corrected for every variant you tried (Romano-Wolf step-down, the stepwise form of White's Reality Check)
- The Deflated Sharpe Ratio and Probabilistic Sharpe Ratio, as published and corrected for dependence, and the minimum track record length
- The probability of backtest overfitting (CSCV) for grids
- Costs, turnover and the breakeven transaction cost
- A lookahead check: the Sharpe with every position taken one bar later
- Concentration (both tails trimmed), consistency across time, and the minimum backtest length
We measured the tests
How often does each method call a strategy with no edge profitable? At p < 0.05 it should be 5%.
| P&L with no edge | plain t-test | published PSR | TokIO |
|---|---|---|---|
| overlapping 20-day trades, simulated | 38% | 38% | 4–8% |
| overlapping 20-day trades, 20y of 10 real assets | 30.5% | – | 5.4% |
| best of 50 strategies, 20y of 10 real assets | 83.5% | – | 4.2% |
The full study, including what broke on the way →
Catches lookahead
Rebuilt from its positions (no costs) the Sharpe is 13.26; with every
position taken one bar later, -0.44. Most of the edge needs same-bar
execution: check for lookahead (a signal using the bar's own close or
later) before trusting it.A cheating SPY strategy that every significance test passes (p = 0.0000). An honest crossover on the same data keeps its Sharpe a bar later.
On the TradingView chart
Paste pine/tokio_check.pine under any Pine v6 strategy and the verdict appears as a table on the chart: dependence-corrected p-value, the correction for how many variants you tried, the Deflated Sharpe Ratio, and the minimum track record. Free, no export needed.
Also: a research agent
pip install "tokio-ai[agent]" adds an LLM agent that pulls price history and SEC filings and routes every pattern claim through the same tests, correcting for everything asked in the session.