PolyEdge

Reconstructs per-wallet PnL from Polymarket's on-chain activity, scores trader skill net of variance, and backtests skill-weighted forecasts against market price.

Prediction markets are an unusually clean lab for this. Fills are wallet-attributable and markets resolve to 0 or 1, so both the skill estimates and the forecasts have exact labels — you can check whether a trader was right, not just whether they got paid. Data and backtests only, no execution.

Headline

	Result
PnL reconstruction	92.0% of positions match the exchange's own numbers exactly (829/901 across 51 wallets; 43 wallets reconcile perfectly)
Skill	Population raw edge +0.126 collapses to −0.095 after shrinkage. The apparent edge is mostly variance.
Signal	Skill-weighted flow does not beat the price. Brier 0.2088 vs the market's 0.2046; ROI −3.3% after costs, −1.3% without.

The last row is a negative result and it is the point of the exercise. "Copy the smart money" is an easy story to tell and a hard one to earn; measured point-in-time with costs, on this sample, it does not work.

What it does

1. Rebuild PnL from raw activity. Polymarket publishes every event but not what anyone made. Six event types have to be folded — buys, sells, and the SPLIT/MERGE/REDEEM operations that mint, burn and settle outcome tokens and never appear on the trade tape.

2. Validate against the exchange. Checked on token balances, not PnL. /positions.size is an unambiguous quantity we must agree on; PnL is a definition — theirs marks against average cost, mine accumulates cash flow — and matching a definition proves nothing. Matching a quantity proves the fold.

3. Score skill, net of luck. Edge per dollar risked, plus a calibration gap (hit rate minus the rate the trader's own entry prices imply), shrunk toward the population mean in proportion to evidence. Without shrinkage the top of any ranking is populated entirely by small samples.

4. Backtest point-in-time. Markets split by resolution date: skill is estimated only on markets resolved before the cut, so a wallet's edge never sees the market it is used to forecast. Entry pays the spread.

Results in detail

validation   120 wallets sampled, 22 skipped over row budget, 51 validated
             829/901 positions matched within 1 token  (92.0%)
             43/51 wallets reconciled exactly, 0 positions missing metadata

skill        285 wallets pulled, 274 scored, 8,361 resolved bets
             mean raw edge     +0.1261
             mean shrunk edge  -0.0954
             sd raw / shrunk    0.5759 / 0.2205   (shrinkage removes 62% of the spread)

backtest     28,929 activity rows, 4,953 markets, 16,039 resolved bets
             2,707 train markets / 1,805 evaluation markets
             123 wallets with a usable prior (10+ training bets)
             567 candidates, 104 traded
             ROI  -3.34% with costs,  -1.31% without
             hit rate 48.1%
             model Brier 0.2088  vs  market Brier 0.2046   (improvement -0.0042)

Raw outputs are in results/. Reproduce with:

python scripts/validate_pnl.py --wallets 120
python scripts/score_skill.py  --wallets 300 --max-rows 3000
python scripts/run_backtest.py

What the residual 8% is

Eight wallets account for nearly all of it. The interesting ones are split/merge arbitrage bots — 441 merges, 297 redemptions, zero trades — that merge far more than they split. Tokens are entering those wallets by a route /activity does not report. The fold's negative_token_balance invariant fires 208 times on one of them, which is it correctly announcing that rows are missing rather than quietly absorbing them.

This is a data-completeness limit, not an accounting bug, but it is unverified: confirming the gap is on-chain transfers would need chain data this project does not pull.

API findings

All four of these silently produce wrong answers rather than errors. Details in LOG.md.

• /activity caps offset at 5000, not the 10000 that /trades uses. Every wallet with more than ~5.5k lifetime events 400s. The iterator rotates its time window before the cap.
• gamma ANDs condition_ids with closed. Querying by condition id alone returns [] for a resolved market, with no error — indistinguishable from "no such market". Every lookup runs twice.
• outcomeIndex uses 999 as a sentinel, mostly on pre-2022 REDEEM rows. Only the winning token can be redeemed for value, so the winner is the only thing it can mean.
• SPLIT/MERGE/REDEEM carry an empty asset, so market metadata is a hard dependency of the fold, not an enrichment.

Setup

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python scripts/smoke_test.py     # hits all three live APIs
pytest                            # 86 tests
ruff check . && mypy              # strict over src and tests

Three public APIs, no auth: gamma (market metadata, resolution), data-api (trades, activity, positions, leaderboard), clob (price history). Storage is parquet batches under data/ with DuckDB views; SELECT DISTINCT absorbs the duplicate rows windowed pagination produces at seams.

Limitations

The signal doesn't work. Reported above rather than buried. Possible reasons, none of them established: 123 wallets with a usable prior is thin; entry price is a crude stand-in for the pre-trade mid; and the forecast pivots at the median trade time rather than at a fixed horizon before resolution, so "how early is the signal" is unmeasured.

Sampling is tape-weighted, not random. Wallets come from recent fills on closed markets, which over-samples active traders. Leaderboard wallets are deliberately excluded — they are selected on the very quantity being estimated, and their histories exceed the row budget anyway.

Wallets with more than ~3–6k lifetime events are skipped, not truncated. /activity pages 500 at a time with no larger limit, so full histories for the largest accounts cost hundreds of requests each. Truncating would produce wrong balances indistinguishable from a fold bug; skipping is visible and counted.

CONVERSION events are skipped. Rare enough here to be a non-issue — zero among the mismatched wallets — but negRisk-heavy portfolios would need them.

No execution modelling beyond a flat per-share cost. No book depth, no partial fills, no latency between observing flow and acting on it.