Backtest
Does it beat the teams?
For every class, the model only learns from the drafts before it, and its settings are tuned on those earlier drafts too. Then it ranks the new class, and I compare that with the order teams actually picked in.
What changed in V4
- Honest numbers. V3's settings were picked on the same backtest it reported. Now every setting is tuned only on classes before the one being tested, so the numbers below are lower than V3's but they're real.
- Fixed how busts are counted. V3 gave a pick who never played a 0 but a rotation player with a bad season a negative number, so never playing beat playing badly. Now a season below replacement counts as replacement. That alone took the draft order's Immediate correlation from 0.06 to 0.42, so V3's big Immediate win was mostly that bug.
- Two per-possession targets that don't reward a pick for just getting minutes: Peak BPM, and my own plus-minus (RAPM) built from every NBA lineup since 2007-08.
- A ridge regression blended with LightGBM, and college stats from the last three seasons instead of one, plus shot types (rim, mid range, dunks, assisted) from 2013-14 on.
- Ranges that are checked: 80% ranges calibrated on earlier classes, and the share that held is shown below for every class.
Before and after
Average over the test classes. Spearman and NDCG@10 are inside each class; top 10 hits are out of the real top 10; coverage is how often the real outcome landed inside the 80% range.
Rank correlation with real outcomes, by class
Spearman correlation between each ranking and what the class actually did. Higher is better; 1 would be a perfect ranking.
Do the ranges hold?
Share of each class whose real outcome landed inside its 80% range, for the stats-only model, the model with the slot, and a range built from the pick number alone.
Top 10 hits
Of the ten best players in each class, how many the model had in its top ten, against picks 1 to 10.
Calibration
Every backtest projection, split into ten equal groups. Dots on the dashed line mean the model's numbers are the right size, not just the right order.
Every projection
Each dot is a player, projected against real. Hover for names.
What I tried and kept or dropped
Each idea against the version without it, averaged over all three targets' test classes. With 11 to 15 classes per target, a change smaller than about 0.03 in Spearman is noise, so an idea had to help on average and in most classes to stay.
The bust fix
Draft order's own score under the old and new way of counting seasons below replacement, and where players who never played landed inside their class (0 is last, 1 is first).
What the model leans on
Average size of each feature's SHAP value in the final blend (LightGBM's tree SHAP plus the ridge's exact linear SHAP, mixed by the blend weight). In class percentile points: 3 means the feature moves a typical player about 3 places in a 100-player class.
Class by class
Spearman correlation with the real outcome, then top 10 hits and range coverage.