What we tried, what broke, what held.
Credit assignment from recorded play is an old problem with a well-known trap: the data shows what people did when it worked, not what would have happened otherwise. These are our results against that trap, including the ones that went the wrong way.
-
negative
Ten-second credit rewards the brave
Our first credit layer compared each ten-second decision with what strong play would do in the same state. Across all moments it found a real signal: decisions before a drop were worse than shuffled ones in the same situations.
On the largest drops the sign flipped. Committing to an engagement looks good in recorded data because people commit when they are already winning, and the state did not show that. We logged it instead of tuning it away.
ρ +0.12
-
held
Engagements, not seconds
We moved the unit of analysis from fixed windows to engagements. What each unit did before an engagement (where it was, what it had ready, whether it could be seen) predicts how the engagement ends beyond the context around it.
The confound from R01 is still there in the most common case, "the engagement started around you". That is exactly the kind of call we send to people.
R² 0.154
-
held
The score knows who is better
Summed over engagements, the credit score orders four skill tiers correctly, and the intervals do not overlap. The ten-second layer could not do this: inside the same situation, strong and weak players chose the same things.
This is a sanity check, not a product. A credit score that cannot tell a top player from a beginner should not be trusted to explain a single loss.
4 / 4
-
partial
What each side could know
Units act on partial information, and a model that sees everything blames them for things they could not see. We reconstruct what each side could observe at each moment.
Actions that need a visible target agree with the reconstruction 97.9% of the time, and 88% at long range, where our model of sight is weakest. Training on that state barely moved the policy model (top-1 0.470 against 0.460). Useful, not decisive.
97.9%
-
open
Do experts agree with the model?
This is the question the company exists to answer, and we cannot answer it yet. Twelve verdicts are not enough to report agreement, and all twelve come from the founders.
We will publish agreement between reviewers and with the model when the numbers mean something, with the same dates and intervals as above.
12