Can an agent turn noisy evidence into useful predictions—and useful predictions into better decisions?
Forecast quality is only one part of an agent’s performance. JockeyBench follows what happens next.
JockeyBench places LLM agents in a historical replay of Japanese horse racing. Agents acquire evidence, build forecasting procedures, and revise them as outcomes become available.
Time-gated records, priced tools, and paired frozen-program controls make it possible to examine prediction, adaptation, and resource use separately.
How much predictive signal does the agent recover?
Does the predictive reward justify the research cost?
What returns and risks follow a fixed decision rule?
Change where LLM reasoning happens while keeping information cut-offs and outcome release consistent.
The LLM writes an executable forecasting program. The saved program queries evidence and produces rankings and win probabilities for each race.
Paired frozen control: the same initial program runs on the same races without LLM revision.
Figure 2 from the paper. Dashed arrows indicate next-day revision after outcomes are released.
Evidence becomes available under the historical replay clock.
Rankings and probabilities are committed before outcomes.
Feedback supports reflection and revision for the next day.
Archive coverage: 2015–2026. Evaluation uses 2024 races. Odds, expert ratings, and future outcomes are excluded from agent inputs. Identities are pseudonymized; external tools and program network access are disabled.
Results from the arXiv v1 evaluation, interpreted across shared historical outcomes.
Paired tests identify seven significant revision gains and two declines after Holm correction. The two highest-scoring configurations show no revision gain.
A strong initial predictor can rank highly without improving during evaluation.
Across four models, Program has higher mean betting profit and lower research fees than Adaptive Acquisition. Both workflows still lose on average.
Workflow comparisons are descriptive and have unequal trajectory counts.
Adaptive Acquisition underpredicts mean win frequency across all top picks, yet overpredicts it among Kelly-selected bets for all four models.
Aggregate probability bias can conceal errors on the decisions actually taken.
13 of 138 individual trajectories finish in profit under capped Kelly, while every configuration mean is negative. Profitable runs can rely on concentrated returns.
Repeated executions capture variability on fixed outcomes, not independent temporal replications.
Evaluate predictive competence, adaptation, resource use, and decision value separately.
Protocol, experiments, analyses, and appendices.
arXiv:2610.1958902 / CODE ↗Evaluation framework, configuration, and reproduction guidance.
HKUST-KnowComp / JockeyBenchRaw JRDB records are not distributed. Consult the repository for authorized data reconstruction and the current release scope. This page summarizes arXiv v1; repository results may use a newer evaluation cohort.
@misc{xie2026jockeybench,
title = {JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making},
author = {Zhongwei Xie and Xingsu Li and Jingqiao Yu and Zihao Wang and
Hong ChiTat and Zhaowei Wang and Weizhi Du and Yuxuan Liu and
Huihao Jing and Jiaxin Bai and Yangqiu Song},
year = {2026},
eprint = {2610.19589},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2610.19589}
}