Research

Hill Research Presents ESDA at KDD 2026 in Jeju, Korea

Hill Research |
Hill Research Presents ESDA at KDD 2026 in Jeju, Korea

Hill Research’s Charles Gao presented “From Tool Traces to Strategy Banks: Mining Failure Signatures for Budgeted Repository-Level Code Repair” — by Shuang Cao and Hill Research CTO Dr. Jack (Rui) Li, who contributed equally — at the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026), held August 9–13, 2026 at ICC Jeju in Jeju, Korea.

About KDD 2026

ACM SIGKDD is the premier international forum for research in data mining, data science, and machine learning. KDD 2026 ran August 9–13, 2026 at the International Convention Center Jeju, with workshops and tutorials on August 9–10 and the main technical program from August 11–13.

The Challenge

When an AI agent works on a failing repository, it generates a detailed record along the way: the tests it ran, the errors it encountered, and the order in which it tried things. Almost every system discards that record and starts the next task from the same fixed debugging loop.

The consequence is that the agent never gets better at debugging. It only gets another attempt. In a repository-level setting this is expensive in a way that compounds, because every evaluator call means a real build and a real test run — measured in minutes and compute cost, not tokens.

Introducing ESDA

ESDA keeps the record. It mines execution traces into structured failure signatures, then builds a bank of reusable debugging strategies out of them. Strategies are stored in typed slots rather than as fixed scripts, so components of one strategy can be recombined with components of another as new tasks arrive, and the bank evolves as it accumulates evidence.

A cost-aware learning-to-rank stage then decides what to try first, promoting the strategies most likely to succeed within the first few attempts. This is the part that converts a strategy bank into a budget saving: ranking for early success is what reduces the number of expensive evaluator calls, rather than simply improving the final answer.

ESDA: Mining Failure Signatures for Budgeted Repository-Level Code Repair

“From Tool Traces to Strategy Banks: Mining Failure Signatures for Budgeted Repository-Level Code Repair” — Shuang Cao and Dr. Jack (Rui) Li (equal contribution).

DOI: 10.1145/3770855.3817699

Results:

  • RepoBench success rate: 84.7% — an improvement of 38.7 percentage points over the strongest baseline
  • Evaluator calls required: 4.0x fewer
  • Strategy retriever ranking quality: 53.4 MRR, with a rank correlation of 0.89
  • Gains retained when transferred to a frozen setting: 71–74%
  • Integrity audit for reward hacking: 0.7%

The transfer result is the one the team finds most notable. What the agent learned about failing was not tied to a single language or build system, which suggests failure signatures capture something general about debugging rather than something specific to one stack. The integrity audit matters for the same reason a benchmark number alone would not: it establishes that the gains come from genuine repair rather than from exploiting the evaluator.

Relevance to TriClick: TriClick’s agents generate statistical code, run it against clinical datasets, and must produce outputs that hold up to regulatory scrutiny — a setting where each validation cycle is costly and where silent failure is unacceptable. A system that learns from its own failure traces, ranks its next attempt by expected cost, and can prove it is not gaming the evaluator maps directly onto that problem.

Why It Matters

Most work on AI coding agents focuses on the patch — the diff the agent finally produces. ESDA moves the unit of reuse up a level, to the strategy that found the patch. Treating debugging experience as a retrievable, rankable, evolvable asset is a different bet from making the underlying model larger, and it is one that pays off specifically under a budget.

This continues Hill Research’s publication record at premier venues, following five papers accepted at ACL, SIGMETRICS, MLSys, and JAMIA earlier in 2026 and three presentations at the AAAI 2026 SPARTA workshop in January.

Learn More