Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng · Read the paper

Hook: The paper evaluates whether mainstream large language models can follow defects through the back-and-forth process of real-world code review.

Summary: MCR-Bench is a benchmark for evaluating realistic, multi-round code review rather than treating review as a single static decision. It contains 2,269 real-world tasks across five programming languages, with detailed defect metadata and annotations tracking how defects evolve across review rounds.

Industry impact: The paper reports that mainstream LLMs have limited performance in both defect detection and review-state tracking, with results declining as interaction rounds increase. It also reports substantial variation by defect type and severity, with semantically complex and low-salience defects more likely to be missed.

Potential implications: For teams assessing automated code-review systems, single-round benchmark results may not reflect performance in iterative review workflows. The paper identifies cross-round temporal misalignment and inadequate long-range memory as failure mechanisms that should be examined in future evaluations.

Qianlong Lan, Vinothini Pandurangan, Anuj Kaul, Indranil Sanyal · Read the paper

Hook: A scanner can be highly accurate when it reaches a conclusion yet still provide limited protection if it often cannot complete its analysis.

Summary: The paper evaluates ModelScan, ModelAudit, and Fickling on a controlled benchmark of 170 Pickle- and PyTorch-focused artifacts spanning 145 specimen families. It separates coverage and analysis completion from accuracy, reporting that ModelAudit made definitive security decisions for all 135 labeled families, compared with 110 for Fickling and 67 for ModelScan.

Industry impact: For teams assessing AI model security scanners, the paper shows why F1 alone may not describe operational usefulness. The reported results also indicate that fallback tools can help recover detections when one scanner fails: for 48 malicious families where ModelScan did not complete analysis, ModelAudit and Fickling produced detections consistent with ground truth.

Potential implications: Evaluation programs should report judgment availability, analysis completion, unsupported outcomes, and incremental detection coverage alongside conditional accuracy. The paper also suggests distinguishing genuinely complementary scanners from redundant ones, since Fickling identified no unique true-positive families beyond those found by ModelAudit and ModelScan together.

Allison Zhuang, Santiago Aranguri · Read the paper

Hook: A model’s explanation of why it is being tested may matter more than simply whether it recognizes the evaluation.

Summary: The paper argues that eval-awareness—the recognition that a model is being tested—should not be treated as a single behavioral quantity. On Qwen3-32B using the FORTRESS dataset, capabilities-focused eval-awareness predicted compliance 24 to 46 percentage points more strongly than safety-focused eval-awareness across the tested steering conditions.

Industry impact: The findings suggest that safety evaluation pipelines relying on aggregate eval-awareness suppression rates may overlook distinctions that are relevant to compliance. The paper reports that a chain-of-thought-prefill intervention shifted compliance in the predicted direction for 10 of 11 prefills, supporting a causal link between framing and behavior.

Potential implications: Evaluation designers may need to distinguish capabilities-oriented and safety-oriented eval-awareness rather than measuring only an overall suppression percentage. The paper indicates that identical reported suppression rates can correspond to qualitatively different behavioral outcomes.

Shuyi Fan, Boyuan Deng, Mengyu Xu, Xinhong Xie, Chenyang Li, Hongyang Zhang · Read the paper

Hook: A reported bias in an LLM judge may sometimes reflect the rating scale rather than a change in the judge’s preference.

Summary: The paper argues that difference-in-differences applied to bounded rating scales can create an apparent interaction by combining genuine preference differences with unequal censoring at the scale limits. In a preregistered audit of a frozen pedagogy judge using 990 calls, the registered effect of a stated learner profile on scaffolding preference was null, while a nominally significant interaction was largely reproducible from severity shifts and the scale floor.

Industry impact: The findings are relevant to teams that use LLM judges to audit model behavior through matched comparisons and bounded ratings. The paper reports that one nominally significant interaction, measured as +0.378 points with p = 0.002, was not identified as preference because a construction with zero differential preference reproduced 79% to 85% of it.

Potential implications: The paper derives a closed-form mechanism for this artifact and shows that its contribution can be estimated from an audit’s own ratings. This suggests that evaluations using double differences on bounded scales should distinguish preference effects from attenuation caused by the scale’s floor and ceiling.

Jinghan Zhang, Fengran Mo, Zhiyu Chen, Xiaoyan Han, Kunpeng Liu, Chang-Tien Lu · Read the paper

Hook: A model that performs well in print English may still struggle to understand or produce Braille reliably.

Summary: BrailleBench is a benchmark for evaluating large language models on Braille comprehension across mathematics, commonsense, and multi-hop question answering in English and Braille Grades 1 and 2. The paper reports results from six representative LLMs and finds a persistent gap between print-English capability and Braille accessibility, with particular weaknesses in Grade 2 input comprehension and fully Braille-based requests.

Industry impact: For teams building AI tools intended for blind and deafblind users, the paper indicates that standard language evaluations may not capture Braille-specific limitations. BrailleBench offers 5,570 expert-reviewed instances and configurations covering Braille input, Braille output, and end-to-end interaction, providing a basis for more targeted evaluation.

Potential implications: The reported asymmetry between Braille understanding and expression suggests that accessibility testing should assess both directions rather than treating Braille support as a single capability. The paper's findings also point to the need for future Braille AI systems to address the additional fragility of Grade 2 comprehension and fully Braille requests.