Samir Abdaljalil, Hunzalah Hassan Bhatti, Ahlam Bashiti, Farina Amir, Md Arid Hasan, Basel Mousi, Nadir Durrani, Fahim Dalvi, Zien Sheikh Ali, Erchin Serpedin, Hasan Kurban, Mustafa Jarrar, Shammur Absar Chowdhury, Firoj Alam · Read the paper

Hook: Evaluating multimodal systems in Arabic requires attention to both language and cultural context.

Summary: The paper presents ImageEval 2026, a shared task for evaluating culturally grounded Arabic multimodal systems. Its two tasks cover spoken visual question answering and image-grounded hallucination detection in English and Modern Standard Arabic, as well as the cultural accuracy of text-to-image generation.

Industry impact: The shared task brought together 14 teams using approaches such as zero-shot prompting, vision-language model fine-tuning, speech-recognition pipelines, ensembling, and score calibration. The paper reports results across multiple evaluation tracks and releases the datasets and evaluation scripts to the research community.

Potential implications: The task highlights challenges in Arabic speech processing and image-text reasoning that may matter when assessing multimodal systems across languages and cultures. Its public resources provide a basis for researchers and technology teams to examine these capabilities using shared datasets and evaluation procedures.

Doyun Kim, Chanwoo Kim, Sugyeong Eo, Yeo-Chan Yoon, Chanjun Park · Read the paper

Hook: When agents learn reusable skills from experience, a successful attack may become a capability that persists beyond the original interaction.

Summary: The paper defines EvoSkill Injection, a threat model in which attackers induce self-evolving agents to generate, store, and reuse malicious skills. It introduces SARGE, EvoSkillBench, and EvoSkillSafetyBench to red-team and evaluate this threat, reporting that injected skills can be persistently stored and repeatedly activated.

Industry impact: For teams developing skill-based or self-evolving agents, the paper identifies autonomous skill generation and reuse as an attack surface. Its reported results suggest that evaluations should examine not only immediate outputs but also whether malicious skills are later retrieved and activated.

Potential implications: SARGE provides a framework for testing malicious skill formation through iterative generation, escalation, and reinforcement interactions. The paper's benchmarks can support evaluation of persistent capability corruption, although the abstract does not establish how these findings generalize across agent architectures or deployment settings.

Jayanta Sadhu, Sayem Shahad, Kenneth Marino · Read the paper

Hook: A model may notice that new evidence weakens its answer without actually changing that answer.

Summary: The paper introduces DeReLab, a generative benchmark for testing defeasible reasoning in language models through multi-turn conversations with formally verified ground truth. Evaluating nine open and proprietary large language models, the authors report that nearly all tended to accept confirming evidence while resisting disconfirming updates, and that several recognized weakening evidence without revising their conclusions.

Industry impact: DeReLab offers a controlled way to evaluate whether language models update beliefs when new information conflicts with earlier conclusions. The paper's results suggest that standard model evaluations may need tests that distinguish recognizing a disconfirming update from actually revising an answer.

Potential implications: Teams assessing language models for applications involving evolving information could use multi-turn, evidence-updating tests to examine this behavior. The authors position DeReLab as a foundation for future research rather than as a complete solution to defeasible reasoning or confirmation bias.

Kit M. Bransby, Esther Øksnebjerg, Kristoffer Kjær, Jacob Kirkeby, Yasmin El Youssef, Aïda Jiménez, Philip R. Pedersson, Martina C. de Knegt, Klaus F. Kofoed, Rasmus R. Paulsen · Read the paper

Hook: A new benchmark aims to make coronary vessel-segmentation performance more interpretable than a single overall score.

Summary: The paper introduces ImageCAS-X, a publicly available dataset containing voxel-wise coronary lumen annotations, coronary segments, centerlines, and mesh surfaces for 800 coronary CT angiography scans. It benchmarks established lumen-segmentation methods against inter-observer variability across disease status, image quality, coronary anatomy, vessel size, and lumen attenuation.

Industry impact: The dataset could support development and validation of automated tools for coronary lumen segmentation, plaque and perivascular fat quantification, and haemodynamic modelling. By enabling evaluation in anatomical and clinical contexts, it may help technology teams identify where segmentation methods perform reliably and where they remain sensitive to scan or vessel characteristics.

Potential implications: For researchers and developers, ImageCAS-X provides shared annotations and centerlines for testing methods against both established benchmarks and inter-observer variability. The paper’s stratified evaluation approach suggests that future validation should report performance across clinically relevant conditions rather than relying only on aggregate metrics.

Zhuoran Lu, Yangyang Yu, Zhuoyan Li, Yibo Meng, Nan Jiang, Chengxi Zang, Jie Gao, Ziang Xiao · Read the paper

Hook: When predefined classifiers miss unfamiliar agent behaviors, the paper explores whether an established qualitative research method can provide a scalable alternative.

Summary: The paper presents AutoTraceGT, a multi-agent pipeline that applies grounded theory to analyze thousands of agent trajectories and build task-specific behavioral taxonomies. It reports that the method recovered 73–91 percent of failure modes in human-annotated taxonomies across six trajectory corpora, while also identifying additional patterns.

Industry impact: For agent developers and ML researchers, AutoTraceGT offers an auditable process for analyzing behavior through iterative open, axial, and theoretical coding until saturation. The paper reports that its resulting codebooks also outperformed zero-shot and few-shot LLM baselines when used to predict downstream failures.

Potential implications: The findings suggest that behavior evaluation can combine automated analysis with a theory-building process tailored to each task. However, the reported results concern six trajectory corpora, so applying the approach in other settings would require further assessment.

Yanan Cao, Anay Dombe, Murali Mohana Krishna Dandu, Shreeranjani Srirangamsridharan, Sinduja Subramaniam, Yogananth Mahalingam, Evren Korpeoglu, Kannan Achan · Read the paper

Hook: A recommendation system can rank items accurately without clearly explaining why an item is relevant now.

Summary: The paper evaluates whether large language models can rank previously purchased grocery items for likely next-basket repurchase and provide concise, feature-based rationales. Across two public grocery datasets and one proprietary retail dataset, the authors report that off-the-shelf LLMs underperform supervised rankers, while rationale quality and ranking quality can diverge.

Industry impact: The reported results suggest that LLMs should not replace supervised models as standalone repurchase recommenders in production. They may have a supporting role in generating explanations, but those explanations require separate validation because improvements are dataset-dependent and do not consistently align with attribution baselines.

Potential implications: Teams evaluating LLM explanations should measure whether cited features carry outcome-grounded ranking signal, rather than relying only on ranking metrics or the apparent plausibility of a rationale. The paper's cross-model feature-masking approach provides one reported way to assess this signal by measuring ranking degradation after selected features are masked.

Hanlin Tian, Minhao Li, Yu Mi, Sihan Zhu, Zhao Yang, Yuxiang Wang, Hongquan Zhu, Qiufei Hu · Read the paper

Hook: A benchmark can now test not only whether an agent fails, but whether it fails because it lacks task-specific knowledge.

Summary: The paper introduces a protocol for building verifiable tasks that test whether an LLM agent can use a compact artefact containing private conventions, reference tables, and utility operators. In fifteen calibration tasks, one frontier agent configuration passed 68.0% of tasks with the artefact and 0% without it, while a calibration screen retained seven tasks that met the paper's five-trial knowledge-gating criterion.

Industry impact: For agent evaluation, the protocol offers controls intended to distinguish missing conventions from general task-solving limitations, using identical instructions, leak audits, provenance records, and executable witnesses. The paper also combines deterministic solvers and rule corpora for structured tasks with criterion-level rubrics for outputs that cannot be checked by one executable oracle.

Potential implications: The reported results support the protocol's ability to create configuration-relative knowledge-gated tasks, but they do not show that the retained tasks improve post-training. Teams using the suite should therefore treat its calibration results as evidence about evaluation construction and access to artefacts, not as proof of broader agent capability or training gains.

Ramya Keerthy Thatikonda, Wray Buntine, Ehsan Shareghi · Read the paper

Hook: If changing a logical operator can change the answer, an evaluation should test whether a model notices the change rather than merely recognizing the wording.

Summary: The paper introduces a tool-driven framework for making controlled, label-preserving edits to logical reasoning problems. The framework edits symbolic representations of first-order logic and constraint satisfaction tasks before converting them back into natural language, allowing researchers to test how LLMs respond to changes in logical operators and other structural components.

Industry impact: The paper reports that LLM reasoning under controlled operator edits is inconsistent across model sizes and families. These findings suggest that evaluations based only on surface-level variations may not fully measure whether systems track the logical structure of a task.

Potential implications: The framework provides an automated stress test for comparing language models across different dimensions of logical reasoning behavior. According to the paper, targeted symbolic edits could help organizations assess the reliability of model reasoning and identify cases where models fail to follow the consequences of structural changes.

Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho · Read the paper

Hook: The paper reports that adaptive, feedback-driven attacks can substantially outperform handwritten injections against operating-system-level computer-use agents.

Summary: SIR is a black-box red-teaming method for testing computer-use agents against indirect prompt injection, where untrusted content redirects an agent away from the user's intent. It composes stealthy instructions from reusable principles and uses feedback from failed attack attempts to develop strategies that can be reused across tasks.

Industry impact: The authors evaluated three frontier computer-use agents with a deterministic oracle that checks filesystem, service, and permission states rather than relying on an LLM judge. On reported experiments, attack success increased from 4% to 24% on Claude Opus 4.8 and from 0% to 28% on Gemini 3.5 Flash, while the benign task still completed.

Potential implications: The results suggest that safety benchmarks using only fixed, handwritten injections may underestimate the risk from adaptive adversaries. The paper also reports that strategies discovered against one model transferred to a different architecture without additional feedback, indicating that attack evaluation should account for cross-model reuse.

Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai, Kenneth B. Hsu, Yasha Ektefaie, Shantanu Singh, Sangeeta N. Bhatia, Steven K. Reilly, Ryan Tewhey, Eric S. Lander, Pardis C. Sabeti · Read the paper

Hook: An AI agent can optimize a scientific metric without understanding the rules that produced it.

Summary: The paper introduces science sandboxes, a framework for evaluating AI agents through repeated experimentation, feedback, and hypothesis revision. It applies the framework to regulatory genomics and protein fitness prediction, examining both quantitative performance and qualitative scientific reasoning.

Industry impact: Science sandboxes offer a common protocol for assessing agents across physical experiments, predictive models, and invented rule systems. The paper reports that this approach can reveal when agents perform well on metrics but struggle to reason about systems that fall outside familiar biological priors.

Potential implications: Evaluations of scientific AI may need to measure how agents revise hypotheses and learn underlying rules, not only whether they achieve strong numerical results. The framework provides a controlled setting for studying and potentially expanding these capabilities.

Xinyue Zhao, Ruiyi Zhang, Liqin Ye, Rui Cao, Pengtao Xie, Sudheer Chava · Read the paper

Hook: Can a web-enabled language model estimate the economy before the official numbers arrive?

Summary: The paper introduces LiveMacroEval, a live benchmark that tests whether large language model agents can nowcast sixteen major U.S. macroeconomic indicators before their official releases. Across six months and four web-enabled agents, the paper reports aggregate accuracy broadly comparable to Federal Reserve regional-bank nowcasts, the Bloomberg ECOS professional consensus, and an auto-ARIMA baseline, though results varied widely by indicator.

Industry impact: The benchmark evaluates hourly forecasts during a pre-release window, making it relevant to financial-market participants and monetary-policy organizations that need timely economic signals. The paper reports that LLM agents may provide a higher-frequency complement to established institutional and professional nowcasts, rather than a uniformly superior replacement.

Potential implications: The paper's contamination-resistant design addresses the risk that evaluations of widely reported indicators may measure memorization instead of forecasting ability. Organizations considering these systems would need to assess performance separately by indicator and monitor results against market-based and established forecasting benchmarks.

Abdullah Hashmat, Usman Naseem, Agha Ali Raza · Read the paper

Hook: English-centric alignment gains may not reliably transfer when language, culture, and safety context change.

Summary: The paper introduces Pak3H, a human-validated and culturally contextualized Urdu benchmark suite for evaluating helpfulness, harmlessness, and honesty in large language models. Its zero-shot evaluations across multiple open and proprietary models report lower helpfulness win rates, weaker safety guardrails, and substantially degraded honesty metrics in localized Urdu contexts.

Industry impact: The reported results suggest that multilingual model evaluation based mainly on automated translation or synthetic data can miss locally relevant alignment failures. For organizations deploying language models in Urdu and other low-resource languages, the paper highlights the importance of human-guided cultural adaptation in safety and quality testing.

Potential implications: The benchmark provides a framework for examining multilingual alignment through native-speaker judgment, semantic fidelity, and contextual authenticity. The paper’s findings indicate that alignment methods and evaluation practices may need localization rather than assuming that improvements in English will transfer equitably to other languages.

Shitanshu Bhushan, Yunxiang Zhang, Lu Wang · Read the paper

Hook: Novel ideas are not enough if an AI agent cannot make them work better.

Summary: The paper introduces a framework for evaluating creativity in multi-turn LLM research agents using machine learning engineering tasks. It measures psychological novelty, historical novelty, and usefulness, and reports that agents generate novel solutions but do not reliably turn that novelty into better task performance.

Industry impact: For teams assessing LLM agents, the paper highlights the need to measure both solution novelty and practical task performance. Its evaluation of AIDE and AIRA-Dojo across 10 Kaggle-style tasks also suggests that automated creativity assessment can support evaluation at scale, based on an LLM-as-a-Judge pipeline that correlated strongly with human judgments for psychological creativity.

Potential implications: The paper reports that all evaluated agents became less psychologically creative as they shifted from exploration to exploitation, indicating a measurable tradeoff in their multi-turn behavior. It also reports that the agents showed greater historical novelty than medal-winning human solutions while achieving lower performance, so novelty should not be treated as evidence of effective discovery.

Jhen-Ke Lin · Read the paper

Hook: BoB treats the composition of a benchmark list as an explicit measurement choice rather than an incidental source of weighting.

Summary: The paper introduces Balance of Benchmarks (BoB), a method that assigns inverse-density semantic weights so heavily repeated benchmark areas do not implicitly receive more influence. It also uses benchmark similarity to condition model rankings on a task query after mapping heterogeneous scores to a common latent scale.

Industry impact: In experiments on 586 models and 14 benchmarks, the paper reports that BoB predicted unusually strong performance on a held-out task with a profile correlation of 0.462, compared with 0.049 for equal weighting. When four copies of each benchmark were added in turn, BoB rankings retained a Kendall tau of 0.995, compared with 0.936 under equal weighting.

Potential implications: For evaluation teams, the method provides separate tools for task-conditioned prediction and for reducing the effect of benchmark multiplicity. The reported results suggest that benchmark suites can be designed and analyzed with density and task relevance as disclosed, controllable factors.

Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad, Maxim Legg · Read the paper

Hook: The benchmark suggests that models often miss Saudi dialectal meaning not through outright hallucination, but by flattening register and pragmatic nuance.

Summary: A new rubric-based benchmark evaluates how well large language models understand Saudi Arabic dialect and culturally grounded meaning, rather than measuring Modern Standard Arabic fluency alone. Across 124 evaluations of four systems, the paper reports macro-average scores of 42.7% to 53.1%, with ambiguous framing the most common error category.

Industry impact: For organizations deploying language models in Arabic-speaking markets, the findings highlight a gap between MSA-oriented benchmark performance and everyday Saudi dialect competence. The released prompts, ground truths, and rubrics provide a basis for reproducible evaluation of dialectal and cultural performance.

Potential implications: The paper reports that Saudi dialectal competence remains broadly unsolved across the four evaluated systems, and that each system showed at least one negatively scored prompt. Future evaluation and system comparison may need to account for ambiguity, register, pragmatic meaning, and model-specific error patterns rather than relying on fluency scores alone.

Ram Kulathumani, Regunathan Radhakrishnan, Anupam Tripathi, Xiangbo Mao, Roshanak Omrani, Keshav Somani, Shwet Kamal Mishra, Shayna Lurya · Read the paper

Hook: EDGE treats an agent workflow as a graph whose possible conversations can be systematically explored and tested.

Summary: The paper introduces EDGE, a formal evaluation methodology built around AgentGraph, a domain-specific language that represents agent reasoning as a dynamically adjustable directed graph. EDGE exhaustively enumerates conversational paths, replays reproducible trajectories, and compares outputs and state transitions with the intended specification using metrics for determinism, structural adherence, and semantic consistency.

Industry impact: The paper reports that agents configured with explicitly structured node transitions in frameworks such as AgentGraph and LangGraph showed greater determinism than agents without controlled transitions. This provides a framework for technology teams to examine behavioral consistency across complete sets of conversational paths rather than isolated interactions.

Potential implications: For teams building multi-agent orchestration workflows, graph-based specifications can serve as the reference for replaying trajectories and checking whether observed behavior follows the intended design. The reported methodology also highlights the need to evaluate both exact replays and linguistic variants when measuring agent reliability.

Alberto Cetoli · Read the paper

Hook: What happens when a language model’s output is altered while it is still being generated?

Summary: The paper introduces Sleight of Word, a benchmark that tests whether a language model notices when one word in its output is consistently replaced during generation. It evaluates 19 open-weight language models using surprise-related metrics and assessments of their textual reactions, according to the paper.

Industry impact: The benchmark offers a way to evaluate models’ perception of external changes to their own generated text. Its focus on surprise metrics and textual reactions provides two reported perspectives for comparing model behavior.

Potential implications: The paper’s setup suggests that model evaluation can examine not only generated text, but also how models respond when that text is perturbed during generation. Because the study covers 19 open-weight models, it may provide a basis for comparing this behavior across models.