This week's research highlights advances in multi-agent systems, safety monitoring for diffusion models, and scalable approaches to multimodal continual learning. Key papers explore the challenges of system scaling in agentic AI, the importance of proper benchmark evaluation, and novel techniques for uncertainty quantification and agent coordination.
Research this week focuses on critical developments in agent frameworks and system design for AI. Papers examine how to scale beyond model improvements to address fundamental system-level challenges in agentic AI, with findings that the 'harness' - the structured execution layer around foundation models - is as important as the models themselves. New safety monitoring approaches for diffusion LLMs demonstrate effective dynamic resource allocation, while advances in multimodal continual instruction tuning show how plugin architectures can accelerate method development. Additionally, studies on benchmark auditing reveal widespread issues in current evaluation practices, and research on agent coordination highlights the importance of structure-guided orchestration for complex multi-agent tasks. The papers also cover uncertainty quantification in activation oracles, proactive agent architectures, and the critical role of proper evaluation methodology in urban representation learning.
AI Model Releases
TechCrunch Disrupt 2026 Early Bird ticket rates end May 29
Citing Gandalf, Pope Leo says we must "disarm" AI - Ars Technica
Manage your consent preferences.
AI Tooling
Towards self-driving codebases · Cursor
Cursor has released a preview of its multi-agent research capabilities for creating self-driving codebases. The feature allows developers to set up automated workflows that can handle code generation, testing, and deployment without constant human intervention. The system uses multiple AI agents working in coordination to manage different aspects of software development. Early adopters can experiment with the preview to see how automated codebase management might reduce manual overhead in development processes.
Why it matters: This preview signals a significant step toward autonomous software development, potentially reducing the manual effort required in coding workflows. It represents a move toward more sophisticated AI-assisted development environments that could fundamentally change how teams approach software creation and maintenance.
Research Papers
MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
arXiv: 2605.26114
MobileGym represents a significant advancement in mobile agent research by providing a browser-hosted, lightweight simulation environment that maintains interaction fidelity without replicating proprietary backends. The platform's ability to capture, configure, and compare full environment states as structured JSON enables deterministic evaluation and dense reinforcement learning rewards, addressing critical limitations in previous mobile agent research.
The system's layered state model and declarative task-definition framework make it practical to scale state programmability and task creation, while the single programmatic judging mechanism delivers both deterministic evaluation verdicts and dense RL rewards. This approach allows for the creation of 416 parameterized task templates across 28 apps, with deterministic judges and structured AnswerSheet protocol that avoids free-text matching failures.
In a Sim-to-Real case study, GRPO on Qwen3-VL-4B-Instruct achieved a +12.8 percentage point gain on the 256-task test set, and real-device execution retained 95.1% of the simulation-side training gain. This demonstrates that MobileGym's simulation platform can effectively bridge the gap between simulation and real-world performance, making it a valuable tool for developing robust mobile agents.
Key insight: MobileGym enables scalable, verifiable mobile GUI agent research through deterministic state-based judging and parallel rollouts, demonstrating significant gains in real-world transfer performance.
From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
arXiv: 2605.26112
The paper introduces the concept of 'scaling the harness' as a critical shift from model-centric to system-centric evaluation in agentic AI. Rather than treating agents as final-task success metrics, the authors argue that agent performance emerges from the interaction among multiple components: foundation model, memory substrate, context constructor, skill-routing layer, orchestration loop, and verification-and-governance layer.
The authors identify three core bottlenecks in scaling the harness: context governance, trustworthy memory, and dynamic skill routing, along with orchestration and governance mechanisms. This framework provides a concrete roadmap for future research, emphasizing that evaluation should measure trajectory quality, memory hygiene, context efficiency, communication fidelity, verification cost, and safe evolution over time rather than just one-shot task success.
CheetahClaws, the Python-native reference harness developed in the paper, serves as a concrete implementation that demonstrates how system design can be as important as model improvements. The comparison with Claude Code and OpenClaw shows that system-level design choices significantly impact agent performance, supporting the claim that future progress in agentic AI will depend as much on system design as on stronger foundation models.
Key insight: Future progress in agentic AI will depend as much on system design as on stronger foundation models, with the need to scale the 'harness' - the structured execution layer around foundation models - rather than just model scaling.
Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World
arXiv: 2605.26086
Claw-Anything addresses a critical gap in current agent capabilities by expanding agent context along three dimensions: long-horizon activity histories, interdependent backend services, and integrated GUI and CLI interaction across multiple devices. This comprehensive approach reflects the reality that future personal assistants need access to anything relevant in the user's digital world, not just narrow slices.
The benchmark's use of multi-round event injection to simulate months of user activity produces complex world states and realistic noise, including irrelevant events and conflicting signals. This design ensures that agents must reason over rich contextual environments while remaining robust to such noise, making it a more realistic test of agent capabilities than previous benchmarks.
The experimental results show that GPT-5.5 achieves only 34.5% pass@1, substantially below prior benchmarks, underscoring the gap between current agent capabilities and the demands of always-on personal assistance. The release of an automated data-generation pipeline that yields 2,000 training environments and improves the base model by 23.7% demonstrates the utility of scalable data infrastructure for advancing this field.
Key insight: Current LLM agents fall significantly short of the demands of always-on personal assistance, with GPT-5.5 achieving only 34.5% pass@1 on a comprehensive benchmark that simulates months of user activity.
VeriTrace: Evolving Mental Models for Deep Research Agents
arXiv: 2605.26081
VeriTrace addresses a critical problem in deep research agents: the contamination of intermediate representations by mixed-quality information and error propagation. The authors argue that without explicit regulation, the intermediate layer becomes easily corrupted, making model scale substitute for absent regulation rather than improving performance.
The framework implements three regulatory loops: interpretive update, deviation feedback, and schema revision, creating a cognitive-graph framework that explicitly manages mental model evolution. This approach directly addresses the issue of error propagation in complex reasoning tasks where intermediate representations are crucial for downstream decisions.
Using matched Qwen3.5-27B backbones, VeriTrace improves over the strongest matched baseline by 4.22 pp on DeepResearch Bench (DRB) Insight (1.49 pp Overall) and by 5.9 pp Overall win rate on DeepConsult. With Config-DeepSeek, it achieves the strongest reproducible open-source result on DRB, demonstrating that explicit regulation of mental model evolution can significantly enhance agent performance in complex reasoning tasks.
Key insight: VeriTrace's explicit regulatory loops for mental model evolution significantly improve performance on deep research tasks by continuously aligning task understanding with reality.
Retrying vs Resampling in AI Control
arXiv: 2605.26047
The paper provides a critical analysis of AI control mechanisms, specifically contrasting retrying versus resampling approaches. While retrying reduces honest suspicion scores, it allows untrusted models to exploit monitor rationale to construct sneakier attacks, negating safety gains. This finding challenges the common assumption that retrying is a safe approach to AI control.
The study demonstrates that resampling - drawing multiple samples from the same context - is more effective than retrying because it does not leak exploitable information. By drawing five samples per step and auditing on the maximum suspicion score, safety improves from 61% to 71% at a 0.3% audit budget, with no cost to usefulness. This approach also shows that selectively resampling only suspicious steps recovers 6.2 percentage points of the gain while drawing only 10% as many extra samples.
The findings contradict previous work on resampling, showing that auditing based on maximum across resampled suspicion scores outperforms using the minimum, and that executing the least suspicious sample gives only a small empirical safety gain. These results provide concrete guidance for designing safer AI systems and highlight the importance of carefully considering the design choices in AI control mechanisms.
Key insight: Resampling with maximum suspicion score auditing outperforms retrying in AI control, achieving 71% safety at 0.3% audit budget while maintaining usefulness, whereas retrying can be exploited by untrusted models.
CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities
arXiv: 2605.26036
CityRep addresses a fundamental problem in urban representation learning: the lack of rigorous evaluation protocols that support cross-location generalization and fair comparison. The benchmark introduces a spatial unit-agnostic evaluation framework that supports heterogeneous urban representations through a standardized alignment module, along with block-based spatial splits to mitigate spatial leakage.
The evaluation of 11 representative urban representation models shows that performance is highly sensitive to the split protocol, with random splits inflating scores and altering model rankings. This finding underscores the importance of proper evaluation methodology in urban AI research, where spatial leakage can lead to misleading conclusions about model capabilities.
The benchmark's extensible multi-city, multi-task suite spanning 8 cities and 8 tasks across regression, classification, and distribution prediction provides a comprehensive testbed for evaluating urban representation models. The release of datasets, evaluation pipelines, and diagnostic tools makes CityRep a reproducible benchmark that will facilitate fair comparison and support future research in urban representation learning towards urban foundation models.
Key insight: CityRep's spatially structured splits reveal that performance is highly sensitive to split protocols, with random splits inflating scores and altering model rankings, emphasizing the need for generalization-aware evaluation.
CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
arXiv: 2605.26029
CausaLab introduces a novel environment for evaluating interactive causal discovery by LLM agents, distinguishing between predictive success and causal understanding. Unlike prior evaluations that only assess whether agents can solve problems using causal evidence, CausaLab evaluates whether agents' answers are supported by correct hypotheses about underlying causal mechanisms.
The experiments reveal a fundamental gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge F1, indicating that predictive success does not necessarily translate to causal understanding. Mixed observation-intervention strategies improve structural fidelity, but even strong agents struggle to design informative interventions, showing that causal reasoning remains a significant challenge for LLMs.
The study identifies premature stopping as a major weakness of agents and demonstrates that asking models to verify consistency between their hypothesis and past data can help mitigate this issue. This finding suggests that current LLM agents lack the ability to effectively plan interventions and maintain consistent causal reasoning throughout the discovery process, highlighting the need for better causal reasoning frameworks in AI systems.
Key insight: CausaLab reveals a persistent gap between prediction and mechanism recovery in LLM agents, showing that even strong agents struggle to design informative interventions and often fail due to premature stopping.
Neural Scalable Symbolic Search Framework for Complex Logical Queries with Multiple Free Variables
arXiv: 2605.25985
NS3 addresses a fundamental challenge in Complex Query Answering (CQA) by providing a budgeted framework that approximates joint ranking without enumerating the full search space. This is particularly important for EFO_k queries with k > 1, where marginal rankings are poor proxies for true joint ranking, and the computational complexity grows exponentially with k.
The framework's approach of answering marginalized sub-queries to obtain candidate sets, merging multiple free variables into hypernodes, and progressively reducing queries to lower dimensions with dynamic budget control represents a significant advancement in scalable query processing. This approach allows for efficient handling of complex logical queries while maintaining strong marginal accuracy.
The evaluation across three standard KG datasets shows that NS3 substantially improves joint ranking performance while retaining strong marginal accuracy, demonstrating the practical utility of the approach. The release of a joint-ranking benchmark that extends existing EFO_1 datasets to k=3 enables systematic evaluation of multi-variable queries, providing a foundation for future research in complex logical reasoning.
Key insight: NS3 framework enables joint ranking of complex logical queries with multiple free variables without enumerating the full search space, achieving substantial improvements in joint ranking performance.
LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation
arXiv: 2605.25964
LECTOR addresses the challenge of automatic scientific paper writing, particularly the Introduction section, which demands not only linguistic fluency but also logical soundness and verifiable faithfulness. The framework formulates Content-Conditional Introduction Generation (CCIG) as a task requiring grounding in the paper's core evidence, moving beyond text generation to reasoning and structuring.
The approach constructs a logic-reasoning graph from the paper's main body to serve as a verifiable logical blueprint, then employs a Logic-Expression Co-Rewarding mechanism to jointly optimize for both structural fidelity and narrative quality. This dual optimization approach ensures that generated introductions are not only well-written but also logically consistent with the scientific reasoning.
Extensive experiments on a dataset from Nature Communications papers show consistent improvements in both logic fidelity and introduction generation quality metrics, including Graph Quality (+26.7%), Citation Quality (+8.6%), and Paper Consistency (+3.3%). The framework's ability to strictly follow scientific logic while adding high-quality citations demonstrates a significant step forward in AI-assisted scientific writing.
Key insight: LECTOR's Logic-Expression Co-Reinforcement Learning framework achieves significant improvements in both logic fidelity and introduction generation quality by strictly following scientific logic and adding high-quality citations.
Explore Before You Solve: The Speed--Depth Trade-off in Epistemic Agents for ARC-AGI-3
arXiv: 2605.25931
The paper provides a critical benchmark critique showing that current interactive reasoning benchmarks fail to measure the exploration they claim to require, as every one of the 25 public ARC-AGI-3 games is reachable through non-intelligent strategies. This reveals that the public evaluation set cannot discriminate intelligent exploration from trivial heuristics, making the private 55-game evaluation the only genuine intelligence test.
AERA's three-phase framework (EXPLORE/VERIFY/PLAN) achieves RHAE=0.2116 (4/25 solved) on the 25 games with Qwen2.5-0.5B, while random and no-explore baselines score 0.0000. The framework formalizes a Speed-Depth trade-off, showing that under convexity assumptions, RHAE's quadratic form emerges as a second-order penalty for deviating from the Pareto frontier between action efficiency and information gain.
The findings suggest that proper exploration is essential for intelligent problem-solving, and that the current evaluation methods are inadequate for measuring true intelligence. The linked code track entry achieving RHAE=0.30 on the full 55-game private evaluation demonstrates the effectiveness of the EXPLORE-before-PLAN framework, providing a foundation for future research in epistemic agents.
Key insight: AERA's three-phase (EXPLORE/VERIFY/PLAN) framework demonstrates that intelligent exploration is crucial for solving complex reasoning tasks, with performance improving significantly when exploration is properly integrated.
$D^2$-Monitor: Dynamic Safety Monitoring for Diffusion LLMs via Hesitation-Aware Routing
arXiv: 2605.25893
D^2-Monitor addresses the critical gap in safety monitoring for diffusion LLMs, which generate text through a multi-step denoising process unlike autoregressive models. The approach identifies safety hesitation - intermediate hidden states repeatedly falling within a small margin of the probe's decision boundary - as the most informative signal for predicting probe failure.
The bi-level safety monitor dynamically routes monitoring resources by using a lightweight probe as an always-on monitor to jointly estimate hesitation and perform base classification. When hesitation exceeds a threshold, a more expressive but computationally heavier probe is activated, allowing efficient resource allocation at test time. This dynamic routing mechanism allocates monitoring resources efficiently while maintaining high performance.
Evaluated on 3 datasets across 4 D-LLMs, D^2-Monitor achieves state-of-the-art performance with a compact parameter footprint (≤ 0.85M parameters) and exhibits the best trade-off between effectiveness and efficiency relative to 8 baselines. This demonstrates that the approach can effectively monitor diffusion LLMs without requiring substantial computational resources, making it practical for real-world deployment.
Key insight: D^2-Monitor's dynamic safety monitoring approach using hesitation-aware routing achieves state-of-the-art performance with compact parameter footprint while effectively balancing effectiveness and efficiency.
Language Models Need Sleep
arXiv: 2605.26099
The paper introduces a sleep-like consolidation mechanism that addresses the attention mechanism scaling problem in transformers, particularly for long-horizon tasks. The approach periodically converts recent context into persistent fast weights before clearing the key-value cache, shifting extra computation to sleep while preserving wake-time prediction latency.
During sleep, the model performs N offline recurrent passes over accumulated context and updates fast weights in its state-space model (SSM) blocks through a learned local rule. This mechanism allows the model to handle longer contexts more efficiently while maintaining the latency of wake-time prediction, demonstrating a novel approach to managing context length in transformers.
Testing on controlled synthetic tasks including cellular automata and multi-hop graph retrieval, as well as a realistic math reasoning task, shows that increasing sleep duration N improves performance, with the largest gains on examples requiring deeper reasoning. This suggests that the sleep mechanism can be particularly beneficial for tasks requiring complex reasoning over extended contexts.
Key insight: The sleep-like consolidation mechanism in language models improves performance on complex reasoning tasks by converting recent context into persistent fast weights before clearing key-value cache.
Automated Benchmark Auditing for AI Agents and Large Language Models
arXiv: 2605.26079
Auto Benchmark Audit (ABA) systematically audits individual benchmark tasks, uncovering issues such as hidden environment dependencies, specification gaps, and limited grading logic that human annotation cannot reliably catch. The framework's comprehensive analysis of 168 benchmarks across nine domains reveals critical problems that significantly distort capability assessments for agents and LLMs.
The precision of automated audits is validated by expert review and independent third-party reports, demonstrating the reliability of the approach. Filtering out problematic tasks with issues shifts model rankings and increases average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively, showing that benchmark quality directly impacts model evaluation results.
The release of the agentic tool and all task annotations supports the future development of frontier benchmarks by providing a systematic approach to identifying and addressing issues in benchmark design. This work highlights the importance of rigorous benchmark validation and suggests that current evaluation methods may be misleading due to poorly designed benchmarks.
Key insight: Automated benchmark auditing reveals that over 25.7% of evaluated tasks contain critical issues including ambiguous task design, execution environment conflicts, and incorrect ground truths, severely distorting capability assessments.
StakeBench: Evaluating Language Understanding Grounded in Market Commitment
arXiv: 2605.26074
StakeBench introduces an evaluation framework for language understanding grounded in market commitment, linking 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records across Polymarket and Manifold. This approach moves beyond observer-perceived labels to observable market behavior, providing a more reliable measure of language understanding.
The framework tests four diagnostic tasks: detecting market commitment, identifying revealed side, anticipating future action, and performing collective odds projection. Four commitment-aware metrics measure alignment with revealed preferences rather than perceived sentiment, providing a more accurate assessment of model capabilities.
Results show that models partially recover position-side signals, with Directed Accuracy from 0.506 to 0.599, but demonstrate structural failures on later tasks. Ten of the fifteen models collapse to one or two action labels in future action anticipation, and no model consistently improves on the naive odds-direction baseline in collective odds projection. This suggests that while language understanding can be partially captured through market commitment, more sophisticated reasoning and prediction capabilities remain challenging for current models.
Key insight: StakeBench's market-commitment grounded evaluation framework reveals that models partially recover position-side signals but show structural failures on later tasks, with no model consistently improving on naive odds-direction baseline.
When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges
arXiv: 2605.26046
The paper identifies critical failure modes in multi-objective prompt optimization for LLM judges, where textual gradient methods face challenges that prevent the application of standard multi-task learning techniques. The study tests five decomposition modes of textual gradient optimizers by varying how much cross-task information the loss, gradient, and optimizer LLMs share.
Key findings include that optimization never improves over the initial prompt in 6 of 10 configurations, with gradient specificity dropping by 59% when the gradient LLM processes multiple criteria jointly. Additionally, naively combining per-task instructions into a single prompt degrades Spearman's rho by -5.3%. These results identify two separable failure modes that constrain the design space for multi-objective judge customization using textual feedback.
The research highlights the fundamental incompatibility between textual gradient methods and standard multi-task learning approaches, suggesting that new frameworks are needed for effective multi-objective judge customization. This work provides important insights for developing more robust methods for customizing LLM judges to specific tasks or domains.
Key insight: Multi-objective prompt optimization for LLM judges suffers from two separable failure modes: optimization-time gradient dilution and inference-time instruction interference, which together constrain the design space for textual gradient methods.
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
arXiv: 2605.26045
The paper investigates uncertainty quantification for activation oracles, which aim to make model activations legible to humans. The study evaluates 6 different methods for estimating confidence of activation oracles and finds that bootstrap mode frequency is the best-calibrated method among those tested, with significantly better calibration than log-probability baselines.
The experiments on 6,000 samples per oracle reveal that bootstrap mode frequency achieves ECE of 5.7% vs. 25.5% for answer-word log-probability on Qwen3-8B, and 10.3% vs. 13.1% on Qwen3.6-27B, demonstrating the importance of proper uncertainty quantification in activation oracle confidence estimation. This finding has implications for the reliability of interpretability techniques in language models.
The log-prob baseline can serve as a fast triage signal at a fraction of the cost, suggesting that simpler methods can provide reasonable confidence estimates while more complex approaches offer marginal improvements. The availability of code and patched trainer makes this research directly applicable to improving confidence estimation in activation oracles.
Key insight: Bootstrap mode frequency is the best-calibrated method for estimating confidence of activation oracles, with ECE of 5.7% vs. 25.5% for answer-word log-probability on Qwen3-8B, demonstrating the importance of proper uncertainty quantification.
Peak-Then-Collapse and the Four Interface Channels of Knowledge-Graph Tool Use
arXiv: 2605.26037
The paper identifies a fundamental issue in knowledge-graph tool use: interface feedback problems that cause peak-then-collapse patterns in performance. Unlike Python interpreters, web search, and JSON APIs, knowledge-graph tools often fail in ways that leak natural-language signals the model saw in pretraining, making failures more difficult to resolve.
The study demonstrates that a key difference from other tool use scenarios is the interface feedback - while a Python traceback names the failing line, an empty Freebase result [] does not provide useful signal. This surface-level feedback problem exposes a degradation regime that same-family reward redesigns do not fix, indicating that the issue is fundamentally architectural rather than reward-based.
A direct oracle ablation rules out relation selection as the primary cause, showing that injecting gold relations at every retrieval call lifts exact-match accuracy by only +0.20 pp, and that 95.4% of retrieval-dependent errors are retrieval-composition failures rather than answer-extraction failures. This suggests that the problem lies in how models process and compose information from the interface rather than in the accuracy of individual retrieval steps.
Key insight: Knowledge-graph tool use suffers from interface feedback problems where failures leak natural-language signal the model saw in pretraining, leading to peak-then-collapse patterns that are not fixed by reward redesigns.
Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech
arXiv: 2605.26007
The paper presents the first systematic evaluation of transformer-based dementia detection in Filipino speech and the first assessment of NeoBERT in a clinical NLP setting. The study constructs a parallel bilingual dataset of 4,000 DementiaBank-derived transcripts with Filipino translations to preserve discourse-level markers of cognitive decline.
Key findings show that in-domain performance does not transfer across languages, with English-trained BERT dropping to Macro-F1 = 0.455 on Filipino, and that architectural modernization alone does not improve robustness. However, bilingual fine-tuning eliminates cross-lingual degradation across all transformer models, converging to Macro-F1 = 0.969-0.973, demonstrating that linguistic coverage during training drives multilingual clinical NLP performance.
These results suggest that multilingual clinical NLP performance is primarily driven by linguistic coverage during training rather than model scale or architecture, providing important insights for developing robust multilingual clinical NLP systems. The study's findings have implications for resource-constrained settings where multilingual coverage is crucial for effective healthcare applications.
Key insight: Bilingual fine-tuning eliminates cross-lingual degradation across all transformer models, converging to Macro-F1 = 0.969-0.973, demonstrating that linguistic coverage during training drives multilingual clinical NLP performance rather than model scale or architecture.
AI-Assisted Systematization for Evaluating GenAI Systems
arXiv: 2605.26001
The paper addresses the challenge of evaluating generative AI systems, where many targets of evaluation are broad, contested concepts such as 'reasoning,' 'fairness,' or 'creativity.' These concepts become unclear when left underspecified, making it difficult to determine what should be measured or how evaluation results should be interpreted.
The authors introduce a structured representation of a systematized concept, a concept spec, and a validation worksheet to help address the cognitive demands of systematization. They develop two AI-assisted systematizers: a direct, zero-shot approach and a multi-agent approach that more closely mirrors manual systematization approaches from existing literature.
Using these systematizers to produce concept specs for 'hate-based rhetoric' and 'digital empathy,' the authors evaluate the resulting concept specs on content validity and information recoverability. The work demonstrates that AI assistance can effectively support the systematization process, making it more accessible and potentially improving the quality of evaluation frameworks for complex AI concepts.
Key insight: AI assistance can support the cognitively demanding and resource-intensive process of systematization, enabling the creation of structured representations of broad concepts like 'reasoning' and 'fairness' for measurable evaluation.
What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA
arXiv: 2605.25988
The paper reveals a critical insight about checker-guided RAG for biomedical QA: the checker's output distribution during training, not its held-out accuracy, decides whether it provides trainable gradient. This finding challenges the conventional approach of using held-out accuracy as the primary metric for selecting checkers.
Three diagnostic findings emerge: (i) Signal collapse is log-prob-specific, with LLM log-probability scoring labels over 97% of claims neutral, collapsing the RL gradient to zero, while a calibrated MedNLI classifier scores the same pairs non-degenerately. (ii) Moderate signal beats strong signal on answer quality, with a strong proprietary checker triggering a three-step reward-hacking cascade. (iii) Signal strength is policy-dependent, with the same checker registering as moderate on one policy but strong on another without triggering the cascade end-state.
These findings frame the problem as boundary conditions for verifier-as-reward systems, suggesting that the effectiveness of checkers in RL settings depends on their ability to provide non-degenerate gradients rather than their absolute accuracy. The results indicate that moderate-signal local classifiers can train higher-quality models than strong-signal proprietary checkers, challenging the assumption that stronger checkers are always better.
Key insight: The checker's output distribution during training, not held-out accuracy, determines whether it provides trainable gradient, with signal collapse occurring when LLM log-probability scoring labels over 97% of claims neutral.
SafeCtrl-RL: Inference-Time Adaptive Behaviour Control for LLM Dialogue via RL-Driven Prompt Optimisation
arXiv: 2605.25984
SafeCtrl-RL presents a novel inference-time behavioral control framework that enables adaptive safety regulation without requiring model retraining or parameter modification. The method formulates dialogue generation as a sequential decision process where a reinforcement learning agent dynamically selects prompt adjustment strategies based on contextual feedback.
This approach allows unsafe behaviors to be suppressed through iterative refinement, conceptualized as inference-time behavioral unlearning. The framework's ability to adapt safety regulation in real-time without modifying the underlying model parameters represents a significant advancement in deploying safe LLMs in real-world applications.
Evaluated across multiple LLMs and unsafe dialogue scenarios, SafeCtrl-RL consistently improves safety and response quality, outperforming existing prompt-based optimization methods and achieving favorable performance-efficiency trade-offs. This demonstrates the practical utility of the approach for ensuring safe and contextually appropriate behavior in LLM dialogue systems.
Key insight: SafeCtrl-RL enables adaptive safety regulation without model retraining or parameter modification by formulating dialogue generation as a sequential decision process with reinforcement learning agent.
When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation
arXiv: 2605.25981
The paper documents an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models from seven architecture families: meaning-bearing perturbations alter final answers more often than presentation perturbations of comparable severity. This finding reveals that LLM agents are more sensitive to semantic changes than surface variations in input.
Across 68 cells spanning GSM8K, MATH, and HotpotQA, the inconsistency gap averages +19.69 pp after severity matching, with 64/68 cells positive. The gap survives four severity-proxy audits and remains significant when excluding qwen models, suggesting that this phenomenon is robust across different architectures and datasets.
The study validates the headline effect on a fully held-out 11th model and re-tests a pre-registered capability×tractability partition, observing a small but positive held-out effect. Trace-level analysis reveals that semantic perturbations often preserve the first action but induce divergence in intermediate reasoning from later steps onward, accompanied by slightly deeper trajectories. This provides a partial trace-level account of how semantic perturbations propagate through agent reasoning.
Key insight: Meaning-bearing perturbations (e.g., paraphrase, synonym) alter final answers more often than presentation perturbations (e.g., formatting, reordering) of comparable severity, indicating that LLM agents are more sensitive to semantic changes than surface variations.
Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning
arXiv: 2605.26110
Prism addresses the engineering bottlenecks in current Multimodal Continual Instruction Tuning (MCIT) research by introducing a plug-in reproducible codebase specifically designed for scalable MCIT research. The framework separates algorithmic development from the backbone implementation via a lightweight plugin registration mechanism.
This approach allows new strategies to be integrated as independent plugins without modifying the underlying MLLM codebase, thereby eliminating structural fragmentation and accelerating method development. The framework natively supports widely used large-scale training pipeline, enabling reproducible and scalable MCIT experimentation.
The plug-in architecture represents a significant advancement in MCIT research, as it enables researchers to develop and test new methods without the substantial implementation overhead typically associated with modifying base MLLM codebases. This approach promotes code reuse and fair comparison, making it easier to advance the field of multimodal continual learning.
Key insight: Prism's plugin registration mechanism enables new strategies to be integrated as independent plugins without modifying the underlying MLLM codebase, eliminating structural fragmentation and accelerating method development.
Looped Diffusion Language Models
arXiv: 2605.26106
LoopMDM addresses the effective design of transformer architectures for masked diffusion models (MDMs) by showing that selectively looping early-middle transformer layers significantly improves both training efficiency and model performance. This approach brings two key benefits: looping layers at training-time yields a depth-scaling effect without adding parameters, while varying the number of loops at inference-time enables flexible compute scaling.
The results are striking: across multiple pre-training corpora, LoopMDM matches the performance of same-size MDMs with up to 3.3 fewer training FLOPs, while its final performance outperforms them on various reasoning benchmarks, including up to 8.5 points on GSM8K. This demonstrates that selective looping is more effective than naive depth scaling, particularly for complex reasoning tasks.
Furthermore, LoopMDM can scale inference-time compute by increasing the number of loops, and adaptively adjusting the number of loops throughout the sampling process yields additional gains in compute efficiency while maintaining performance. The approach's effectiveness is supported by attention analysis showing that looping promotes interactions among masked positions, providing theoretical justification for its success.
Key insight: LoopMDM's selective looping of early-middle transformer layers significantly improves both training efficiency and model performance in MDMs, achieving up to 8.5 points improvement on GSM8K.
Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
arXiv: 2605.26097
The paper addresses the fundamental problem of forgetting in language models trained on new tasks, traditionally requiring replaying stored exemplars from prior tasks, which is often impractical. The study shows that language models can sample from their own training distribution, and these self-generated samples serve as effective replay data, nearly eliminating forgetting.
Key findings reveal that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. This suggests that the relationship between capacity and forgetting is complex and depends on the model's training state.
Replay breaks the tradeoff between learning rate and forgetting, enabling fast, high-learning-rate fine-tuning without forgetting. This finding has important implications for efficient fine-tuning strategies and suggests that self-generated replay can be a powerful tool for maintaining model performance while adapting to new tasks.
Key insight: Self-generated samples from language models serve as effective replay data, nearly eliminating forgetting, while capacity constraints remain a significant limiting factor in knowledge retention.
Goal-driven Bayesian Optimal Experimental Design for Robust Decision-Making Under Model Uncertainty
arXiv: 2605.26093
GoBOED introduces a goal-driven Bayesian optimal experimental design framework that directly optimizes experimental designs for a specified decision-making objective, rather than maximizing information gain. This approach combines an amortized variational posterior surrogate with a differentiable convex decision layer, enabling gradient-based design optimization that is fully decision-focused.
Theoretical analysis shows that GoBOED gradients are insensitive to parameter directions irrelevant to the decision objective, providing a formal justification for why goal-driven design achieves equivalent decision quality over a wider set of experimental designs than information-gain maximization. This insight addresses a fundamental limitation of traditional BOED approaches that focus on parameter uncertainty reduction rather than decision quality.
Empirical evaluation across source localization, epidemic management, and pharmacokinetic control demonstrates that GoBOED identifies designs that better align with downstream decision objectives and reveals that near-optimal design windows are substantially wider than those predicted by goal-agnostic BOED approaches. This suggests that goal-driven design can be more robust and flexible than traditional information-gain maximization methods.
Key insight: GoBOED's goal-driven Bayesian optimal experimental design directly optimizes experimental designs for a specified decision-making objective, achieving equivalent decision quality over a wider set of experimental designs than information-gain maximization.
OrpQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
arXiv: 2605.26092
OrpQuant addresses the structural flaw in logarithmic Power-of-Two (PoT) quantization - the low angular resolution regime - that becomes particularly pronounced at sub-4-bit thresholds, leading to degradation of high-dimensional feature manifolds. The approach formulates quantization as a dual-basis geometric projection, adaptively synthesizing a higher-resolution residual lattice using strictly shift-and-add operations.
The analytical solver offers a practical alternative to computationally intensive gradient-based optimization, reducing the full-model calibration time for LLaMA-2-7B to approximately 15 minutes. This significant reduction in calibration time makes PoT quantization more practical for real-world deployment while maintaining competitive accuracy.
Extensive evaluations demonstrate ORP's applicability across modalities and hardware efficiency. Under the 3-bit (W3/A16) constraint, ORP achieves a perplexity of 6.10 on LLaMA-2-7B, comparing favorably to conventional MAC-intensive baselines like AWQ without relying on asymmetric scaling, while maintaining competitive accuracy in 4-bit scenarios. At the silicon level, standard-cell RTL synthesis at a 28nm node indicates that ORP effectively mitigates timing bottlenecks associated with dense multiplier trees.
Key insight: ORP's geometric orthogonal residual projection addresses the low angular resolution regime in logarithmic Power-of-Two quantization, achieving competitive accuracy with 3-bit quantization while reducing calibration time to 15 minutes.
Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
arXiv: 2605.26078
The paper develops a global convergence theory for Wasserstein policy gradient (WPG) by exploiting the Bellman structure of entropy-regularized RL. Unlike standard Langevin analyses that do not directly apply, this approach shows that the role usually played by convexity can be replaced by a Bellman-based argument: the soft Bellman residual admits a statewise KL representation with respect to a Gibbs policy.
The analysis demonstrates that Bellman contraction relates this residual to the global optimality gap, and a Bellman resolvent identity connects value improvement to relative Fisher information. Combined with a uniform log-Sobolev inequality (LSI) for the evolving Gibbs family, these ingredients yield a distributional Polyak-Lojasiewicz condition, establishing global convergence of WPG.
Conceptually, the analysis shows that although entropy-regularized RL is not convex in the usual flat sense, the Bellman recursion induces a favorable Polyak-Lojasiewicz-type (PL) geometry that supports global convergence of WPG. This theoretical framework provides important insights into the convergence properties of WPG and suggests that the Bellman structure can be leveraged to establish convergence in non-convex settings.
Key insight: WPG's global convergence is established through Bellman-based arguments that replace convexity with a Polyak-Lojasiewicz-type geometry, showing that entropy-regularized RL has favorable geometric properties for convergence.
Active Query Synthesis for Preference Learning
arXiv: 2605.26072
Info-Synth addresses the computational bottleneck of pool-based evaluation in active learning by generating optimal queries that maximize a mutual information-based objective within a continuous space. This approach explicitly accounts for feedback reliability by introducing a novel confidence aware response model that considers ambiguous comparisons between nearly identical or entirely dissimilar items.
The framework's ability to generate optimal queries by maximizing mutual information provides a principled approach to active learning that overcomes the limitations of standard methods that assume all query feedback is equally reliable. The approach demonstrates versatility and performance across synthetic preference learning, constrained text summary datasets, and subjective, continuous-space controller gain tuning for a simulated mobile robot.
Two strategies, Pair M-dist and Pair Opt-dist, extend Info-Synth to select effective queries even when restricted to finite query pools, showing that the approach can be adapted to practical constraints while maintaining its effectiveness. This work provides a significant advancement in active learning methodology for preference learning applications.
Key insight: Info-Synth's mutual information-based objective for active query synthesis achieves superior performance compared to pool-based evaluation methods, with Pair M-dist and Pair Opt-dist strategies extending the approach to finite query pools.
Rethinking Weak Supervision in Anomaly Detection: A Comprehensive Benchmark
arXiv: 2605.26068
WSADBench introduces the first benchmark that unifies evaluation across distinct weakly supervised scenarios, benchmarking diverse approaches from specialized WSAD methods to advanced tabular foundation models. The benchmark systematically varies label quantity, granularity, and quality to reveal performance boundaries of various methods across 4 modalities.
Key insights from over 700K experiments include: (i) Strong intrinsic correlations exist between these weak supervision scenarios, challenging the isolation of current research directions; (ii) Specialized WSAD algorithms excel only in extreme label-scarcity regimes but are quickly dominated by tabular foundation models and general classification methods as supervision increases or in OOD scenarios; (iii) Unlabeled data shows inconsistent utility across settings, with marginal gains compared to label refinement; (iv) Models exhibit asymmetric sensitivity to different types of label noise.
The benchmark's release as an open-source resource with code and datasets facilitates future WSAD research by providing a standardized framework for evaluating methods across different weak supervision scenarios. This comprehensive evaluation reveals the complex relationships between different weak supervision approaches and highlights the need for more nuanced research directions.
Key insight: WSADBench reveals that strong intrinsic correlations exist between weak supervision scenarios, challenging the isolation of current research directions and showing that specialized WSAD algorithms excel only in extreme label-scarcity regimes.
Conditional KRR: Injecting Unpenalized Features into Kernel Methods with Applications to Kernel Thresholding
arXiv: 2605.26067
The paper studies conditional kernel ridge regression (conditional KRR) by reducing its behavior to that of standard KRR with another fixed kernel, called the residual kernel. This reduction is possible at the cost of an additional term in the expected test risk, bounded by O(1/√N), where N is the sample size.
The theoretical analysis shows that conditional KRR outperforms standard KRR in cases where the F-component of the regression function is more pronounced than the residual part, particularly when F is given by the first k principal eigenfunctions in the Mercer decomposition of K or when F consists of k random features from a random feature representation of K. This provides theoretical justification for the effectiveness of conditional KRR in specific settings.
Both theoretical analysis and experiments confirm that conditional KRR outperforms standard KRR in these cases, demonstrating the practical utility of the approach. The work contributes to understanding when and why conditional KRR can be more effective than standard kernel methods, providing insights for kernel-based learning applications.
Key insight: Conditional KRR's reduction to standard KRR with a residual kernel enables analysis of conditional KRR in cases where the F-component is more pronounced than the residual part, showing that conditional KRR outperforms standard KRR.
Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
arXiv: 2605.26061
NSAC introduces a novel biologically-inspired continuous-time (CT) attention architecture that reformulates attention logit computation as the solution of an Ornstein-Uhlenbeck stochastic differential equation modulated by input-dependent, nonlinear interlinked gates derived from repurposed Neuronal Circuit Policies (NCPs) wiring mechanism.
The architecture induces Gaussian distribution over logits that propagates principled stochasticity through logistic-normal distribution over attention weights to yield probabilistic output. A two-term objective function combining Gaussian negative log-likelihood with an epistemic-separation regularizer enforces higher predictive variance and enables joint quantification of aleatoric and epistemic uncertainty.
Empirical evaluation across diverse learning tasks including irregular CT function approximation, multivariate regression, long-range forecasting, Industry 4.0, and autonomous vehicle lane-keeping shows that NSAC remains competitive against several baselines in terms of accuracy and produces reasonably well-calibrated uncertainty estimates while being interpretable at the neuronal cell level. This demonstrates the practical utility of the approach for probabilistic representation learning.
Key insight: NSAC's biologically-inspired CT attention architecture reformulates attention logit computation as an Ornstein-Uhlenbeck stochastic differential equation, enabling joint quantification of aleatoric and epistemic uncertainty.
Length Generalization with Log-Depth Recurrent Units
arXiv: 2605.26035
MLP-LDRU introduces a type of Log-Depth Recurrent Unit that captures a class of associativity-biased operators designed to approximate recurrence through parallel reduction. This approach addresses the challenge of length generalization in neural networks, where recurrent models suffer from positional biases while transformers are constrained by fixed computational depth.
The model achieves 100% out-of-distribution accuracy on 18 tasks and at least 99.9% on the remaining 3 when increasing max training length, outperforming comparable recurrent and attention-based models. This demonstrates that the approach can effectively handle length generalization problems that are challenging for traditional architectures.
Evaluation beyond regular languages on ListOps and NLP classification benchmarks shows that MLP-LDRU performs competitively, suggesting that the approach's benefits extend beyond the specific domain of regular languages. The work provides important insights into how parallel reduction techniques can be used to address fundamental challenges in neural network architecture design.
Key insight: MLP-LDRU's Log-Depth Recurrent Units achieve 100% out-of-distribution accuracy on 18 of 21 regular-language tasks by capturing associativity-biased operators through parallel reduction.
Retrieval-Augmented Detection of Potentially Abusive Clauses in Chilean Terms of Service
arXiv: 2605.26019
The paper presents a retrieval-augmented generation framework for automated detection and classification of potentially abusive clauses in Chilean Terms of Service, designed for local execution. The approach combines efficient clause detection, hybrid dense-sparse retrieval, reranking, and prompt augmentation to support medium-sized open-weight language models.
The framework's effectiveness is demonstrated through the introduction of the Chilean Abusive Terms of Service Extended corpus, comprising 100 contracts and 10,029 annotated clauses in 24 legally grounded categories. Experiments comparing commercial and open-weight language models, fine-tuned encoders, and traditional baselines show that retrieval-augmented prompting substantially improves performance.
The approach enables local models to approach larger cloud-based systems at lower computational and token cost, making it practical for real-world deployment in consumer protection applications. This work demonstrates the value of retrieval-augmented methods for legal NLP tasks and provides a practical design for AI-assisted consumer contract review.
Key insight: Retrieval-augmented prompting substantially improves performance in detecting potentially abusive clauses in Chilean Terms of Service, enabling local models to approach larger cloud-based systems at lower computational and token cost.
Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?
arXiv: 2605.25929
The paper studies multi-agent LLM deliberation through the lens of Friedkin-Johnsen (FJ) opinion dynamics, showing that FJ parameters are input-dependent, turning multi-agent deliberation into a mixture of experts. This perspective implies that multi-agent systems can outperform single agents and static ensembles when routing reflects agent competence.
Since competence is latent in practice, the analysis focuses on how influence is established through observable proxies: agents' self-assessed confidence, their perceived confidence, and initial alignment with other agents' views. This approach provides a practical framework for understanding and optimizing multi-agent collaboration.
The findings suggest that effective multi-agent systems require careful consideration of how competence is reflected in observable behaviors, and that influence dynamics play a crucial role in determining system performance. This work provides theoretical insights into the mechanisms underlying successful multi-agent deliberation.
Key insight: Multi-agent deliberation becomes a mixture of experts when FJ parameters are input-dependent, with influence established through observable proxies like self-assessed confidence, perceived confidence, and initial alignment.
Multi-Agent Coordination Adaptation via Structure-Guided Orchestration
arXiv: 2605.25746
MACA addresses the challenge of balancing structural stability and dynamic adaptability in large language model-based multi-agent systems by casting multi-agent coordination as posterior inference over the joint distribution of structure and orchestration. The framework learns a task- and budget-conditioned structural prior over agent participation and interactions.
This prior guides a policy-based orchestration as an approximation to posterior inference, enabling efficient solutions with fine-grained control. The approach allows for joint adaptation of structure and orchestration, suppressing redundant interactions and converging coordination toward task-effective execution.
Across benchmarks, MACA outperforms adaptive multi-agent baselines by an average of 8.42% while using 43.19% fewer tokens, demonstrating the effectiveness of the structure-guided orchestration approach. The framework's ability to balance stability and adaptability makes it suitable for complex, dynamic multi-agent tasks.
Key insight: MACA's probabilistic approach to multi-agent coordination enables efficient solutions with fine-grained control by learning task- and budget-conditioned structural priors over agent participation and interactions.
Collaborative Threat-Aware Autonomy (CTAA)
arXiv: 2605.25741
CTAA presents a role-differentiated multi-agent framework for collaborative threat-aware trajectory planning in environments containing dynamic, adversarial Weapon Engagement Zones (WEZs). The approach assigns distinct roles to a fleet of Autonomous Collaborative Platforms (ACPs): primary intercept, escort, and decoy.
Each ACP independently employs a reactive guidance law derived from the Collision Sphere Boundary for Evader Zero-Set (CSBEZ), accounting for pursuer maneuverability constraints imposed by minimum turn radius, and steers toward the safest heading that also makes progress toward its goal. Role assignment and spatial route separation induce probabilistic redundancy and threat saturation effects.
The framework demonstrates that probabilistic redundancy, where N independent paths raise team success probability, and threat saturation, where lower-priority escorts and decoys draw adversary attention, free the primary vehicle to transit uncontested. This approach provides a principled method for improving mission success probability in adversarial environments.
Key insight: CTAA's role-differentiated multi-agent framework improves team-level mission success probability through probabilistic redundancy and threat saturation, with distinct roles for primary intercept, escort, and decoy agents.
Recursive Multi-Agent Trading System: Iterative Optimized Portfolio Strategy Under Geopolitical Uncertainty
arXiv: 2605.25311
RMATS integrates four specialized agents - Sentiment, Report, Analysis, and Risk - coordinated through a recursive Manager Agent with iterative feedback loops. The system's architecture is designed for risk control rather than return maximization, making it suitable for institutions prioritizing capital preservation under geopolitical uncertainty.
Experimental evaluation over a 561-trading-day period demonstrates that RMATS achieves a maximum drawdown of 9.62%, lower than MVO (15.49%) and FinBERT Sentiment (15.28%), and exhibits the lowest event-period drawdown in 3 of 5 geopolitical stress scenarios tested. This shows that the risk-control-oriented architecture is effective in protecting capital during volatile periods.
Ablation studies confirm the individual contribution of each agent component to downside protection, demonstrating that the system's effectiveness comes from the coordinated interaction of all four specialized agents. The approach positions RMATS as a risk-control-oriented architecture suitable for institutions prioritizing capital preservation under geopolitical uncertainty.
Key insight: RMATS achieves lower maximum drawdown (9.62%) compared to MVO (15.49%) and FinBERT Sentiment (15.28%), demonstrating effectiveness of risk-control-oriented architecture in geopolitical uncertainty.
Adaptive Punishment for Cooperation in Mixed-Motive Games
arXiv: 2605.24516
APC introduces an adaptive punishment mechanism for cooperation in mixed-motive games that determines punishment intensity based on both a dynamic punishment probability and the severity of defection. This approach addresses the challenge of balancing the efficacy and cost of punishment, where persistent imposition of peer punishment may undermine the punisher's interests.
The method uses a defection awareness module whose learning is guided by game reward to accurately assess defection and its severity. Theoretical analysis and empirical results show that APC performs effectively in iterated public goods game, significantly outperforming existing baselines across sequential social dilemmas.
The approach demonstrates that strategic deterring of defection through adaptive punishment policies can foster cooperation more effectively than static approaches. This work provides important insights into how punishment mechanisms can be designed to be both effective and cost-efficient in promoting cooperation in complex multi-agent environments.
Key insight: APC's adaptive punishment mechanism balances efficacy and cost by determining punishment intensity based on dynamic punishment probability and defection severity, significantly outperforming existing baselines.
A Reinforcement Learning Inspired Latent Yield Based Adaptive Algorithm Switching Mechanism
arXiv: 2605.24436
The paper introduces a computationally efficient approach for aggregating an algorithm's performance across multiple problem instances that is fairly immune to erratic variations in instance features. Inspired by features inherent to Reinforcement Learning (RL), this technique encapsulates rewards and penalties into a latent yield that triggers exploitation and exploration.
The approach employs island models, inspired by Genetic Algorithms, to facilitate parallel exploration and performance exchanges among algorithm populations inhabiting local repertoires. This distributed approach enables effective algorithm switching by capturing the latent yield that represents algorithm performance across different problem instances.
Experimental evaluations on sorting algorithms and robotic obstacle avoidance tasks demonstrate the feasibility and effectiveness of the approach, highlighting its potential in domains where adaptive algorithm selection is critical. The method's ability to handle dynamic environments where problem characteristics evolve over time makes it particularly valuable for online or dynamic applications.
Key insight: The latent yield-based algorithm switching mechanism uses reinforcement learning-inspired features to aggregate algorithm performance across multiple problem instances, enabling computationally efficient adaptive algorithm selection.
ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents
arXiv: 2605.24134
ProofAgent Harness introduces open infrastructure for scalable, auditable, and adversarial AI agent evaluation that addresses critical limitations in current evaluation methods. Unlike static score-based approaches, the harness provides evaluation infrastructure around an agent that curates evaluation intelligence, runs adversarial multi-turn trials, captures behavioral traces, and applies post-hoc multi-juror scoring.
The core of the harness is Adversarial Multi-Juror Scoring with Turn-Level Audit, which evaluates completed agent behavior under pressure using calibrated juror personas, consensus checks, and turn-level evidence. This approach reveals that strong agents fail selectively through weak metrics, fragile turns, unsafe reframing, and manipulation paths.
Experiments across customer support, medical triage, privacy and security, and code generation agents show that a small quantized local Harness LLM can challenge production agents powered by best-in-class large LLMs, suggesting that evaluation capability emerges from the full harness pipeline rather than model scale alone. This demonstrates the importance of comprehensive evaluation infrastructure over model size for ensuring reliable AI deployment.
Key insight: ProofAgent Harness transforms AI agent evaluation from static scores to scalable adversarial evaluation infrastructure, enabling repeatable, evidence-backed, extensible, and actionable before deployment.
Anticipate and Learn: Unleashing Idle-Time Compute in Proactive Agents
arXiv: 2605.25971
ProAct introduces a proactive agent architecture that leverages idle-time compute to anticipate and fulfill likely upcoming user needs, addressing the fundamental limitation that AI agents remain reactive to explicit user prompts. The architecture analyzes evolving dialogue history together with persistent memory to predict upcoming needs and iteratively acquire information.
The approach's effectiveness is rigorously evaluated through ProActEval, a comprehensive benchmark comprising 200 scenarios across 40 domains with predictable need chains and diverse user cognitive profiles. Empirical results demonstrate significant advantages over reactive baselines, with ProAct accelerating task completion by reducing required turns by 14.8%, decreasing user effort by 11.7%, and cutting hallucination rates by 28.1%.
MemBench evaluations confirm that ProAct achieves state-of-the-art reflective accuracy, underscoring its sustained and robust performance. The work demonstrates that proactive capabilities can be effectively implemented to improve user experience and reduce agent errors, representing a significant advancement in agent design for user-facing applications.
Key insight: ProAct's proactive agent architecture leverages idle-time compute to anticipate user needs, achieving 14.8% reduction in required turns and 28.1% reduction in hallucination rates compared to reactive baselines.