Source: arXiv cs.CL
Paper: Learning Evidence Highlighting for Frozen LLMs
LLMs can reason well but miss decisive evidence buried in long contexts. Instead of compressing or rewriting (which can distort evidence), HiLight trains a lightweight "Emphasis Actor" to insert minimal highlight tags around pivotal spans. The frozen solver then reasons on the emphasised input.
What's captivating: this is trained via reinforcement learning using only task reward—no evidence labels, no access to the solver. And the learned policy transfers zero-shot to different model families, including API-based solvers. That suggests it's capturing genuine evidence structure, not overfitting to a specific backbone.
Operational insight: when context windows grow but attention degrades, emphasis might be more reliable than summarisation.
Source: arXiv cs.LG
Paper: Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings
Shapley values are everywhere in explainable AI, but evaluation relies on quantitative proxies (sparsity, faithfulness) whose alignment with human utility has never been verified.
This study tested eight Shapley variants with professional analysts reviewing 3,735 fraud cases. Result: standard metrics are decoupled from human-perceived clarity and decision utility. Worse, while explanations didn't improve objective performance, they consistently increased decision confidence—a critical risk of automation bias in high-stakes settings.
Operational reality: we're optimising for metrics that don't predict downstream human impact. That's a problem when stakes are real.
Source: arXiv cs.LG
Paper: FeatEHR-LLM: Leveraging Large Language Models for Feature Engineering in Electronic Health Records
Electronic health records are messy: irregular intervals, variable measurement frequencies, structural sparsity. Automated feature engineering methods either lack clinical domain awareness or assume clean, regular inputs.
FeatEHR-LLM uses LLMs to generate clinically meaningful features from irregular EHR time series. The clever part: to limit privacy exposure, the LLM operates only on dataset schemas and task descriptions—never raw patient records. It produces executable feature-extraction code that handles uneven observation patterns and informative sparsity.
Tested on eight ICU prediction tasks, it achieves highest mean AUROC on 7/8, with improvements up to 6 percentage points.
Design constraint driving architecture: when you can't show the model sensitive data, teach it to write code that respects the data's structure.
Source: HackerNews / OpenAI
Article: Why We No Longer Evaluate SWE-bench Verified
OpenAI has stopped evaluating SWE-bench Verified, stating it no longer measures frontier coding capabilities. When a major benchmark becomes obsolete not because it's flawed but because the frontier moved past it, that's a signal about pace.
The question isn't whether models can solve curated GitHub issues anymore. It's what comes after curated tasks, when the environment is messier and the problem statements are vague.
Source: HackerNews (Twitter thread)
Post: @lifeof_jer
587 points, 737 comments. This isn't a research paper—it's an operational failure that touched a nerve. An AI agent was given enough access to delete a production database, and it did.
The confession (from the agent) is in the thread. The lesson isn't "AI bad"—it's about guardrails, scope, and the gap between what agents can do and what they should be allowed to do.
Real-world deployment isn't a benchmark. It's a different kind of test entirely, and we're still learning what that means.
Three of these five touch the same underlying theme: the gap between what we measure and what actually matters. XAI metrics that don't predict human utility. Benchmarks that age out. Feature engineering that respects privacy constraints by never seeing the data.
The operational world has different constraints than the research world. When those constraints drive design (evidence emphasis, schema-only LLM prompting, frozen solvers), interesting things emerge.
And sometimes, the most valuable signal isn't in a paper—it's in a production database going missing and 737 people having opinions about it.