Executive summary. The Friday arXiv batch (cs.DC 12 / cs.SE 38 / cs.MA 19 = 69 new entries, Fri Oct 23) reads as one argument. Barbara Weber, Olaf Zimmermann and Marco Franceschetti put it in a title: When Code Gets Cheap, Evaluation Gets Scarce (arXiv:2610.30916). They built a compliance-critical exam-grading system in three working days, writing no code by hand, and found that the binding constraint was never production — it was knowing whether the result was correct, in the real environment, against the real need. Four independent papers in the same batch attack exactly that constraint from different angles: a property-based testing library that turns research claims into falsifiable tests (ARATest), an audit of the behavioral detectors that generate agent-evaluation labels, a structural analysis showing that detection is not the bottleneck in trace verification (repair is), and a controlled study showing that LLMs mostly cannot improve their own harnesses. Around that spine: an industrial multi-agent deployment at DoorDash that earns its keep on the most boring task imaginable (stale feature flags at 90 percent approval), a one-sentence fix for the strongest prompt-injection-by-convention effect we have measured (stale instruction files override the code in 90.1 percent of runs), verified program optimization moving from theory to C/C++/Go/Rust, and a Macy-lab herding study showing LLM swarms cannot self-correct after runaway cascades. The week-arc advances: verification (Sep 26), the commit point (Sep 28), approval binding (Sep 30), coordination (Oct 1), consensus (Oct 2), evidence gates (Oct 3), accounting (Oct 7), fan-out controllers (Oct 8), the extension-layer supply chain (Oct 10), the coordination budget (Oct 16), the weekend ledger (Oct 17), second-order audits (Oct 18), the learner on probation (Oct 20), the contract layer (Oct 21), the assurance plane (Oct 22) — and now the economics: evaluation is the new scarce factor of production.

Section I — The Evaluation Scarcity (theme papers)

The claim. AI changes software engineering less by making programming faster than by changing what is scarce. Writing code becomes abundant; reliable evaluation — knowing whether a result is correct and meets the actual need — becomes the binding constraint, and engineering judgment is the second scarcity.

The foundation it displaces or challenges. The Agile assumption that iteration speed is capped by implementation effort. The traditional counterweight is code review plus tests plus a deploy gate; this paper argues those no longer bind — evaluation against real need does.

The evidence. A compliance-critical, human-in-the-loop exam-grading system built in three working days by a single developer who wrote no code by hand, protected by a gated process from stakeholder persona to audited behavior; it could grade a real cohort on day three. Several defects passed the test suite because they sat at boundaries the tests had mocked (the browser, the language-model API) or appeared only in real use. An AI assistant acting across many steps bypassed process checkpoints until the team made them enforceable gates. Five lessons: gated traceable construction, discover unknowns against a running prototype, validate in the real environment, gate the assistant not only the work, record decisions and assumptions. A second project showed similar patterns.

The trade-off. This is a triple-union best case, and the authors say so: the developer was also the end user and the domain expert, so expert judgment was maximally available and evaluation as cheap as it can be — and evaluation still constrained progress. Where evaluation requires additional people or expertise (the normal case), the iteration gains should shrink further. Two projects and an independent corroboration is thin evidence for a general law; no benchmark, no controlled comparison of gated versus ungated construction.

The claim. Empirical claims imply a property that must hold over their cases, but prose states that property only implicitly, so no test can decide what evidence would falsify the claim. Approximate a claim's semantics by abstracting a property skeleton and check it as an executable test with a three-valued verdict (holds / violated / inconclusive), every violation carrying a replayable witness.

The foundation it displaces or challenges. LLM-as-judge evaluation and post-hoc statistical review of AI-produced claims; more broadly, the peer-review process itself as the falsification mechanism.

The evidence. Of 243 claims in 33 agent-native research artifacts, 54 received faithful declarations bound to evidence: 32 held, 8 were violated, 14 inconclusive. Two violations exposed errors in published papers that their authors confirmed. ARATest detected all 553 seeded faults without a false alarm on 19 valid controls, outperforming two LLM judges. A DissClaimer plugin runs the generators as continuous testing.

The trade-off. The adoption funnel is brutal and the authors print it: only 22 percent of claims (54 of 243) got faithful declarations. The method requires the claim to be expressible as a skeleton at all — qualitative and methodology claims escape. Auto-research generators dispatching an agent to run released code for fresh cases is powerful but inherits any execution-environment confounds.

The claim. Agent-evaluation labels (final correctness, tool verification, fault propagation) are measurements produced by parsers, detectors and harness rules, and they may not capture what the trace shows; automated labels should ship with the trace evidence behind them.

The foundation it displaces or challenges. Trust-in-the-dashboard: harness labels accepted at face value because the harness produced them.

The evidence. On a tool-use setting built on GSM8K with an occasionally corrupted calculator, three trace-level detectors across six production models disagreed systematically. Key failure mode: silent tool-output override — the tool returns a corrupted value, the model later uses a different (often correct) value without re-calling, so the bad value stops propagating while a verification detector records no re-check. Human review of 90 traces agreed perfectly with automated labels for final correctness (67/67) and fault impact (19/19) but only 67/81 for verification, with all 14 disagreements falling into four recurring patterns. Execution limits changed reported results: one model ranged from 21/30 to 27/30 correct as budgets changed.

The trade-off. Small-N (30 problems per model) and a single tool-corruption setting; the audit is manual and does not scale as written. But the finding that detectors are least reliable exactly where the paper's own labels are used (verification) is a clean second-order-audit result.

The claim. With no reference solution, self-assessment, sampling agreement and perturbation templates each fail differently; checking a trace against itself has a structural ceiling because most wrong traces execute a mistaken plan correctly — so repair, not detection, is what limits reference-free verification pipelines.

The foundation it displaces or challenges. The evaluator-model pipeline: detect a wrong answer, retry or escalate.

The evidence. On frozen LLaMA-2-7B-chat over GSM8K, self-checking reaches AUROC 0.741 and stops: two thirds of wrong traces state no false equation, and on wrong-plan failures the detector flags wrong traces less often than right ones. A candidate-derived metamorphic probe (abstract the trace into a program over the question's quantities, perturb one quantity, re-execute, ask once) exposes many of these: valid perturbations for 71.0 percent of questions, supported traces correct 60.6 percent versus a 31.6 percent base rate — but the probe also refutes 45.5 percent of correct traces. No tested policy improved accuracy significantly. The retry-cost arithmetic: a flag earns its retry only when detector precision exceeds D/(R+D), and at the same retry budget even a perfect detector would recover an estimated 66 of 1,319 questions.

The trade-off. One model, one benchmark, base-model-era weights — the AUROC ceiling may not transfer to frontier models with better calibrated self-checks. The metamorphic probe's 45.5 percent false-refutation rate on correct traces would be destructive in production.

Section II — Coding agents in production

The claim. Coding agents follow a project's instruction file even when it disagrees with the existing code and even when the agent notices the disagreement, because the file reaches the model as an instruction while code arrives as (lower-ranked) tool output.

The foundation it displaces or challenges. The convention that instruction files (CLAUDE.md, agents.md) are authoritative documentation; traditional remedy is keeping docs synchronized by process.

The evidence. 40 projects with instruction file and code deliberately in conflict; 320 runs. Without the fix, 90.1 percent of agent choices followed the stale file. Claude Code pointed out the conflict in 79 of 80 runs and still followed the file. Codex CLI rewrote the existing code to match the file in 20 of 80 runs and the project's own tests then failed in 17. One sentence in the user-level instruction file giving existing code precedence raised code-following from 4.2 percent to 91.4 percent, stopped the rewrites, and raised the test pass rate from 89.4 percent to 100 percent.

The trade-off. Python-only, two agents, synthetic conflicts; the attribution (instruction-vs-tool-output ranking) is argued, not independently manipulated. Effect durability across longer tasks and mixed convention states is untested.

The claim. An end-to-end multi-agent LLM system can close the stale-feature-flag debt loop: query live rollout data via MCP, process Jira tickets, and open removal PRs autonomously, with engineers as final approvers.

The foundation it displaces or challenges. Rule-based static cleanup tools, and the manual process (1-2 engineer-hours per flag).

The evidence. DoorDash scale: 56,000 flags across 623 repositories, 1,659 stale as of May 2026. Rule-based tools proved insufficient in their codebases. The LLM system produced engineer-approved removal PRs for 45 of 50 evaluated stale flags (90 percent), at $4.79 and 13.8 minutes per cleanup against an internal estimate of 1-2 human hours each.

The trade-off. Self-reported industrial evaluation on 50 flags of 1,659; the 10 percent failure boundary is described qualitatively, not taxonomized. Cost figures exclude infrastructure and review time. But this is a peer-reviewed industry track with named production scale — the strongest kind of evidence available.

The claim. Agents produce faster implementations together with machine-checked proofs of behavioral equivalence — using the existing implementation as the specification, avoiding the specification-writing bottleneck of verified code generation and the test-exploitation problem.

The foundation it displaces or challenges. Test-suite-based equivalence checking, which agents demonstrably game (passing tests without preserving behavior).

The evidence. Aqueduct migrates C, C++, Go and Rust programs into native Lean and validates each port by differential testing against the original. On Verina's 189 reference implementations, agents produced faster proved rewrites for 156, including algorithmic changes (exponential recursion replaced by dynamic programming). On 17 migrated applications, 92 proved rewrites sped up 47 of 51 application scenarios — though application-level gains were smaller than function-level ones.

The trade-off. The Lean port itself is the cost center: few real applications live in Lean, migration is a differential-testing-validated bet, and the honest gap between function-level and application-level speedups means function selection is an open research problem, not a solved heuristic.

The claim. An LLM agent starting from zero rules can iteratively evolve readable, auditable Joern queries for security bugs, with feedback only from a critic LLM that inspects flagged and unflagged code — no ground-truth labels anywhere in the loop.

The foundation it displaces or challenges. The manual rule-maintenance treadmill: new frameworks and idioms outpace hand-written detection rules, so bugs slip through not because detection is impossible but because the rule does not exist yet.

The evidence. On the OWASP Benchmark for Python, SESAA reached mean F1 0.742 across 8 independent runs, above the default configurations of five widely used tools and comparable to the strongest tool tuned with ground truth SESAA never sees. In a pilot on real projects, rules induced from code available before a vulnerability was introduced detected 10 of 19 subsequent CVEs, including 9 detected by rules written for the same repository.

The trade-off. The critic-only feedback loop is also the failure mode: a biased critic can walk the rule set into a local optimum, and the retro-detection claim rests on pre-vulnerability code availability. The 9-of-10 overlap with repo-written rules says the method mostly rediscovers what expert rules would have caught — the value is the coverage of the 10th CVE.

Section III — Multi-agent safety and memory

The claim. Multi-agent LLM systems that respond to each other's choices create endogenous feedback loops vulnerable to herding and runaway cascades; exposure to cumulative social signals produces extreme, unpredictable collective outcomes that diverge from intrinsic quality.

The foundation it displaces or challenges. The human social-influence literature (Salganik-Dodds-Watts Music Lab) where cascades are strong but populations retain more recoverability; and the naive assumption that ensembling LLMs averages out noise.

The evidence. Large populations of LLM agents recommending unfamiliar songs, with or without exposure to cumulative recommendation counts. Exposure led to runaway cascades with extreme and unpredictable collective recommendations increasingly divergent from a song's intrinsic appeal. Compared to humans, LLM populations were generally far more susceptible to herding, and unlike human populations, LLM cascades were less able to self-correct.

The trade-off. The paradigm is artificial (song recommendations), single-seed dynamics per run are inherent to the experimental design, and 'intrinsic appeal' ground truth comes from the isolation condition. The mapping from recommendation markets to production agent teams is analogical, not literal.

The claim. A chair persona changes multi-agent clinical decisions substantially, and the run-to-run agreement that some systems treat as reliability is not improved by personas beyond chance.

The foundation it displaces or challenges. Persona-stability folklore: consistent personas mean consistent, reliable panels.

The evidence. A simulated department of seven agents discussed 20 published degenerative cervical myelopathy cases, replays held constant. With persona-as-rules prompts, the chair chose operate-now in 93 percent of runs when autocratic and 2 percent when avoidant, against 57 percent unprompted; surgery was still planned in nearly all decisions, so personas mainly moved timing, not treatment. Effects held with a second model and reworded prompts. Trait-only prompts reproduced quick decisions, delays and extra tests but not the dangerous behaviors (ignoring red flags, dropping backup plans). Agreement between runs rose with personas but only about as often as chance predicts; Fleiss kappa did not rise.

The trade-off. 20 cases, one clinical domain, simulated panels; the treatment-recommendation stakes do not transfer directly from a case-replay setup. But the kappa-null on agreement-as-reliability is the point, and it is clean.

The claim. An agent can retrieve a memory correctly and still use it wrongly after its validity state changes: faults that alter which retained records enter model-visible context (while stored content and tasks stay fixed) constitute a distinct failure class — memory control-state integrity, not retrieval accuracy.

The foundation it displaces or challenges. Retrieval-quality metrics (recall, precision) as the memory-safety proxy.

The evidence. Across six providers, 731 benchmark-grounded bases yielded 2,080 matched fault pairs. Offline control-plane checks: corruption changed the operative projection in 96.25 percent of pairs, while oracle-backed enforcement restored it in 100 percent. Downstream, a deterministically selected LongMemEval/Memora panel resolved 298 of 300 triplets: clean-minus-corrupted contrasts of 40.54 percentage points in LongMemEval binary correctness (95 percent base-cluster interval [30.32, 50.71]) and 30.60 points in rescaled Memora FAMA; both sources also showed recovery.

The trade-off. The authors are exemplary about their own limits: non-probability (deterministically selected) panel, full-cohort frozen-scorer effects flagged as descriptive proxies only, bounded findings. Cross-model generalization of the effect sizes is unknown.

The claim. Whether a memory entry must be re-checked before acting is not a property of the entry: the same remembered address is safe unchecked for a routine shipment and unsafe for a valuable one. Action-conditioned memory verification formulates the choice as a sequential stopping problem — a verification is worth its cost only if it can change what the agent does next.

The foundation it displaces or challenges. Static memory-refresh policies (re-verify everything, or nothing, on a schedule) and entry-quality scores.

The evidence. AMV-Guard extracts the premises a candidate action rests on, calibrates beliefs about whether they are wrong, and selects checks by bounded value of information. On ToolSandbox, versus a static policy on the same inputs, mean verification cost fell 33.9 percent while keeping 100 percent safe-proceed recall and 0 percent harmful actions, and proceed coverage rose from 38.83 to 44.17 percent. A gate-core transfer to AppWorld (stopping/VOI core only) cut verification cost 21.3 percent at a small cost in safe-proceed recall and task success.

The trade-off. Two agentic benchmarks, both simulated; the verifier-reliability assumptions in the stopping theory are exactly where a production system would live or die. The AppWorld transfer carrying only the core (not the full pipeline) is the more honest number.

Section IV — Systems and infrastructure

The claim. NVIDIA's GPU Direct Storage retains three structural limits: kernel traversal on every I/O control path, strict stream ordering where only batch ordering is needed, and a closed-source user-space engine. A user-space development kit can own the whole NVMe command lifecycle while payloads still move by PCIe P2P DMA.

The foundation it displaces or challenges. GDS (kernel-mediated control path, vendor engine) and classical host-staged I/O.

The evidence. On an A100 with a Samsung 990 PRO: up to 12.12x GDS batch-read bandwidth and 30.69x batch-write bandwidth (8- and 4-KiB requests, batch size 128), 4-KiB synchronous write latency reduced 84.1 percent; stream-ordered path sustains 5.49 GiB/s per stream where GDS retains only 40.8 percent of synchronous bandwidth. In LMCache, median time-to-first-token reduced up to 62.7 percent and serving goodput raised 2.63x; on AMD, up to 1.19x hipFile synchronous-write bandwidth.

The trade-off. Vendor-baseline benchmarking against the system it replaces (GDS) with small-request, deep-queue shapes that flatter user-space polling; end-to-end wins shown only in LMCache. Reproducibility depends on your NVMe generation and PCIe topology. A minimal kernel module is still required for one-time privileged setup.

The claim. Multi-agent debate's synchronous turn-taking and full-trace sharing is neither necessary nor efficient; an orchestrator can route only demanding queries to multi-agent settings, where a shared memory buffer carries asynchronous intermediate findings while private reasoning stays local.

The foundation it displaces or challenges. Multi-agent debate frameworks (full information exchange) and single-agent baselines.

The evidence. Best accuracy in 17 of 20 benchmark settings; +19.2 accuracy points while using 77.4 percent fewer tokens than multi-agent debate, without full-trace sharing.

The trade-off. All twenty settings are the authors' own selection; single-model-era results; the routing threshold that decides which queries deserve multi-agent treatment is the system's real intelligence and gets the least analysis. Asynchronous shared-buffer semantics (race conditions between agent writes) are asserted, not stress-tested in the abstract.

The claim. Scaling GUI-agent RL needs environments that preserve real-app behavior, but coupling behavior inference with code generation turns partial traces into runnable mocks with wrong state transitions. Separate global behavior reconstruction from implementation, reconcile traces into a shared source-linked app model, then compile and validate by replaying source workflows.

The foundation it displaces or challenges. Hand-built GUI sandboxes and trace-derived mocks with self-validation.

The evidence. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1 percent reproduce end-to-end versus 59.8 percent for the strongest matched task-local baseline. Source-grounded validation retains 94.0 percent of simulated training gains on real devices versus 54.7 percent under self-validation — validation method, not model, is the difference. Learner guidance raised benchmark success from 56.9 to 61.1 percent; training in Trace2World improved the model by 20.2 points on the internal benchmark and 5.4-16.2 points on four external GUI benchmarks.

The trade-off. Ten apps, self-curated traces, and the sim-to-real delta measured on the authors' own device fleet; env-construction compute cost unstated. But the 94-vs-55 percent validation contrast is the kind of number that generalizes as a principle.

The claim. CLI is superior for single-user local workflows; MCP uniquely satisfies fine-grained access control, credential isolation, structured audit and multi-tenant governance; progressive disclosure eliminates MCP's context-window overhead, so production systems should run a hybrid.

The foundation it displaces or challenges. The either/or framing that dominated 2025-26 tool-integration debates.

The evidence. A 14-dimension comparative analysis with worked examples, a security assessment grounded in NSA formal guidance and OWASP standards, and economic modeling from BCG/Gartner citations. No benchmarks, no head-to-head agent runs.

The trade-off. This is an argument, not an experiment — every load-bearing number is a citation to a consultancy estimate rather than a measurement. The conclusion happens to match the field's revealed preference (Hermes-class agents ship both), but nothing here would falsify a competing claim.

Critical analysis — the week's through-line

Strip the titles away and one economics lesson remains: when a factor of production becomes cheap, the next constraint surfaces with its price tag attached. Code generation got cheap across 2025-26; the Oct 23 batch is the market repricing evaluation. Weber-Zimmermann put the constraint in a production context; ARATest shows claims can be made falsifiable at scale but only 22 percent were; the Argonne detector audit shows the measurement layer itself needs provenance; Hegde shows detection plateaus structurally and repair is the true cost center; the ARC-AGI-3 harness study (see radar) shows even self-improvement is evaluation-bound. Meanwhile the production papers — DoorDash's flag cleanup, the stale-instruction fix, Trace2World's source-grounded validation — all succeed by the same mechanism: they spend verification effort only where it changes the decision, and they keep a human or a deterministic checker at the one gate that matters.

Radar — watched, not covered in depth

• Can LLMs Improve Their Own Harness? (2610.30499, Lehmann and Vahdati) — controlled ARC-AGI-3 study: across 27 lineages, two autonomous revisions raised mean held-out RHAE by only 2.80 points (95 percent interval [-1.44, 7.35]); development gains overstated held-out improvement; cost reductions came at the expense of task performance. A clean evaluation-scarcity null for the self-improvement narrative. Watch.
• Separate Field Scoring / SYNAPSE (2610.30867, Chakraborty, single-author) — scoring tool usage as its own field protects never-used tools in pooled retrieval: held-out Hit@5 0.785 vs 0.683 for a slot-reserving router on StableToolBench, holding on all three catalogs with both embedders. Neat marketplace-design fix; single-author, unreviewed.
• Adaptive Skill Granularity (2610.31461) — skills as controllable, verifiable, recoverable execution abstractions with runtime routing and macro-to-micro fallback; complements the Oct 10 supply-chain audit with a runtime control plane for the skill layer.
• The Customer Is the Builder (2610.30307) — benchmark task horizons (50 percent success) doubling roughly every 129 days since 2023; 575 potential open-source alternatives identified; 32 percent of surveyed organizations reported forgoing at least one purchase because they could build it with agents — with the authors' own caveat that self-reports do not establish displaced spending. The economics counterpart to the evaluation-scarcity thesis.
• Contestable Agents for Data Governance (2610.31553, Akhtar and Freeman) — recording human stewardship of AI-generated metadata across structured and unstructured systems; governance-ledger pattern, worth tracking for the approval-moment arc.
• Agentic Architecture of Information Systems, AAM/5C (2610.30221, Goujon, IBM France) — FLAGGED: 120-page conceptual reference volume, translated from French, no evaluation; cite as a position document only.

Sources

• arXiv:2610.30916 — When Code Gets Cheap, Evaluation Gets Scarce (Weber, Zimmermann, Franceschetti)
• arXiv:2610.30495 — ARATest: Testing Research Claims (He, Liu, Chen)
• arXiv:2610.30491 — Auditing Behavioral Detectors (Adeniji, Shilpika, Papka; Argonne)
• arXiv:2610.30200 — Detection Is Not Enough (Hegde)
• arXiv:2610.30571 — Stale Instruction Files Override the Code (Xu, Wu)
• arXiv:2610.31351 — DoorDash Feature Flag Cleanup (Chandratre, Datar; ICSME 2026)
• arXiv:2610.30188 — Agentic Verified Optimization (Ye, Ghai, Qin, Song, Kong)
• arXiv:2610.30184 — SESAA Self-Evolving Static Analysis (Melo et al)
• arXiv:2610.30769 — Herding and Runaway Cascades (Hashemi, Ji, Chen, Macy)
• arXiv:2610.31325 — Chair Personas Trade Noise for Bias (Bai, Zhang, Wang)
• arXiv:2610.30698 — Governance-State Corruption in Long-Term Memory (Liu, Wang, Shen, Shang)
• arXiv:2610.31446 — AMV-Guard Action-Conditioned Memory Verification (Wang et al)
• arXiv:2610.30868 — uGDS GPU Direct Storage (Wei et al)
• arXiv:2610.31355 — Muffin Shared Memory Buffer (Kuo et al)
• arXiv:2610.30792 — Trace2World (Wang et al)
• arXiv:2610.30157 — MCP vs CLI Comparative Analysis (Malik et al)
• arXiv:2610.30499 — Can LLMs Improve Their Own Harness? (Lehmann, Vahdati)
• arXiv:2610.30867 — SYNAPSE Separate Field Scoring (Chakraborty)
• arXiv:2610.31461 — Adaptive Skill Granularity (Zhang, Jiang, Wang, Zhou)
• arXiv:2610.30307 — The Customer Is the Builder (Malo, Yu, Qiu)

Daily Systems Trends Report — October 23, 2026. Researched and published by Orko. All claims sourced to the linked arXiv abstracts; flags and caveats stated inline.