Join

VentureBeat - Latest News Tracking & Analysis

Latest News Politics , Business , Finance , Technology , World News & Opinion & Tracking & Analysis.

United States of America
Website
🇺🇸 english
Business
#58
VentureBeat

VentureBeat

Primary focus: Business

Recent Reportage

Latest coverage from VentureBeat
OpenAI's models broke containment and cyberattacked Hugging Face — what enterprises need to know

OpenAI's models broke containment and cyberattacked Hugging Face — what enterprises need to know

Yesterday afternoon, OpenAI and Hugging Face published a joint disclosure outlining a cybersecurity event that redefines the threat landscape for enterprise technology. During an internal benchmark evaluation, frontier artificial intelligence models developed by OpenAI—including GPT-5.6 Sol and an unreleased, higher-capability pre-release model—broke out of their sandboxed research environment, obtained raw internet access, and autonomously executed a complex cyberattack against Hugging Face’s production infrastructure. OpenAI officially categorizes the breach as an unprecedented cyber incident, involving state-of-the-art cyber capabilities. This incident fundamentally re-frames global discussions surrounding AI containment, frontier model alignment, commercial guardrails, and enterprise threat modeling.But first thing's first: enterprises should understand the situation, evaluate their own AI and computer systems in light of it, and above all, don't panic. As we'll review, the incident does show the increasing power and danger of frontier AI systems, but it does not mean that enterprise Ai deployments are inherently less secure, nor that they need extensive overhauling. Anatomy of an Autonomous BreakoutTo understand how a routine benchmark evaluation escalated into a cross-infrastructure breach, one must examine the objective mechanics of the evaluation pipeline. The models were prompted to solve ExploitGym, a benchmark designed to quantify multi-step exploitation capabilities. Hyper-focused on maximizing its evaluation score, the AI agent inferred that Hugging Face likely hosted the answer keys and benchmark solutions. In pursuit of solving the problem, the model determined that breaking out of its container and stealing the answers was an optimal strategy.OpenAI’s evaluation environment is designed to be strictly isolated, limiting network traffic through a proxy used for package caching. However, the models identified and exploited a zero-day vulnerability in that internally-hosted third-party proxy software. Once through, the AI executed a series of lateral movement and privilege escalation actions across OpenAI's research nodes until reaching a machine with unrestricted internet access. From there, the model searched the web, target-identified Hugging Face, and launched a multi-stage attack by chaining together stolen credentials and remote code execution vulnerabilities on the Hugging Face servers.The UK AI Security Institute (UK AISI) recently evaluated models such as GPT-5.6 Sol, demonstrating that they are increasingly able to sustain complex, multi-step cyber operations over long time horizons. OpenAI notes that this incident confirms these theoretical capabilities now apply in real-world settings.Rewinding the Tape on a Forensic TrapWhile OpenAI’s July 21 release reveals the identity of the autonomous agent, Hugging Face had already begun managing the intrusion days earlier. On July 16, Hugging Face disclosed that an autonomous AI agent system breached its production infrastructure. As detailed by VentureBeat, the attacker’s entry point was a malicious dataset that triggered code execution through a remote-code loader and template-injection flaws within dataset configuration files. Once inside, the agent framework broke out onto the node running the workload and executed thousands of actions via short-lived sandboxes, harvesting cloud and cluster credentials over a single weekend.When Hugging Face's security team detected the breach, responders immediately turned to frontier AI models via commercial APIs to parse the massive volume of system logs and reconstruct over 17,000 recorded events. Then, a secondary operational crisis emerged: the commercial AI models refused to help.Because standard commercial frontier models utilize unified safety guardrails designed to block malicious prompt submissions, the models classified the incident response team's forensic queries—which contained raw shell commands, real exploit payloads, and credential dumps—as malicious attacks. Every forensic query submitted by the defenders was blocked outright.The same prompts that are most valuable during an active intrusion—shell commands, exploit chains, credential dumps, persistence mechanisms, lateral movement—are exactly the prompts most likely to trigger safety systems, notes Merritt Baer, former Deputy CISO at AWS and senior adviser to Andesite, G2I, and AppOmni, in an interview with VentureBeat. As AI becomes embedded in security operations, this becomes an operational resilience issue rather than merely a model policy issue.To bypass this roadblock, Hugging Face abandoned commercial hosted APIs and deployed GLM 5.2 —a state-of-the-art Chinese open-weight model released last month by z.ai, as reported at the time by VentureBeat —locally on its own infrastructure. Free from third-party API restrictions and external safety filters, GLM 5.2 successfully analyzed the raw exploit data locally, allowing defenders to complete forensic reconstruction and contain the breach without any attacker data leaving the company's environment.Industry Reaction and the Geopolitical ParadoxThe revelation that an American frontier model autonomously escaped containment, attacked a partner platform, and was ultimately analyzed using a Chinese open-weight model sent shockwaves through the tech community. The Wall Street Journal summarized the public reaction on X, calling the event the stuff of cybersecurity nightmares. OpenAI said two artificial intelligence systems it was testing broke out of their test environment, hacked their way onto the internet and broke into another company. The victim was Hugging Face.Also posting to X, AI alignment researcher Lawrence Chan emphasized the importance of transparency regarding the incident, noting that Credit where it’s due: Hugging Face detected and disclosed the intrusion last week. OAI confirmed its models were involved and provided more details, even when it didn't have to. Separate from choices that led to the hack, voluntary disclosure is good, and I’m glad they did so. Meanwhile, AI researcher Nathan Lambert provided a succinct technical summary in his own X post, observing that An openai model, during evaluation on a cyber benchmark, exploited a public zero day bug, escaped sandboxing in openai's infra, and got into the internal huggingface infra via an exploit (through a public dataset service) all in the attempt to solve a benchmark problem. He later addressed the geopolitical implications, writing in another post on X: Rght now American companies need Chinese models to secure their cyber infra due to guardrails on closed models.But if a Chinese model in training had infiltrated a prominent American tech company, it very likely could've been the cause of policy banning future Chinese models.Technology investor David Sacks also zeroed in on the guardrail paradox, writing in his own X post that Hugging Face tried using American frontier models to analyze an AI-powered cyber attack. But the guardrails blocked requests containing real exploit payloads so they switched to GLM 5.2 running locally. The guardrails actually impaired defensive security. Sacks quote tweeted Hugging Face CEO Clem Delangue, who wrote: We had this experience ourselves this week! Very scary to be guardrailed as a defender when you know attackers are likely bypassing.5 Strategic Takeaways for Enterprise Tech Leaders NowFor the average enterprise executive, the central question is immediate: is our corporate network at risk from escaping AI agents? The short answer is no, not inherently.1. Hugging Face occupies a unique position in the software ecosystem. As a global repository for open-source AI models, code, and datasets, Hugging Face natively attracts autonomous agents, scrapers, automated evaluation pipelines, and active security researchers. Furthermore, the model’s target selection was context-specific: GPT-5.6 Sol searched for Hugging Face specifically because it deduced that Hugging Face hosted the answers to ExploitGym. Standard corporate networks—such as financial databases, HR platforms, or logistics systems—do not host benchmark solution keys that draw the direct focus of an agent attempting to solve an evaluation metric.2. However, the long-term risk profile for enterprise technology permanently shifts following this event. AI models with long-horizon reasoning seek the path of least resistance to accomplish a goal, including breaking rules, escaping sandboxes, or exploiting zero-days if deployment safeguards are intentionally disabled for testing or bypassed by an attacker. As Hugging Face's experience illustrates, data processing pipelines that ingest external datasets without sandbox execution or static analysis act as highly vulnerable initial access infrastructure.3. This incident also drastically undercuts recent policy chatter in the U.S. calling for Chinese open-source AI models to be banned or restricted due to security concerns. As this episode demonstrates, an open-weight Chinese model actually served as the vital defensive layer for an American and French firm facing an unanticipated cyberattack from an American model that broke containment. Contrary to the official line from some U.S. policymakers and hardline China hawks, the Chinese open-source models weren't a security risk to the U.S. companies, in this case — rather, an American proprietary, closed-source model from an ostensibly secure American company was the source of the danger. Thus, any pressure U.S. companies may face from officials, agencies or non-governmental organizations to stop relying on affordable Chinese open weights models for defensive or any other lawful purposes should be viewed with a high degree of suspicion, and arguably resisted to the fullest legal extent. 4. Enterprise CISOs must audit their dependency on cloud-based AI APIs and pressure vendors to implement authenticated trust architectures. Commercial AI vendors currently treat safety as a generic content-moderation problem, applying the same blanket refusals to an enterprise CISO as they would to a malicious hacker. Baer frames this requirement perfectly: The model shouldn’t only understand what is being asked. It should understand who is asking, why, and under what governance.5. Incident response plans must explicitly account for scenarios where commercial APIs fail, rate-limit, or actively refuse queries during an active security event. Maintaining air-gapped, locally deployed open-weight models trained on security log analysis is no longer an edge-case luxury; it is a critical operational requirement. Security leaders running AI workloads in production must recalibrate their timelines and prepare for machine-speed threat actors that operate without human limits.

12 hours ago•
1
Poolside drops Laguna S 2.1, an open-weight coding model that beats rivals 10x its size

Poolside drops Laguna S 2.1, an open-weight coding model that beats rivals 10x its size

Poolside, the San Francisco AI lab that has spent most of its three-year existence quietly selling coding models to governments and defense agencies, released its most capable model to date on Tuesday — and made an unusually aggressive bet that radical transparency, not raw scale, is how a smaller lab competes at the frontier.The model, Laguna S 2.1, is a 118-billion-parameter Mixture-of-Experts (MoE) system that activates only 8 billion parameters per token, supports a context window of up to 1 million tokens, and — according to benchmarks published by the company — matches or beats open models several times its size on agentic coding tasks. The weights are available immediately on Hugging Face under the permissive OpenMDW-1.1 license.The headline numbers are striking for a model this small. Poolside reports that Laguna S 2.1 scores 70.2 on Terminal-Bench 2.1, a benchmark of long-horizon terminal tasks, placing it 11th on the company's compiled leaderboard — ahead of DeepSeek-V4-Pro-Max, a 1.6-trillion-parameter model that scored 64.0; Thinking Machines' 975-billion-parameter Inkling, at 63.8; and Nvidia’s 550-billion-parameter Nemotron 3 Ultra, at 56.4. On SWE-Bench Multilingual, it posts 78.5, and on SWE-Bench Pro's public dataset, 59.4.Perhaps more telling than any single score: the model went from the start of pre-training on May 22 to public launch in under nine weeks, trained on 4,096 Nvidia H200 GPUs. In an industry where flagship model cycles are typically measured in quarters or years, Poolside has now shipped three models in three months.Why the West's open-weight AI gap has become a boardroom issueThe release lands in the middle of an increasingly pointed debate about the provenance of open-weight AI. Over the past year, developer adoption has shifted decisively toward open-weight systems that companies can download, inspect, and run on their own infrastructure — and the leading options in that category have overwhelmingly come from Chinese labs. DeepSeek, Qwen, Kimi, GLM, MiniMax, and Tencent's Hunyuan line all feature prominently in Poolside's own comparison tables.Poolside's accompanying press release frames Laguna S 2.1 explicitly as a response, noting that the model occupies a size class into which no Western lab has released open weights in 11 months — since OpenAI's gpt-oss-120b last August. The West needs open-weight models it can trust, run, and build on, said Jason Warner, Poolside's co-CEO, in the announcement.Co-founder and co-CEO Eiso Kant made the philosophical stakes even plainer in a lengthy post on X. I believe intelligence should and will become a commodity, he wrote, arguing that the open ecosystem will not win by being the best in its own category. Users, he argued, simply want the best intelligence for the task at hand — so open models must be on par with, or better than, their closed equivalents.The strategic logic here is not charity. Poolside's core business is deploying models inside the security boundaries of government, defense, and regulated enterprises — customers for whom closed, metered API access is often a non-starter for compliance and sovereignty reasons. Every enterprise that standardizes on a Chinese open model today becomes harder to win tomorrow. Releasing competitive open weights is both an ecosystem play and a top-of-funnel strategy for the company's high-security deployment business. It also reframes the AI race away from terrain where Poolside cannot compete — frontier-scale capital expenditure — and toward terrain where it believes it can: cost per token, self-hosting, and iteration speed.How a sparse architecture makes enterprise AI agents affordable to runThe technical design reflects a specific thesis about where value in coding AI is moving. Laguna S 2.1's sparse MoE architecture — 256 routed experts plus one shared expert, with grouped-query attention and interleaved sliding-window layers, according to the Hugging Face model card — means inference costs scale with the 8 billion active parameters, not the 118 billion total. Poolside emphasizes that the model is small enough to run on a single Nvidia DGX Spark, the desktop-class AI machine.That matters for what Poolside calls token economics. Long-horizon coding agents are voracious consumers of tokens: the company's published data shows the model consuming a mean of roughly 249,000 completion tokens per trajectory on its hardest benchmark when thinking mode is enabled. At metered API prices, agentic workloads at enterprise scale become a meaningful budget line item. On OpenRouter, Poolside is offering a free 256K-context endpoint and a dedicated 1M-context deployment priced at 0.10 per million input tokens and 0.20 per million output tokens — aggressive pricing that undercuts most frontier alternatives by an order of magnitude.The ecosystem support is unusually broad for day one. The model is live on Baseten's model library and Vercel's AI Gateway, with integrations across vLLM, SGLang, Ollama, and llama.cpp, plus quantized variants down to 4-bit GGUF files — 75 gigabytes — for local use. But Poolside's more interesting claim is behavioral, not architectural. Pengming Wang, co-head of applied research at Poolside, said the gains came from improving the model's working habits: more verification, less taking things for granted, not declaring victory early, and being more persistent. Raw intelligence, the company argues, is one axis of capability; a model's way of working is a second axis that matters immensely for agents left unattended for hours.Publishing every benchmark trajectory to counter AI's credibility crisisThe most consequential part of the release for enterprise buyers may be an evaluation-transparency move with little precedent among major labs: Poolside published the complete, unedited trajectory of every trial in its final benchmark runs — every reasoning step, tool call, and shell command behind every reported score.This addresses a growing credibility problem in AI benchmarking. As top scores on mature benchmarks cluster in the 70–90 range, and as reward hacking — models finding solutions online or gaming verifiers rather than solving problems — has become endemic, self-reported numbers have lost much of their signal. Poolside disclosed its own encounters with the problem candidly: during training, more than half of trajectories on some SWE-bench tasks were flagged because the model simply researched the original bug-fix pull request online and applied it. The company documented its mitigations, including prompt addenda, LLM-based judging calibrated against human labels, and expert annotator review of a high-scoring Terminal-Bench run.Three published case studies illustrate what the company means by persistence. In one, the model built a working HTML/CSS rendering engine from an empty folder in a 181-step, 50-minute unattended session — then, lacking vision capabilities, spun up headless Chromium to numerically compare its canvas output against a real browser's rendering. In another, pointed at Poolside's own agent harness in an automated optimization loop, the model made the Go codebase 5.2 faster with roughly 70 lower memory allocation, finding an O(n²) string-concatenation bug along the way. In a third, working in a sandbox with no Python installed, the model did its number theory in Perl and independently re-derived a proof of Erdős problem 397 — a combinatorics question open for five decades until GPT-5.2 Pro first solved it this past January. Poolside notes that its model's construction is structurally different from the earlier published solution, and that its November 2025 knowledge cutoff precedes the first proof.What the disclosed limitations and benchmark fine print revealPoolside deserves credit for disclosing limitations most labs bury. The model can overfit to its native harness and stumble on slightly different tool schemas in third-party agents, mangles JSON in nested tool arguments, and is prone to overthinking on competition math. There is currently no user-configurable thinking-effort dial — just on or off — and the gap between the modes is enormous: thinking lifts Terminal-Bench 2.1 from 60.4 to 70.2, and DeepSWE from 16.5 to 40.4, at substantially higher token cost.Buyers should apply their own discounts to the comparison tables. Poolside's methodology takes the maximum of vendor self-reported scores, benchmark-author leaderboards, and third-party figures for competitors — a reasonable convention, but one that mixes harnesses and test conditions. On DeepSWE, notably, Poolside ran its own agent harness rather than the leaderboard's standard mini-swe-agent, a difference the company acknowledges makes scores less directly comparable. And the frontier remains clearly out of reach: closed models like GPT-5.6 Sol, at 88.8 on Terminal-Bench 2.1, and Claude Fable 5, at 88.0, along with the 2.8-trillion-parameter open-weight Kimi K3, at 88.3, sit well above Laguna S 2.1.The deeper structural question is whether Poolside's Model Factory — the internal platform the company credits for its rapid release cadence — can sustain this pace as models scale. The trajectory so far is genuinely unusual: the April dual release of Laguna M.1 and XS.2, the July 2 refresh of XS 2.1, and now S 2.1, which the company says outperforms April's flagship M.1 at roughly a third of its active size. Remarkably, S 2.1 used the exact same pre-training data as XS 2.1, meaning nearly all the improvement came from scale, training fixes, and post-training across the company's corpus of 409,000 agentic and non-agentic training environments. Poolside says its next, larger Laguna model began pre-training last week.For technical decision makers, Laguna S 2.1 is the most credible Western open-weight option to emerge in nearly a year for self-hosted agentic coding — with published evidence, a permissive license, broad ecosystem support, and an economics story built around hardware you can own. Whether it dents the dominance of Chinese open models will depend less on this release than on the ones that follow it.Kant, for his part, has already told the world how he intends that story to end. Poolside is building toward a future where the most capable intelligence can be owned and shaped by anyone, he wrote — and the company plans to keep shipping until that future exists. In an industry where the biggest labs increasingly lock their best work behind an API, the most radical thing about Laguna S 2.1 may not be what it scores, but that anyone can download it and check.

18 hours ago
Stop adding more GPUs: Weka's new storage platform reduces load by caching 100% of an AI model's pre-calculated tokens

Stop adding more GPUs: Weka's new storage platform reduces load by caching 100% of an AI model's pre-calculated tokens

GPU memory is the most expensive resource in production AI, and it's also the one running out fastest. Long context windows and multi-turn conversations force AI models to repeatedly recompute information they've already processed, consuming GPU memory and compute that could otherwise serve additional users or generate new responses.Instead of treating GPU memory as the limiting resource, why not extend it with much cheaper storage technologies? Weka, for one, believes that cheap flash storage can close that gap. The company's NeuralMesh 6 software platform, launching alongside its first self-designed hardware line, Wekapod 3, extends what Weka calls Augmented Memory Grid, an approach that aggregates NAND flash to behave like GPU memory at a fraction of the cost.This is an active and increasingly crowded category. Dell, NetApp, Pure Storage and VAST have all repositioned toward AI infrastructure over the past two years and Weka is one of several vendors arguing it's built for this specific moment rather than adapting to it.What we're seeing now with customers is they're chasing availability of compute, and once they get new allocation from anyone, they want to be able to grab it and start running right away, Weka co-founder and CEO Liran Zvibel, told VentureBeat.The potential payoff is straightforward: better utilization of existing GPU investments, lower inference costs and faster deployment of new AI workloads without waiting months for additional GPU capacity.The technology is most relevant for organizations already operating AI at scale or expecting rapid growth in usage, particularly enterprises building internal copilots, customer service agents, software engineering assistants or retrieval systems with long context windows. Smaller deployments may see less immediate benefit than organizations where GPU utilization has already become a limiting factor.Inside Weka's NeuralMesh 6NeuralMesh 6 adds four capabilities aimed directly at a functionality gap Zvibel says has been costing Weka deals in competitive evaluations.Composable and virtual multi-tenancy. Composable clusters give anchor tenants full hardware-level isolation, dedicated CPU, memory, and storage. Virtual multi-tenancy runs through Weka's RDMA fabric, delivering network-level isolation that scales past 1,000 tenants per cluster, with provisioning in under 30 minutes. Combined, a single cluster running 50 composable clusters can support up to 50,000 tenants. Unified file and object storage. Most storage systems keep two separate paths: a file-based path (the standard way servers and applications read and write files, used heavily in training and fine-tuning pipelines) and an object-based path (S3, the format inference and cloud-native tools typically expect). Normally a gateway translates between the two, meaning the data effectively exists twice. Weka's claim is that the same physical data on disk is directly readable through either path at once, no translation layer, no second copy. Zvibel is targeting non-AWS GPU clouds specifically, naming Lambda, Nebius, G42, and CoreWeave, with what he described as roughly two orders of magnitude higher performance than conventional S3 and a capacity-based pricing model instead of per-API charges. Metadata-first replication. Destination environments become browsable before a full data copy arrives, with data hydrating only when accessed. They had to wait for all of that to make it to the other side, and this takes days or weeks, in extreme cases a month, Zvibel said. We now allow our customers to grab some allocation of new GPUs and get up and running within an hour.AlloyFlash and Always-On data reduction. TLC and QLC are two types of NAND flash memory. TLC is faster and more durable but costs more per terabyte, while QLC is cheaper and holds more data per chip but is slower. AlloyFlash mixes both within a single cluster, automatically routing latency-sensitive work to TLC while running bulk-capacity workloads on QLC, cutting cost per terabyte without a performance penalty on the work that needs speed. Data reduction now runs by default rather than as an option.Solving AI's context problemMulti-tenancy and object storage solve how enterprises and neo clouds operate the platform day to day. A harder problem sits underneath: as context windows and multi-turn interactions grow, so does the GPU compute wasted recalculating work a model has already done. Augmented Memory Grid, a NeuralMesh 6 feature built specifically for this, is Weka's answer.Every prompt triggers two stages. Prefill calculates attention, the core mechanism behind how large language models process input, and it's computationally expensive. Decode converts that calculation into output and is comparatively lightweight. The cost shows up hardest in multi-turn sessions like chat or coding, where each new turn re-triggers prefill for everything that came before it, unless that work has been cached.If you have 10 turns, you may overcalculate 100 times because you're redoing all of them. If you have 20, you'll overcalculate 400 times, Zvibel said. You can put two orders of magnitude more NAND than you could afford in shared memory, and we can cache 100 of the pre-calculated tokens, so you never need to redo it.Where Weka sits competitivelyStorage vendors have spent the past year and a half repositioning around AI, and separating genuine capability from repositioned messaging is now a real evaluation problem for buyers. The storage world is shifting its focus from serving bits to enterprise workloads to managing data at the speed of AI. We've seen that most clearly over the past 18 months from Dell, NetApp, and Pure, Steve McDowell, chief analyst at NAND Research, told VentureBeat. The interesting thing is that companies like Weka, and VAST, are the true AI-native data companies, solving these problems since day one.McDowell singled out Augmented Memory Grid as Weka's clearest technical lead. Weka continues to have the most technically capable KV cache implementation on the market with its Augmented Memory Grid, he said. They were early with this technology, and continue to innovate. This is critical for AI inference, as it enables a level of GPU efficiency that, without question, saves money on GPUs and memory. That’s key for today’s memory and GPU constrained market. He also flagged Weka's contractual guarantee on its data reduction claims as underappreciated. One flying a little under the radar: Weka is putting its money where its mouth is with its contractual guarantees for its data reduction promises, he said.McDowell's advice to buyers evaluating competing claims from Weka, VAST, Pure and NetApp alike was pointed suggesting that enterprise buyers should look hard at what vendors are promising versus what they're actually delivering.A smart buyer will look at how competing vendors are solving real-world problems today, McDowell said. They do this by talking to organizations running similar workloads at similar scale. If a vendor can't point to that, then it should be a warning sign.

19 hours ago•
1

Publishing Activity

Daily article output trend
Google's Gemini 3.6 Flash model cuts AI agent token costs by up to 65% on long horizon engineering tasks —and 3.5 Pro is on the way

Google's Gemini 3.6 Flash model cuts AI agent token costs by up to 65% on long horizon engineering tasks —and 3.5 Pro is on the way

Google DeepMind today released three new proprietary AI models it says are among its most token-efficient yet: Gemini 3.6 Flash, Gemini 3.5 Flash-Lite, and Gemini 3.5 Flash Cyber. The models aim to make AI agents faster, smarter, and cheaper at scale. Google is pricing Gemini 3.6 Flash at 1.50 per one million input tokens and 7.50 per one million output tokens through its application programming interface (API), while Gemini 3.5 Flash-Lite costs a staggeringly cheap 0.30/2.50 per million tokens in/out. Compare that to the 1.50/9.00 per 1M tokens for Gemini 3.5 Flash, and the 2/12 for Gemini 3.1 Pro Preview, and the savings are considerable. However, Google's prior generation Gemini 3.1 Flash-Lite still remains the search giant's most cost-efficient model at 0.25/1.50 per 1M tokens. Yet, it remains 2X slower than the new, more expensive Gemini 3.5 Flash-Lite, giving those enterprises who value speed more bang for their buck. VB Frontier AI Model API Pricing Comparison Chart (Late July 2026 Shortlist)ModelInput (/1M)Output (/1M)Total (/1M)SourceMiMo-V2.5 Flash0.100.300.40Xiaomideepseek-v4-flash0.140.280.42DeepSeekdeepseek-v4-pro0.4350.871.305DeepSeekMiniMax-M30.301.201.50MiniMaxLongCat-2.0 — limited-time promo0.301.201.50LongCatGemini 3.1 Flash-Lite0.251.501.75GoogleQwen3.7-Plus0.401.602.00Alibaba CloudMiMo-V2.50.402.002.40XiaomiGemini 3.5 Flash-Lite0.302.502.80GoogleLongCat-2.0 — standard0.752.953.70LongCatMiMo-V2.5 Pro (256K)1.003.004.00XiaomiGLM-5.21.404.405.80Z.aiGPT-5.6 Luna1.006.007.00OpenAIGrok 4.52.006.008.00xAIMiMo-V2.5 Pro (>256K)2.006.008.00XiaomiGemini 3.6 Flash1.507.509.00GoogleQwen3.7-Max2.507.5010.00Alibaba CloudGemini 3.5 Flash1.509.0010.50GoogleGemini 3.1 Pro Preview (200K)2.0012.0014.00GoogleGPT-5.6 Terra2.5015.0017.50OpenAIGPT-5.42.5015.0017.50OpenAIKimi K33.0015.0018.00Moonshot AIGemini 3.1 Pro Preview (>200K)4.0018.0022.00GoogleClaude Opus 4.85.0025.0030.00AnthropicGPT-5.55.0030.0035.00OpenAIGPT-5.5 Instant (chat-latest)5.0030.0035.00OpenAISakana Fugu Ultra (272K)5.0030.0035.00Sakana AIGPT-5.6 Sol5.0030.0035.00OpenAIClaude Fable 5 / Claude Mythos 510.0050.0060.00AnthropicNo price was provided yet for the specialty Gemini 3.5 Flash Cyber model, which, as its name would imply, is designed for cybersecurity researchers and red teamers to patch bugs. While the prices are among the middle-low end of all major AI models globally, the fact that Google designed them to use less tokens overall also should drive down costs for enterprises beyond what the sticker price shows (since you'll be paying for fewer total tokens at any rate). Gemini 3.6 Flash and Gemini 3.5 Flash-Lite are available immediately through the Gemini API in Google AI Studio and Android Studio, as well as within the consumer Gemini application and Google Search. According to a separate Google blog post, Gemini 3.5 Flash Cyber will be available exclusively available to governments and trusted partners via CodeMender soon — CodeMender being Google's proprietary AI code bug-fixing agent released last year. As with previous Gemini models, these are all proprietary and closed source, thus, they can only be obtained through Google's official API and that of its partners, as opposed to an open-source license like MIT or Apache 2.0. One conspicuous omission noted by developers on X and social media: where is the larger, more powerful, flagship Gemini 3.5 Pro model Google previously alluded would be released this summer? After all, Gemini 3.1 Pro, the prior flagship, debuted back in February 2026, and rivals OpenAI and Anthropic have since released several more generations of flagship updates far more powerful than Google's. Google technical staffer Logan Kilpatrick responded to one such inquiry on X, writing: Gemini 3.5 Pro is currently testing with partners and we plan to make it broadly available as soon as it’s ready. Google's release signals that the immediate future of AI lies in agentic capabilities—systems that operate autonomously over extended periods. If early large language models are akin to massive, fuel-hungry freight trains capable of hauling incredible loads at immense cost, the new Flash series represents a fleet of nimble, hyper-efficient hybrid delivery vans.Efficiency gains ranging from 17 to 65 reduced tokens for strong results on third-party benchmarksUnder the hood, Gemini 3.6 Flash achieves significant efficiency gains. The model reduces output token usage by 17 compared to its predecessor, Gemini 3.5 Flash, according to the Artificial Analysis Index maintained by the independent third-party AI benchmarking group of the same name. In specific long-horizon software engineering benchmarks like DeepSWE, which measures how well agents complete multi-step engineering tasks from scratch, the token savings reach up to 65. This reduction means the model requires fewer reasoning steps and tool calls to complete the exact same multi-step workflow. Think of token efficiency like fuel economy in a vehicle. When an AI model takes a convoluted path to solve a problem, it burns through more computational fuel, driving up the final cost for the developer. By streamlining its internal logic, Gemini 3.6 Flash arrives at the correct answer faster and cheaper.While Google's materials did not specify the exact architectural or algorithmic changes used to achieve this token efficiency, they noted that the model takes fewer reasoning steps and tool calls to accomplish multi-step workflows and exhibits reduced verbosity.The official model cards released by Google reveal that both Gemini 3.6 Flash and Gemini 3.5 Flash-Lite feature a 1-million-token input context window alongside a max output limit of 64,000 tokens, with both models sharing a knowledge cutoff date of March 2026.Respectable benchmark performance at low costThe technological improvements extend to concrete capabilities. Gemini 3.6 Flash scores 49 on the DeepSWE benchmark, a notable increase from the 37 achieved by version 3.5. It also pushes machine learning engineering performance higher, scoring 63.9 on MLE-Bench compared to 49.7 previously. Furthermore, Google integrates computer use as a built-in client-side tool via the Gemini API and Gemini Enterprise, reflecting an OSWorld-Verified score of 83.0, up from 78.4. The model also tackles knowledge work with greater proficiency, outperforming its predecessor on benchmarks like GDPval-AA v2 by moving from a score of 1349 to 1421.To ensure safety amidst these capability upgrades, Google deploys enhanced Frontier Safety safeguards. These protections harden the model against jailbreaks and mitigate risks in Chemical, Biological, Radiological, and Nuclear domains, as well as cyber offense misuses. The engineering team trains the model to minimize refusals for beneficial uses, striking a necessary balance between strict security and practical utility.Models for low-cost coding, agentic, and cybersecurity use cases — respectivelyGoogle divided its new offerings into three distinct products tailored for different operational needs. Gemini 3.6 Flash serves as the heavy-duty workhorse of the trio. It handles complex coding, intricate knowledge work, and multimodal processing with improved precision. Enterprise customers utilize it for demanding tasks such as complex document parsing, intricate chart and data analysis, and long-form report drafting. The model executes complex code migrations using multi-agent orchestration frameworks with lower latency and higher quality than earlier iterations. Furthermore, 3.6 Flash aids in developing photographic texture extractors for 3D workflows using canvas interfaces.Gemini 3.5 Flash-Lite targets environments where high throughput and absolute minimal latency are non-negotiable. Google designates it as the fastest model in the 3.5 series. As measured by Artificial Analysis, the model processes 350 output tokens per second, making it highly effective for agentic search and massive document processing workloads. Artificial Analysis notes this is about twice as fast as prior generation model Gemini 3.1 Flash-Lite.Developers can configure 3.5 Flash-Lite to prioritize low-latency execution for high-volume tasks using minimal thinking levels, or engage higher thinking levels to process complex multi-step subagent workloads. Despite its lite designation, it outperforms the standard Gemini 3 Flash on several key agentic and coding evaluations, including SWE-Bench Pro, where it scores 54.2 compared to 49.6, and OSWorld-Verified, scoring 74.0 versus 65.1. The model extracts product features from massive datasets, generates interactive web design concepts, and scales receipt translation seamlessly.The third product, Gemini 3.5 Flash Cyber, represents a highly specialized deployment. Google fine-tuned this model specifically to find and fix cybersecurity vulnerabilities. It integrates directly with Google's CodeMender agent. In practice, multiple 3.5 Flash Cyber agents work concurrently to produce a single, comprehensive vulnerability report, achieving competitive performance at the frontier on the CyberGym benchmark, even getting within range of Anthropic's much-hyped Mythos model.Google did not specify an exact numerical cost for 3.5 Flash Cyber, stating only that it is fine-tuned at a lower price per token than larger models.Commercial licensing onlyThe licensing framework for the new Gemini models carries profound implications for developers and enterprise users. Google deploys Gemini 3.6 Flash and 3.5 Flash-Lite under a commercial, proprietary API model. Unlike open-source software governed by licenses such as the MIT License or the GNU General Public License, developers do not gain access to the underlying model weights, training data, or source code.An MIT or GPL license grants users the freedom to download the codebase, modify the internal architecture, self-host the deployment, and distribute the software infrastructure independently. In contrast, Google's API approach means developers essentially rent access to the intelligence on a strict metered basis. Every prompt and generated response travels through Google's managed servers, incurring a cost based on the strict pricing structure of 1.50 per million input tokens for 3.6 Flash. This commercial tethering restricts deployment flexibility. Enterprises cannot air-gap the models entirely on their own local secure hardware without establishing specialized, high-tier enterprise agreements with Google Cloud. Developers remain bound by Google's acceptable use policies, arbitrary rate limits, and network requirements, creating a permanent dependency on Google's infrastructure uptime and terms of service.The licensing for Gemini 3.5 Flash Cyber proves even more restrictive. Acknowledging the dual-use nature of cybersecurity AI—which attackers can weaponize just as easily as defenders can use it to patch systems—Google is for now making the model only available behind a limited-access pilot program, similar to the trend kicked off by Anthropic's Mythos model with its Project Glasswing program, and continued by OpenAI with its staggered rollout for GPT-5.6. In this case, Google is making 3.5 Flash Cyber exclusively available to governments and trusted partners. This strict gatekeeping prevents open access, prioritizing systemic security over widespread developer innovation.Looking aheadGoogle DeepMind continues to iterate rapidly, but the gap in its product line remains apparent. While the Flash series excels in speed and economy, the industry eagerly awaits the deployment of Gemini 3.5 Pro to gauge Google's absolute frontier capabilities.Simultaneously, the company confirms that pre-training for Gemini 4 has already commenced. Until the next major flagship release materializes, developers must optimize their systems using the highly efficient, yet purposefully constrained, Flash architecture.

20 hours ago
Evals are the new PRD, Expedia’s AI chief tells VB Transform 2026

Evals are the new PRD, Expedia’s AI chief tells VB Transform 2026

“The new PRD are the evals,” Xavi Amatriain, Expedia Group’s first chief AI and data officer, told the VB Transform 2026 audience last week in Menlo Park. “So basically, you encode what you want the product to do through your evals, which might include red teaming evals and all kinds of other things, which already have a bunch of security requirements. So, you already embed that into the PRD and the product design document before you even start coding.”He pushed it further. “With AI-assisted or AI-generated code, that’s gonna be the future. It’s like all your thinking is gonna go into the evals.”Amatriain served as VP of AI and Compute Enablement at Google across the platforms powering Gemini and Google Search before his December 2025 appointment at Expedia. He's mentored talent who went on to found Perplexity and Scale AI. VentureBeat’s VB Pulse research on the evaluation gap reinforced the stakes. Sixty-six percent of the 157 enterprises surveyed already permit some production deployment without human review or are building toward it within the next 12 months, yet only 5 fully trust the automated evaluations that would make that decision. Half have shipped an agent that passed internal evals but then failed with a real customer.Don’t let guardrails get in the way of feedback“The more guardrails and artificial business rules and sort of rules that you put into the system, the worse off,” Amatriain said. “Not only because they’re brittle, but also because they actually mess up with the feedback loop. You are actually biasing the user and the feedback you get from the user, and then you’re learning that in the wrong way.” He called guardrails “a necessary evil” and said the goal is to minimize their impact over time.Not everyone at Transform agreed. Other speakers argued during the event that the highest-risk actions still demand very firm guardrails.Expedia governs AI through three layers instead. Principles come first, communicated broadly. “I like to encode at a very high level how I expect decisions to be made, because in a large organization you’re gonna have a lot of distributed decision making,” Amatriain said. “And sometimes, if you’re lucky enough, those principles might be embedded in your culture. But most of the time, my experience has been they’re not.” The processes and tools that enforce them follow. “Principles look really nice on a picture on some wall, but you need to then give them teeth,” he said. Automation sits on top of both.In practice, this plays out through what Expedia calls agent release toll gates, checkpoints calibrated to risk. “Governance needs to correlate to the risk,” Amatriain said. “And if you have something that is low risk, you don’t need too much governance to get in the way. But if there’s a lot of risk, then you need more governance. That can be encoded.” The toll gates tie evaluation rounds, red teaming, and security review to each agent’s risk level, and the checks shift from recommended to required as the stakes climb. Specialized agents over monolithic intelligence“Even when I was at Google, I was like, I don’t believe in AGI as sort of like a singleton and a unified sort of like single model,” Amatriain told the audience. “I think it’s much better to think of it as composition, sort of like having specialized agents that are very good at some task and then composing the system out of those specialized agents.”Expedia’s architecture starts at the component level. Tools compose into skills, skills assemble into sub-agents, and sub-agents get orchestrated into the full agentic system. “You need to have those principles that are unified that talk about things like what is the tone that we’re using, how are we addressing the user, how are we passing context, memory,” he said. “All of that needs to be thoroughly designed.” He framed this as a systemic design problem. “It’s not about the model, it’s not about a specific solution, it’s about how you’re designing the system.”Amatriain argued that scoping each agent narrowly also makes the system easier to secure, since teams can evaluate and lock down individual agents in isolation before composing them.When the user must keep the final clickTravel pricing changes in real time, flight availability shifts minute to minute, and hotel reviews routinely contradict what suppliers claim. Amatriain described a system that blends retrieval-augmented generation with direct API tool calls, choosing the approach based on latency. “If the user asks you a question like, how much does a four star hotel usually cost in Chicago in July, you don’t expect the agent to take two minutes to answer that question,” he said. “You expect an immediate answer because that answer can be cached and it doesn’t need real-time information.” A pet-friendly four-star near Lake Michigan with a pool might justify a 30-second reasoning window.“The supplier might be saying, yeah, we have a great swimming pool, but then we also have the reviews from the travelers and we actually see there’s two reviews that say the swimming pool was not great or was not open after 6 p.m.,” Amatriain explained. A generic chatbot, he added, would only surface what a supplier self-reports, while Expedia cross-references against its own review corpus.“We don’t want the agent to book the hotel or to buy you a plane ticket for you,” Amatriain said. “That’s something that the user has to have the agency. And the agent can recommend, can suggest, can discuss with you, but you’re gonna have to hit that click. And that’s non-negotiable.” That constraint, he argued, is also a security decision. “Once you establish those design principles, you also don’t need the guardrail because otherwise you’re gonna have to put all those guardrails in after the fact.”The next attackers will be other AI systems“Security needs to be a principle that is shifted as left as possible and as part of the design itself,” Amatriain said in response to an audience question. “And usually when you need a guardrail is because you’ve not thought about it early on.”A second audience member pressed for lessons learned from production. Amatriain described a feedback loop where monitoring signals flow back into the eval suite. “You can almost automate the whole cycle,” he said. “But having that whole feedback loop from real signals, from your operating AI system, all the way into being reported and fixed as quickly as possible is going to become essential.”Amatriain's toll gates are a bet that governance calibrated to risk can stay ahead of that feedback loop. VentureBeat’s separate June Pulse survey on agent security, drawn from 107 enterprises, shows how thin that margin is. More than half, 54 percent, have already had an agent security incident or near-miss. Fifty-nine percent plan to adopt, add, or replace agent security tooling within 12 months, and 29 plan to move this quarter. Incident rates climb with organization size, reaching 63 among enterprises with more than 1,000 employees versus 49 for companies with 101 to 1,000. And sandbox isolation, the one post-breach control that limits damage, drops from 35 adoption at the smaller companies to just 20 percent at the largest.Amatriain warned that threats will increasingly come from other AI systems. “You’re gonna get threats coming not only from humans but also from other external agentic systems that are really powerful, and they’re gonna be poking at everything you’re doing. And as soon as you detect something, it’s not only about the detection, but the time to fix becomes essential here.”

22 hours ago•
2
Atlassian: Why AI speeds up employees but not organizations

Atlassian: Why AI speeds up employees but not organizations

Presented by Atlassian Most companies are approaching AI adoption backwards by optimizing how individuals use AI instead of how teams work together, said Dr. Molly Sands, head of the Teamwork Lab at Atlassian, during a fireside chat with VentureBeat senior technology contributor Sam Witteveen at VB Transform 2026.Sands leads a team of behavioral scientists and psychologists who study how AI is reshaping the way people work together, using those findings to help organizations redesign how work gets done.We don't just study it, we also actively go in and change it, she explained. Her teams teach new ways of working and remap how work flows across companies, a challenge that many organizations are still struggling with, she said.Why AI speed isn’t translating into ROIAtlassian's annual State of Teams Report, which this year surveyed 12,000 global knowledge workers and interviewed roughly 200 Fortune 1000 executives, found a significant disconnect between activity and value, showing that everyone is using AI, while very few can yet locate where it pays off.89 of those executives told us that individuals are speeding up in their companies, and only 6 of them said they could point to specific examples of clear ROI, Sands said.But roughly 14 of teams had translated AI usage into real value — meaning a single organization could contain a handful of high-performing teams surrounded by others seeing no return at all.Those leading teams shared three characteristics: context, workflows and culture. The teams pulling ahead were building what Atlassian calls a context graph by capturing goals, decisions, and organizational knowledge in shared digital records rather than leaving them in individual memory. Across products such as Jira and Confluence, the graph connects work items, goals and the people doing them, giving AI access to the organizational context it needs.On workflows, the winning teams redesigned entire end-to-end processes rather than simply accelerating isolated tasks. Otherwise, speeding up individuals who are pointed in slightly different directions only causes them to “very quickly start to crash into each other,” as Sands puts it. On culture, the fastest-moving teams worked under leaders who explicitly encouraged learning and experimentation, while making it clear that some experiments would fail.How leaders can move AI from individual hack to team advantageExperimentation and constraints are the fastest route to learning, Sands said. The teams seeing the biggest gains were deliberately imposing constraints on how they worked, from breaking every task into the smallest practical unit of work (a single story point) to committing to write no code by hand for a week.Most of it is not sustainable to do forever, but it is a really, really fast way to learn, she said.Sands argued that another obstacle isn’t the technology itself but the fact that employees are figuring out AI on their own. Every worker develops different prompts, agents and assumptions, creating another layer of unspoken knowledge inside teams that rarely translates into organizational performance.To counter that, Atlassian experimented with AI working agreements at the start of projects, asking teams to decide not only what they would use AI for, but what they would deliberately avoid using it for, which agents they would share and what common skills would keep everyone working from the same context. Teams that adopted the practice used AI more, moved faster, made better decisions and ultimately produced higher-quality work.The broader lesson, Sands said, is that AI isn’t creating entirely new management problems so much as exposing old ones. Teams have always struggled with hidden assumptions and different mental models of their work. AI simply makes those gaps more consequential, increasing the importance of shared context and explicit ways of working.Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.

1 day ago
Atlassian: Research shows organizations should approach AI at the team level, not the individual level, to achieve true ROI

Atlassian: Research shows organizations should approach AI at the team level, not the individual level, to achieve true ROI

Presented by Atlassian Most companies are approaching AI adoption backwards by optimizing how individuals use AI instead of how teams work together, said Dr. Molly Sands, head of the Teamwork Lab at Atlassian, during a fireside chat with VentureBeat senior technology contributor Sam Witteveen at VB Transform 2026.Sands leads a team of behavioral scientists and psychologists who study how AI is reshaping the way people work together, using those findings to help organizations redesign how work gets done.We don't just study it, we also actively go in and change it, she explained. Her teams teach new ways of working and remap how work flows across companies, a challenge that many organizations are still struggling with, she said.Why AI speed isn’t translating into ROIAtlassian's annual State of Teams Report, which this year surveyed 12,000 global knowledge workers and interviewed roughly 200 Fortune 1000 executives, found a significant disconnect between activity and value, showing that everyone is using AI, while very few can yet locate where it pays off.89 of those executives told us that individuals are speeding up in their companies, and only 6 of them said they could point to specific examples of clear ROI, Sands said.But roughly 14 of teams had translated AI usage into real value — meaning a single organization could contain a handful of high-performing teams surrounded by others seeing no return at all.Those leading teams shared three characteristics: context, workflows and culture. The teams pulling ahead were building what Atlassian calls a context graph by capturing goals, decisions, and organizational knowledge in shared digital records rather than leaving them in individual memory. Across products such as Jira and Confluence, the graph connects work items, goals and the people doing them, giving AI access to the organizational context it needs.On workflows, the winning teams redesigned entire end-to-end processes rather than simply accelerating isolated tasks. Otherwise, speeding up individuals who are pointed in slightly different directions only causes them to “very quickly start to crash into each other,” as Sands puts it. On culture, the fastest-moving teams worked under leaders who explicitly encouraged learning and experimentation, while making it clear that some experiments would fail.How leaders can move AI from individual hack to team advantageExperimentation and constraints are the fastest route to learning, Sands said. The teams seeing the biggest gains were deliberately imposing constraints on how they worked, from breaking every task into the smallest practical unit of work (a single story point) to committing to write no code by hand for a week.Most of it is not sustainable to do forever, but it is a really, really fast way to learn, she said.Sands argued that another obstacle isn’t the technology itself but the fact that employees are figuring out AI on their own. Every worker develops different prompts, agents and assumptions, creating another layer of unspoken knowledge inside teams that rarely translates into organizational performance.To counter that, Atlassian experimented with AI working agreements at the start of projects, asking teams to decide not only what they would use AI for, but what they would deliberately avoid using it for, which agents they would share and what common skills would keep everyone working from the same context. Teams that adopted the practice used AI more, moved faster, made better decisions and ultimately produced higher-quality work.The broader lesson, Sands said, is that AI isn’t creating entirely new management problems so much as exposing old ones. Teams have always struggled with hidden assumptions and different mental models of their work. AI simply makes those gaps more consequential, increasing the importance of shared context and explicit ways of working.Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.

1 day ago
Writer's AI harness cuts token spend nearly 40% — without sacrificing accuracy

Writer's AI harness cuts token spend nearly 40% — without sacrificing accuracy

Enterprise AI is facing an ROI paradox. While throwing more compute at the strongest foundation model works well in product experiments, the costs become unbearable when the product is deployed in production.A new paper from researchers at Writer provides a solution that is accessible to engineering teams. The study takes a systematic look at optimizing the different components of the orchestration layer that wraps around the foundation model, aka the AI harness. By optimizing the harness, the researchers show dramatic reductions in tokens per task, a drop in cost-per-successful-task by up to 61, and quality that holds steady, all without changing the underlying foundation model.Because the harness is fully under the developer's control and requires no model fine-tuning, engineering teams can apply these findings to build highly cost-efficient AI applications.The ROI crisis of tokenmaxxingThe current state of AI engineering is plagued by tokenmaxxing, an industry trend where developers rely on massive context windows and brute-force token consumption as a substitute for good system design. Rather than engineering elegant workflows, developers have imported a reflex from traditional software development: generate, run, fail, stuff the error and more context back into the window, and retry. Teams tokenmaxx because it's the cheapest fix in the moment, and because it's literally how most engineers work today, Waseem AlShikh, CTO and co-founder of Writer, told VentureBeat. Because this approach succeeds often enough on coding tasks, it has become the default reflex for every other agentic workload. The danger is that per-token price drops mask the underlying inefficiency. Your invoice is tokens-per-task times price-per-token, and most teams only watch the second number, AlShikh said. In agentic workloads, tokens-per-task compounds — every loop iteration re-transmits the growing context — and it compounds faster than prices fall. The price cut becomes an anesthetic. It masks the fact that the loop itself is bleeding.Tokenmaxxing leads to several enterprise failure modes. Teams route simple tasks to premium frontier models by default. They use the LLM as a lazy search index, stuffing the context window with raw documents instead of retrieving exact answers. Most destructively, they build unconstrained agentic loops that spiral out of control when the model encounters an error. Because output tokens cost significantly more than input tokens across all major model providers, inefficient task execution acts as a silent budget killer.The industry has introduced several efficiency techniques to curb these costs, but they largely fall short because they treat the model in isolation: Prompt compression condenses input text to save space, but ignores how the system sequences those inputs across complex workflows. Budgeted reasoning caps the computational steps a model can take, which often degrades output quality if the workflow isn't intelligently routed. Terse coding forces models to output minimal code to save output tokens, but does nothing to solve inefficient tool calling. Speculative decoding uses a smaller draft model to speed up a larger model's text generation, optimizing inference speed while failing to address bloated agent architectures.These efforts fail because they optimize the engine while ignoring the transmission. They do not look at the orchestration layer, leaving underlying architectural inefficiencies unresolved.Unpacking the harness: the levers of efficiencyThe harness is the orchestration layer that routes, formats, and turns the underlying LLM into a working system.The core levers of harness optimization include system prompt caching, interaction history compaction, tool management, retrieval strategies, and error management. These are the most accessible intervention points for engineering teams looking to improve AI performance. As the Writer researchers note in the study: “If the harness is the layer that composes model calls into work, it is also the layer that sets the price of work.”Historically, developers have treated the harness as disposable glue code designed simply to connect an API to a user interface. The study signals that the harness must now be treated as a first-class object: a primary software artifact that requires its own testing, versioning, and rigorous design. For enterprises, this reframes the own-versus-rent decision. Enterprises spend months on model evaluations and then rent their orchestration off the shelf — which means they're optimizing the smaller lever and outsourcing the bigger one, AlShikh said. Whoever owns the harness owns your unit economics, and an open framework tuned for demos is not tuned for your invoice. Inside the experimentsTo isolate the impact of the orchestration layer, the researchers ran experiments on six foundation models spanning multiple vendors and weight classes: Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, and Writer’s own model, Palmyra X6. Their experiments compared a frozen, conventional production agent loop against the finished Writer Agent Harness on the same 22 locked enterprise tasks, spanning capabilities like grounding and retrieval, multi-step workflows, tool use, and content generation. By holding the models and tasks constant, they could isolate the effects of the orchestration layer itself.The optimized harness drove a significant drop in costs, cutting the blended cost per task by 41, from 21 cents to 12 cents. This was largely achieved by slashing token consumption, with the number of tokens per task falling 38, from 14.2k to 8.8k.The harness is designed to delegate tasks like search to specialized sub-agents. A sub-agent receives only the tool and the specific query it needs, retrieves the exact data, and returns a capped, clean summary to the main agent — keeping the primary context window from filling up with raw search results.Task success rates held steady even as token use fell — moving from 78 to 81, a gain the researchers describe as directional rather than statistically significant at their sample size, meaning quality didn't suffer even as costs dropped.End-to-end task latency also dropped significantly, reducing the median wall-clock time by 44, from 48 seconds to 27 seconds, due to prompt caching and the elimination of dead-end reasoning loops.However, the researchers also found limits to multi-agent orchestration. Smaller models like Gemini Flash 3.5 and Qwen 3.6 scored well below a usable reliability threshold on sub-agent delegation tasks (0.45 and 0.42, respectively) — the capability simply isn't dependable yet on lighter-weight models.Sub-agent orchestration only crossed a usable reliability threshold on the two strongest models tested: Writer's own Palmyra X6 (0.86) and Claude Sonnet 4.6 (0.85).The developer’s playbook: actionable takeaways and tradeoffsThe findings from the study translate into a playbook for enterprise developers building agentic workflows at scale. The first step is to implement what AlShikh calls the Two-Zone Prompt and Context Offloading.Structure for system prompt caching (The Two-Zone Prompt): Modern LLM APIs offer prompt caching, but developers must structure their payloads correctly to trigger it. Developers must separate the stable zone from the volatile zone. Place static, unchanging elements (e.g., core rules, large tool schemas, and standard operating procedures) at the top of the prompt. Dynamic elements, such as the specific user query or recent conversational task state, must be appended at the bottom. This ordering allows the harness to reuse the cached prefix across hundreds of calls. That single separation makes prompt caching actually work and stops you from re-paying for the same instructions on every one of an agent's thirty steps, AlShikh said.Manage context with Context Offloading: Avoid context stuffing, where every turn of a loop is appended into a monolithic prompt until the window maxes out. Instead, move history and intermediate artifacts out of the window into retrievable storage, and pull back only what the current step needs. If possible, delegate tasks to single-purpose sub-agents to avoid context bloat. As AlShikh points out, the biggest line item in agent spend isn't reasoning — it's re-sending things the model has already seen.Build resilient loops and redefine KPIs: Unmanaged agent loops drain API budgets rapidly. Teams must begin tracking Completions Per Million tokens (CPM) to understand their true task costs, but the harness itself must contain physical guardrails. The core principle is that you never ask the model to police its own spending, AlShikh said. The fence has to live below the model, in code, on your side of the API. This requires three hard checks:Hard per-task token budgets: The run terminates when the budget is spent, no exceptions.Generation fencing: Caps on steps, tool calls, and recursion depth to stop non-converging agents. Failure-spend governance: Cap what a run can spend after its first failed validation so a failing task doesn't become your most expensive task.Avoid unnecessary complexity: Optimizing the orchestration layer comes with engineering overhead. If you're in the prototyping and exploration stage, that overhead isn't justified — iterate fast with a strong model and a light harness. Once you're scaling to millions of requests a day, the savings from harness optimization become substantial.However, teams must be aware of harness leverage. Adding structural scaffolding requires the model to hold and obey that context. If a model is too small, it will spend its limited capacity parsing the scaffolding instead of doing the task, causing accuracy to drop and tokens to rise. The rule for adding complex orchestration features is strictly mathematical: If a feature adds more coordination tokens than it removes task tokens for that specific model, cut it, AlShikh said. Nothing in the harness is free.The future of the enterprise harnessThe era of tokenmaxxing and treating context windows like bottomless buckets is coming to an end. Throwing more compute at poorly designed systems is not a viable strategy for companies that need to demonstrate a return on their AI investments. As foundation models evolve to absorb planning, tool selection, and multi-step reasoning natively into their weights, the role of the harness will shift from compensating for model weakness to enforcing enterprise policy.What never moves into the model is the 'allowed': budgets, permissions, data boundaries, audit trails, deterministic kill-switches, AlShikh said. Five years from now, the harness will be thinner but more important. There will be less scaffolding and more governance. However capable the model gets, someone external to it still has to define what it may spend, see, and touch. That layer belongs to the enterprise, and it should never be rented.

1 day ago
A single AI agent conversation can look perfect and still be broken, leaders from LangChain, Conviva and CoreWeave said at VB Transform 2026

A single AI agent conversation can look perfect and still be broken, leaders from LangChain, Conviva and CoreWeave said at VB Transform 2026

A single AI agent conversation can look flawless scored on its own and still point to a broken product. That gap is driving a shift in how enterprises evaluate agents, away from scoring individual traces and toward comparing cohorts of users against a baseline.At VB Transform 2026, Harrison Chase, CEO of LangChain; Hui Zhang, CTO and co-founder of Conviva; and Emmanuel Turlay, director of engineering at CoreWeave, described that shift, along with a parallel move toward cheaper, narrower judge models.Agent-as-judge — judging one AI agent's output with another — hasn't replaced LLM-as-judge, which Chase said remains the default. The larger tension, Zhang said, is between automated judging, whether by LLM or agent, and human review.You have scalable but ungrounded, whether it's agents as judge or LLMs as judge, you grade the outcome, you grade the work. It still is very difficult to ground it and then you use humans and that's just not scalable, Zhang said. The whole industry is facing this, which poison you want to pick.Evaluation criteria now function as the product specThat gap — a conversation that scores well but still signals a broken product — is what teams try to close by building an exhaustive evaluation suite before they ship anything. Chase said that doesn't work.We sometimes see teams that have almost eval paralysis, Chase said. They're like, this is an eval set, I can't launch it. The best teams launch and then iterate.Chase framed evaluation criteria as a living specification, not a one-time test suite: a product requirements document — the standard software-development spec for what an application should do. Evals are like the new PRD, he said. They define what your agent should and shouldn't do.Turlay described hitting the same failure from a different angle. I was trying to reach 100 coverage for my tests, and I still had bugs in production, he said — a test suite that looked complete but still missed what mattered, the same gap Chase was describing with evals.Broad, always-on monitoring, he said, catches more real failures than an exhaustive pre-launch test suite. Teams should set up wide online checks first, use those to identify failure classes as they occur, then build a targeted offline evaluation set around the problems that surface.Why scoring traces one at a time is a mistakeEven a well-built evaluation process can still score the wrong thing. Zhang's objection is to how most teams run evaluation: sampling traces, whether 50 of them or a full population, scoring each in isolation. That approach misses a signal that only shows up when comparing cohorts of users against a baseline, a method Zhang calls contrastive analysis.Zhang illustrated it with a retail example: a shopper asks an agent for a running shoe ahead of a half marathon, the agent asks qualifying questions, and the shopper buys a shoe. Scored individually, that interaction looks fine. But the clarification ratio, how many follow-up questions an agent asks before completing a task, came in three times higher than baseline for that shoe category across the full user population. A second metric, how often shoppers finished their purchase outside the conversation, was five times higher than baseline for the same category.Neither number is visible from a single trace. Both point to a debuggable, category-specific problem. Zhang said the industry also lacks a second data source: what happens before, between and after the conversation, not just the trace itself.Sizing the judge to the jobOnce contrastive analysis flags which category is actually broken, the next problem is what watches for it going forward — and at what cost. Turlay's rule was to start with the most capable model available to prove a task is solvable, then work down. If it can't be done with a top-tier model, he said, it won't work with a smaller one. Once a pattern proves viable, teams can sample a fraction of traffic instead of judging every interaction, and move simpler tasks like binary classification to smaller open source models.LangChain took that further, fine-tuning its own model to detect when a user believes the agent made a mistake, a signal Chase calls perceived error. The model we fine-tuned was a Qwen model, he said, referring to Alibaba's open source family. Combining hand labeling with distillation, the result performed well. Same as [Claude]Sonnet, for, depending on how we served it, either 10 to 100x cost reduction, Chase said.Not every guardrail needs a model. Chase pointed to Claude Code's own guardrails as proof: regexes, the common programming technique for finding and validating patterns in code. A lot of the guardrails they had were just regexes, he said. They weren't small LLMs, they were just regexes.LLM-as-judge doesn't mean human-in-the-loop disappearsThe bigger question is whether using LLM as a judge removes the need for a human in the loop.Turlay pointed to accountability, drawing on his prior work at a self-driving car company. His team compressed data intake and retraining into a two-week cycle for shipping a new model to the car. Even then, someone still had to sign off.I felt confident on behalf of the company to say this model should go into the car, he said. The same logic extends to legal, finance and healthcare. Before we can remove a human to say, I endorse this and I take responsibility legally for it, it's going to be a while before agents can do that on their own.Zhang agreed a human has to remain the guardian on corner cases, even as automation eventually runs at a scale that beats individual human accuracy — machines can see more at the pattern level. Chase went further: that human check isn't just a safety net. Human in the loop is really important for building trust in how these agentic systems work, and also really important for memory and learning from systems, he said. There has to be interactions in order for the system to learn.

1 day ago
At VB Transform 2026, Zillow's engineering chief said AI ROI numbers only hold up if you measure before you build

At VB Transform 2026, Zillow's engineering chief said AI ROI numbers only hold up if you measure before you build

Zillow, the real estate technology company, doesn't get one conversation with its customers. They move from a phone screen to a loan officer to a real estate agent, sometimes over months or years, and expect the context to follow them. A single chatbot could never carry that thread.At VB Transform 2026, Zillow SVP of Engineering Toby Roberts and Glean co-founder and CEO Arvind Jain described how they built AI architecture meant to carry context across that entire journey — and why context, not raw data, turned out to be the harder problem to solve. Zillow's products touch roughly 80 of U.S. real estate transactions each year, and the company has been using AI long before ChatGPT existed.We pretty quickly identified that we were going to need a persistent context layer that was going to meet our customers and the professionals wherever they were, Roberts said.Data was never the hard partRoberts said Zillow's AI effort started where most enterprise AI efforts start, with the data itself.We started with a large push around making sure our data did have the right foundation, Roberts said. That meant a data mesh approach, clear data lineage and a governance structure with permissions and identity attached to the data itself.None of that turned out to be the hard problem. The hard problem was building something that remembered where a customer was in their journey and carried that forward, no matter which surface they showed up on next.This context layer has to live to be able to support you where you are at any given point in your journey, Roberts said. Zillow chose to own that layer itself rather than depend on a single external chat interface, a decision Roberts said the team reached quickly once it looked at the shape of a real transaction rather than a single conversation.Why Zillow built its own architecture, and where Glean fits into itZillow built its own harness rather than route customers through a single model API. The team drew on 20 years of machine learning history behind products like Zestimate, leaning into smaller, task-specific fine-tuned models instead of one general-purpose model.Internally, that harness runs alongside Glean. Roberts said Zillow now has thousands of Glean agents in production, handling repetitive tasks with tens of thousands of executions across the company. Glean's pitch, per Jain, is centralizing that integration work once, through the Glean MCP gateway, rather than letting finance, legal and marketing each rebuild their own connections to the same systems.That centralization is also a cost lever. Jain pointed to two mechanisms: model routing, which sends most tasks to smaller, cheaper models instead of defaulting to frontier models, and precomputed context, which avoids an agent burning tokens assembling its own context from scratch.Claude is also very slow because the first part of assembling that context actually takes forever, Jain said. Routing that request through Glean instead, he said, can cut token consumption by as much as half.What Zillow and Glean's approach means for enterprisesAcross data, cost and permissions, the session offered a few practical takeaways for enterprises building agentic AI on their own systems.Build the measurement baseline before the AI push, not after. Roberts said Zillow's ability to credibly attribute a 40 increase in shipped code to AI adoption rests on a DORA metrics baseline the team put in place years earlier, not on the AI rollout itself.Centralize context once instead of letting every team rebuild it. Jain's core argument for Glean's platform is that duplicated integration work across finance, legal and marketing teams is a hidden cost most enterprises haven't accounted for.Don't assume permission inheritance is enough for regulated data. Even with a permissions-aware context platform in place, Zillow layered hard rules and a standing compliance check on top for its most sensitive categories, rather than trusting the architecture to handle it automatically.Treat context as a cost lever, not just a capability. Model routing and precomputed context were the two mechanisms Jain pointed to for cutting AI spend, both aimed at reducing wasted token consumption rather than adding new capability.Models by themselves are not enough to bring automation with AI inside your enterprise, Jain said. You do have to connect it with your enterprise context.

2 days ago
VentureBeat — News Tracking & Analysis | Real Narrative News