DeepSeek changed deepseek-v4-flash on July 31 without changing the model name. A request sent before the update and the same request sent after it did not address the same checkpoint. The benchmark table arrived with a second moving part: DeepSeek Harness, the agent framework behind the public benchmark tasks DeepSeek classifies as code-agent work, is not public.
DeepSeek V4 Flash 0731 is strong enough and cheap enough to test. The launch table is not enough to tell an engineering team where to use it.
That distinction matters because the release is unusually easy to misread. The API price is low. The open weights use an MIT license. The agent scores look competitive. Each fact is useful, but none identifies the cost or reliability of the system a buyer would deploy.
The API name stayed still while the model moved
The DeepSeek changelog separates two releases. V4 Flash and V4 Pro entered preview on April 24. On July 31, DeepSeek moved the stable deepseek-v4-flash identifier to the dated 0731 checkpoint, kept the release in public beta, and left V4 Pro plus the app and web models unchanged.
The dated weights remove one ambiguity for self-hosted evaluation. DeepSeek-V4-Flash-0731 names a checkpoint and provides its files under MIT. The managed API documentation provides no equivalent dated request identifier or snapshot-pinning mechanism. A report that records only deepseek-v4-flash has recorded a moving alias, not reproducible model provenance.
There is another unresolved detail in the package itself. The API material describes a mixture-of-experts model with 284 billion total parameters and 13 billion active parameters. The dated checkpoint page reports 304 billion and separately identifies an attached DSpark speculative-decoding module. No fetched primary source reconciles the 20-billion-parameter difference, and the available records do not establish that DSpark explains all of it.
The practical fix is boring. Store the request date and provider with the returned model metadata; preserve the reasoning effort plus harness commit alongside the traces; for self-hosted runs, add the checkpoint hash and quantization, then tie the serving runtime to its hardware and kernel configuration. A model name is a label. Provenance is a record.
A benchmark score has more than a model inside it
The launch note says the public benchmark tasks DeepSeek classifies as code-agent work used DeepSeek Harness in minimal mode with maximum reasoning effort. It does not enumerate exactly which rows that note covers, and the harness is described as forthcoming. The launch table also leaves the comparison conditions open. Prompts and tools are unspecified. So are the context policy, execution budget, retry rule, and evaluator.
Those omissions change the object being measured.
Controlled work puts a scale on the concern. Zheng and collaborators held the model fixed in Claw-SWE-Bench and found a 27.4-percentage-point difference between harnesses. One direct-diff adapter reached 19.1% Pass@1; a fuller adapter that moved patch transport out of free-form model output reached 73.4%. Vats and Golev later reported about a 42-fold spread in tokens per solved task across harnesses in a 300-trial study with models and tasks held fixed.
That does not make every benchmark a harness benchmark. The authors of NL2Repo-Bench found less than 1% variation for one model across three frameworks. Model capability dominated that experiment. The disagreement is the useful part: harness sensitivity depends on the task and interface design. It has to be measured rather than assumed away.
Two outside checks confirm strength, not equivalence
Independent evidence supports putting 0731 into an evaluation. Artificial Analysis reports an Intelligence Index v4.1 score of 49.93, displayed as 50, at maximum effort. That places the model in serious company without relying on DeepSeek's own composite table.
There are also two outside overlaps with the launch rows. DeepSeek reports 82.7 on Terminal-Bench 2.1. Artificial Analysis reports 78.65 using Terminus 2 in E2B across all 89 tasks with three repeats. The 4.05-point gap belongs to the execution systems. Available evidence cannot allocate it between the harness and its environment. Episode policy and evaluator differences remain in the comparison too.
Toolathlon points in the other direction. DeepSeek publishes an unlabeled 70.3. The benchmark owner lists 70.7 plus or minus 0.9 Pass@1 with its Default agent. Those values are statistically compatible, but they still describe different systems.
The honest read is narrow. Independent evaluation says the model is strong. It does not reproduce DeepSeek Harness, establish the size of the post-training gain under a buyer's harness, or show that every launch row transfers.
Seven rows still need their labels back
The problem with the rest of the table is not evidence that the numbers are false. DeepSeek removed the metric and execution labels that make them actionable.
| Launch row | What the source benchmark measures | What the launch leaves out |
| NL2Repo 54.2 | Average test pass rate inside generated repositories | Whole-repository completion, harness, budget, repeats |
| CyberGym 76.7 | Exploit success under a named level and benchmark split | Level, full set or subset, iteration policy |
| DeepSWE 54.4 | Functional verification under fixed mini-swe-agent conditions | Comparable fixed harness, timeout, rollouts |
| Toolathlon 70.3 | Pass@1, pass@3, or all-three success | Metric label, agent, turns, timeout |
| Agents' Last Exam 25.2 | Mean score or full pass rate across named splits and interfaces | Split, tier, interface, metric, harness |
| AutomationBench Public 25.1 | Binary end-state success on public tasks | Tool version, step policy, retries |
| Two DSBench rows | Internal DeepSeek evaluations | Public tasks, metric, evaluator, leakage controls |
Three examples show why the labels matter. NL2Repo-Bench averages the fraction of tests passed inside generated repositories. A 54.2 result does not mean that 54.2% of repositories were completed. CyberGym contains 1,507 real-world vulnerability instances, but its evaluations vary by information level and subset. Iteration policy changes them again. The launch does not establish 76.7% success across the entire collection. Agents' Last Exam separates public from private tasks and distinguishes difficulty tiers. Its interfaces and metrics vary too. The 25.2 row does not establish that a quarter of professional work was automated.
DeepSWE exposes a second issue. Its benchmark paper fixes mini-swe-agent with the prompt and tool behavior. Timeout plus rollout policy are fixed as well, so model comparisons share a system. Substituting another harness may be a valid agent evaluation, but it is no longer the same controlled model comparison.
The two DSBench rows are harder to use. No public task definitions or task counts were found. Metrics and evaluators are absent, as are traces and leakage controls. These remain vendor claims about internal evaluations. Nothing more should be inferred from them.
Cheap tokens can still produce expensive accepted work
The posted Flash prices are difficult to ignore: $0.0028 per million cache-hit input tokens, $0.14 per million cache-miss input tokens, and $0.28 per million output tokens. Cheap is visible. Value is not.
Artificial Analysis spent $72.02 on its full index run and generated 205,983,768 output tokens, of which 195,798,353 were reasoning tokens. Reasoning accounted for 95.1% of output and about 76.1% of measured spend. Its weighted $0.02713 figure is cost per attempted index item. A production team pays for failed attempts and retries. It also pays for tool infrastructure, execution compute, and review before it gets an accepted result.
The useful denominator changes:
cost per accepted task = total attempts, retries, tools, compute, and review / accepted outcomes
A small public coding suite shows the difference without resolving it. OneShotLM reported 35 builds for $1.39 and 650,000 tokens, with three missing runnable outputs and a 102.3-second average. The scoring is site-specific, and the aggregate changed during the research window. Provider routing was not fully disclosed either. It is a workload signal, not a model verdict.
Even the cheapest input rate depends on a system behavior. The context cache works automatically from matching prefixes. Availability is best effort. Construction takes time, and unused entries expire. DeepSeek has also announced future peak periods at twice the base price without publishing an effective date. Cost models need cache-hit distributions and time-of-day exposure, not the best number from the pricing table.
The runtime can change the model you think you tested
Several early 0731 problems belong to adapters and serving stacks. A deployed route still pays for them.
A reproducible OpenRouter test found that providers rendered requested reasoning levels incorrectly. An SGLang 0.5.16 issue traced a one-step effort mapping error that made the vendor's max level unreachable through the tested runtime. A benchmark requested at maximum effort and served at high effort is not a reproduction, even when the model name matches.
Self-hosting introduced a different class of failure. One published DSpark recipe found that its loader had dropped 12 drafter tensors. Correcting the loader increased mean throughput from 32.7 to 55.4 tokens per second. No before-and-after quality benchmark was reported, so the evidence supports a serving-performance claim, not a quality claim.
The official API has integration traps too. In thinking-mode Chat Completions tool loops, clients must preserve and resend reasoning_content or the next request fails. The Responses API is stateless and does not accept images or files. It silently ignores several unsupported fields. The current Chat Completions schema and the official Oh My Pi guide also conflict on forced tool selection in thinking mode.
These are route defects. Label results with the checkpoint and provider, then identify the protocol and runtime. Preserve the requested effort, harness, tools, and evaluator. Otherwise a change in an adapter can masquerade as a change in intelligence.
The routing decision includes the data path
The low price has no value when the hosted data path conflicts with the workload. The DeepSeek privacy policy says its services may collect user content plus device and log data. It may use data to improve and train its technology. The company says personal data is directly processed and stored in the People's Republic of China, and the service is not designed for sensitive personal data.
The fetched English Open Platform terms and privacy documents did not expose an API zero-retention setting or a no-training control. A precise API-specific retention period was absent too. Those are unanswered procurement questions. Their absence is not proof that every API request receives one particular retention treatment.
Open weights create another route. It is not a small one. The official vLLM recipe uses a four-GPU GB300 node, while early Ampere deployments reported kernel and data-type blockers. Long-context serving added further failures. Self-hosting changes the data path and transfers the serving burden to the operator. It deserves its own evaluation rather than being treated as a privacy toggle on the managed API.
Evaluate the route before replacing the route
The first evaluation stage should freeze the system. Pin the task snapshots and harness commit. Lock prompts, tools, permissions, and context policy. Hold the reasoning effort plus execution budget constant, then fix the retry policy and evaluator with a repetition count. Compare 0731 with the incumbent and one cost-oriented control. Record accepted tasks against failed attempts. Track cached and uncached input, reasoning and answer tokens, wall time, invalid tool calls, recovery turns, reviewer minutes, and severity-weighted defects.
Then optimize the promising model-harness pairs. Change prompts and tools deliberately. Tune compaction, caching, and effort, but report the fixed baseline and optimized pair as separate results. This keeps a better harness from being reported as a better model while still letting the deployed system improve.
Add a version canary. Send a small frozen suite on a schedule, retain response metadata and traces, and alert on behavioral drift. That is the minimum protection when the production model identifier can move without the request changing.
The release clears the bar for a trial. It does not clear the bar for a migration. That line is crossed only when the same tasks, verifier, budget, and runtime produce more accepted work for less total cost. DeepSeek priced the tokens. The buyer still has to price the system.
References
Primary release and implementation sources
- DeepSeek. API updates. July 2026.
- DeepSeek. DeepSeek V4 Flash. April 2026.
- DeepSeek. DeepSeek-V4-Flash-0731. July 2026.
- DeepSeek. Thinking mode. Accessed August 2026.
- DeepSeek. Using the Responses API. Accessed August 2026.
Benchmark definitions and independent results
- Ding and collaborators. NL2Repo-Bench. January 2026.
- Zheng and collaborators. Claw-SWE-Bench. June 2026.
- Huang and collaborators. DeepSWE. July 2026.
- Toolathlon. Model leaderboard. Accessed August 2026.
- Artificial Analysis. DeepSeek V4 Flash analysis. Accessed August 2026.
- Artificial Analysis. Intelligence benchmarking methodology. Accessed August 2026.
Harness and evaluation research
- Vats, Naman, and Oleg Golev. The Scaffold Effect in Coding Agents. June 2026.
- Wang and collaborators. CyberGym. June 2025.
- Berkeley RDI and collaborators. Agents' Last Exam. June 2026.
- Zapier. AutomationBench. April 2026.
Commercial, practitioner, and data-handling sources
- DeepSeek. Pricing. Accessed August 2026.
- DeepSeek. Context caching. Accessed August 2026.
- DeepSeek. Privacy policy. February 2026.
- SGLang. DeepSeek V4 Flash 0731 reasoning-effort mapping issue. August 2026.
- tonyd2wild. DeepSeek V4 Flash 0731 DSpark serving recipe. August 2026.
