DeepSeek open-sources DeepInfer inference framework and DeepSeek-recipe codecs, benefiting third-party deployments

DeepSeek says DeepInfer already powers its large-scale V4.1 Flash service and will host inference code and optimizations for future models. Its previously released kernels and communication libraries now sit within an engine whose execution path developers can follow in source.

Deployment teams can approach the release through a few concrete questions. How many devices should prefill and decode use? Which parts of a computed context deserve days of storage, and which are useful for only minutes? When does verifying another draft token waste compute? The deployment report explains the trade-offs; the source makes parts of those decisions available to inspect and test.

What has DeepSeek actually released?

The initial release centers on the inference engine and its offline interface. The README lists DeepSeek-V4.1-Flash as the supported model; the default offline checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017. Hardware support is explicitly listed as NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. That list does not establish compatibility with every NVIDIA GPU or Ascend device.

The missing KV cache system refers to infrastructure required for a complete service. The engine source already contains device cache operations, block management and tests. DeepInfer uses Apache-2.0; Recipe's code and public documentation use MIT. Model weights and bundled tokenizers have their own licensing information.

The README credits vLLM as the main basis for the original engine, with additional inspiration from SGLang and TensorRT-LLM. Integrated kernels and communication components include FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. The release puts those components within an engine execution path that developers can study.

Why does prefill use smaller parallel groups while decode uses wide EP?

Prefill processes input tokens that have not yet been computed, usually providing substantial work within a batch. Decode repeatedly advances active requests, with less work per request at each step. It benefits from batching across requests and efficient communication. DeepSeek separates vision encoding, prefill and decode so it can allocate resources independently and control time to first token (TTFT) and time between tokens (TBT).

API request→Recipe encoding→Routing & cache lookup

Vision encoder

Images → embeddings
Usually one device

Prefill

Uncached input computation
EP / DP 2 to 8

Decode

Generation & verification
EP / DP 64 to 128

Model token stream→Recipe parsing→Protocol response

In the reported production configuration, attention uses data parallelism (DP), MoE layers use expert parallelism (EP), and the DP and EP counts match. Prefill uses the smallest practical EP/DP group that fits in memory. Decode uses larger groups, typically 64 to 128 devices, within a scale-up communication domain. The repository also supports attention context parallelism (CP); CP, DP and EP describe different ways of distributing work.

Wide EP lets more requests contribute to expert batches, but imbalance can leave devices waiting. The reported expert load balancer keeps redundant copies of popular experts and changes their placement using recent load statistics. Communication and balancing must support the larger group for extra devices to improve utilization.

The routing metrics also differ by stage. Prefill estimates attention cost with a fitted function cost(q,c), where q is the uncached token count and c the computed prefix length. It solves for a chunk size under a time budget. Decode balances both request count and aggregate KV length, accounting for batch size as well as the amount of context attention must read.

How does SWA bounded replay reduce recomputation?

Sliding-window attention directly reads a local range, but hidden states in earlier layers already contain information from further back. Recovering a lost window exactly can therefore require a wider range of computation as dependencies propagate through layers. The report gives a decoder example with 20 layers and a 128-token window: exact reconstruction requires processing the final 2,560 tokens.

V4.1 instead replays the final 128 tokens and restricts attention to that replay segment. Starting at position s, a query at i attends to keys from max(s, i-127) through i. This produces an approximate SWA state with a fixed replay span.

The approximation determines what an evaluation needs to check. On the encoder path, existing block KV is reused while missing SWA state is rebuilt. On the decoder path, the Causal Encoder-Decoder (CED) architecture allows block KV to be projected from the encoder's final hidden states; the decoder mainly reconstructs its own SWA state. DeepSeek reports that total prefill computation is nearly halved in its deployment. That benefit depends on the V4.1 architecture and serving configuration.

What does compact mode do in the source?

At the n_prefill_layer boundary, deepseek.py switches to consumer metadata. In yoco.py, compact_yoco_boundary gathers the retained hidden states, residuals and related indices, reducing the tensor rows handled by later layers. The repository retains a mask implementation and provides metadata and log-probability checks comparing the two paths.

The algorithm's window is 128 tokens; the reported SWA ring has 134 positions (128 plus six for backtracking); the README recommends a decoder retention setting of 140. These numbers describe different objects. Use the documented runtime setting. The mask/compact checks compare implementations of the truncation policy; they do not prove that approximate replay equals exact reconstruction from the full history.

# Decoder bounded replay settings from the official README
export DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
export DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140

The fixed bound applies to the token span. Network layers still execute, so recovery is not free or independent of model depth.

Why keep some KV state for days and other state for three minutes?

DeepSeek gives the two types of state different storage policies. Block KV contains long prefixes that may be reused repeatedly and lives in an independent 3FS cluster. SWA KV is mostly useful during an active conversation and stays in the host memory of the machine that produced it. A remote hit retrieves it; a miss triggers bounded replay.

The report describes approximately 100 PiB of SSD capacity for block KV, peak aggregate reads of 10 TiB/s and peak writes of approximately 850 GiB/s. Those are figures for DeepSeek's storage infrastructure, not default resources provided by installing the engine.

Storage placement also changes routing. The report says any prefill machine can hit any prefix, allowing routing by available compute rather than affinity to a local cache. Our reading is that cache placement becomes part of the scheduling design: it determines how freely the load balancer can assign work, as well as whether a prefix hits.

Why does DSpark avoid verifying every draft token?

A draft token that is likely to be rejected still consumes target-model compute during verification. DSpark adds a lightweight sequential logit correction for adjacent-token conditioning and uses a confidence head to estimate acceptance. The reported online system drafts five tokens at a time, then combines those estimates with a hardware throughput curve to choose a verification budget.

The objective is visible in csrc/engine/mtp_varlen.hpp. The scheduler accumulates expected accepted tokens and divides by estimated forward and sampler time to select a useful batch size. It also maintains runtime estimates that adjust the startup performance curve. More verification work is worthwhile only when it increases the expected rate of accepted output.

DeepSeek reports a 6% production throughput improvement over fixed verification of five draft tokens. That is the baseline for the claim; it does not establish a 6% advantage over other inference frameworks. Asynchronous scheduling adds another constraint: the latest confidence values are not yet back when the CPU schedules the next step. The CPU selects a total budget, and a TileLang kernel on the GPU refines per-request lengths.

What do asynchronous scheduling and structured output change?

A serial loop waits for GPU sampling, runs CPU post-processing, then schedules the next batch. The GPU can sit idle between those steps. DeepInfer enables asynchronous scheduling by default, overlapping CPU scheduling of the next batch with current device execution. The report calls it the Zero-Overhead Batch Scheduler, referring to hiding scheduling overhead in the pipeline. Scheduling still consumes resources.

Structured output constrains which tokens sampling may select. The reported service compiles JSON Schema, regex or EBNF constraints into an automaton and generates a mask from its current state during decoding. The engine accepts constraints through SamplingParams.so_args; Recipe converts protocol-level format requests into backend options. The service must connect those interfaces correctly. Rendering a schema into a prompt alone does not enforce constrained decoding.

Reproducibility has its own configuration. The README lists batch-invariant inference and deterministic sampling as NVIDIA features. Deterministic sampling requires DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed in SamplingParams. A fixed random seed alone is not the documented recipe while changing the dynamic verification policy.

Why does deepseek-recipe deserve separate attention?

Identical weights do not guarantee identical API behavior. Message templates, thinking boundaries, tool parameters and streaming chunks affect either what reaches the model or how clients interpret its output. Recipe converts Messages, Chat Completions and Responses requests into a shared Conversation representation, then renders V4/V4.1 prompts and token IDs. The reverse path parses model output into protocol events. The core is written in Rust with Python bindings.

The report describes a subtle encoding distinction. User text such as <｜User｜> can look like an inserted role token without having its control meaning. Recipe records special tokens explicitly inserted by the template and encodes matching literal strings in ordinary input as ordinary tokens. Output parsing also identifies boundaries by token ID. This addresses control-token confusion; it is not a claim to solve every form of prompt injection.

Streaming adds partial UTF-8 characters, DSML tags and tool arguments that may cross chunk boundaries. The described state machine buffers incomplete fragments, continues incremental matching and returns content as soon as its meaning is established. On input, known template boundaries allow ordinary text segments to be tokenized in parallel. These operations sit outside model execution but affect first-token latency and tool responsiveness.

Recipe explicitly leaves model inference, tool execution and HTTP transport to its caller. The Python and Rust example servers use mock inference. Running those demos can validate protocol conversion, but does not deploy a working model service.

How should readers interpret throughput, latency and revenue?

Production totals, offline engine measurements and model-quality regressions answer different questions. DeepSeek reports 155 trillion tokens per day for a single cluster and revenue of $38 per accelerator-hour, calculated at off-peak pricing. These are company-reported figures, not independently reproduced here. Revenue has not deducted hardware, storage, networking or operational costs, so it cannot establish profit or the cost of each output token.

The benchmark source adds an essential qualification. examples/README.md says raw decode throughput counts all draft tokens as accepted. For fixed-length verification, plot_pareto.py scales with τ = 1 + nextn × a, where a is the supplied acceptance rate. The chart uses nextn=3 and a=0.75, giving 3.25 expected tokens per step. The plotting code explicitly rejects applying that calculation to variable-length verification data. Check this accounting before comparing the chart with application output rates.

The curves illustrate throughput and latency trade-offs. They do not include matched configurations of competing engines and therefore do not establish general superiority over vLLM or SGLang.

What should a deployment team verify first?

Confirm hardware and software compatibility before running offline examples. The README requires Linux, GCC 13.0 or later and Python 3.12, with CUDA 13 for NVIDIA or CANN 9.2 for Ascend. The following is the documented starting flow; it was not executed on supported accelerators for this article.

# Run from the official repository in a clean Python 3.12 environment
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py
# For Ascend, follow the documented DEEPINFER_ACCELERATOR=ascend path

After basic generation works, the following evaluation sequence should expose the practical trade-offs. These are recommendations derived from the public implementation, not test results from this article.

1. Pin the checkpoint, source commit and sampling settings. Check text, images, tool calls and structured output.
2. Measure TTFT, TBT, per-device throughput and actual draft acceptance at the context lengths and concurrency of the application. Include cache hits and misses.
3. Evaluate bounded replay, variable-length verification and asynchronous scheduling separately before attributing gains to an individual mechanism.
4. Before online deployment, integrate cache storage, stage routing, balancing across replicas and failure recovery. Include storage and network resources in total cost.

The repository also supplies offline evaluation workflows for AIME 2026, GPQA Diamond and HLE, with reference Pass@1 results of 99.9%, 93.4% and 42.6%, respectively. These are deployment sanity checks for model quality, not engine speed scores. The documentation requires coverage_complete in summary.json to be true; missing ranks or samples invalidate a complete-result claim.

For teams operating large MoE deployments, DeepInfer offers execution and scheduling code worth tracing. For teams mostly consuming hosted APIs, Recipe's templates, tool calls and streaming behavior are closer to daily integration problems. Evaluation should follow the application's latency, quality and resource requirements, rather than the highest-throughput point on a published curve.

Frequently asked questions

Is this a new model release?

The announcement concerns an inference framework and codec components. The initial DeepInfer support list contains DeepSeek-V4.1-Flash, with the 1017 checkpoint used by default.

Can it run directly on a consumer GPU or Windows?

The repository does not make that support claim. Its README lists Linux, NVIDIA SM100 and Ascend 950. Hardware planning must also account for model memory and parallel configuration.

Does Recipe reproduce the official API service?

It helps align protocol encoding and decoding. The caller still supplies inference, HTTP serving and tool execution, while model output also depends on weights, sampling and backend behavior.